Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-9161

Postgres: log errors from keepalive thread for replication

XMLWordPrintable

    • False
    • Hide

      None

      Show
      None
    • False

      (Additional context at https://debezium.zulipchat.com/#narrow/channel/348249-community-postgresql/topic/Errors.20using.20postgres.20cursor.20on.20ad-hoc.20snapshot/with/523829756 )

      Bug report

      For bug reports, provide this information, please:

      What Debezium connector do you use and what version?

      v2.6.2

      What is the connector configuration?

      N/A

      What is the captured database version and mode of deployment?

      PG 13 on Amazon Aurora

      What behavior do you expect?

      Keepalive threads for streaming connections should immediately log any exceptions that occur with the streaming connection.

      What behavior do you see?

      Errors are suppressed and never logged.

      Do you see the same behaviour using the latest released Debezium version?

      Not tested; code analysis suggests the problematic behavior still exists.

      Do you have the connector logs, ideally from start till finish?

      see https://debezium.zulipchat.com/#narrow/channel/348249-community-postgresql/topic/Broken.20pipe.20on.20streaming.20connection.20after.20blocking.20snapshot/with/498249074 

      the key error was this when resuming streaming, which suggested that the connection might have been broken earlier and not properly logged by the keepalive thread:

      org.postgresql.util.PSQLException: Database connection failed when writing to copyat org.postgresql.core.v3.QueryExecutorImpl.flushCopy(QueryExecutorImpl.java:1176)at org.postgresql.core.v3.CopyDualImpl.flushCopy(CopyDualImpl.java:30)at org.postgresql.core.v3.replication.V3PGReplicationStream.updateStatusInternal(V3PGReplicationStream.java:195)at org.postgresql.core.v3.replication.V3PGReplicationStream.timeUpdateStatus(V3PGReplicationStream.java:186)at org.postgresql.core.v3.replication.V3PGReplicationStream.readInternal(V3PGReplicationStream.java:128)at org.postgresql.core.v3.replication.V3PGReplicationStream.readPending(V3PGReplicationStream.java:82)at io.debezium.connector.postgresql.connection.PostgresReplicationConnection$1.readPending(PostgresReplicationConnection.java:622)at io.debezium.connector.postgresql.PostgresStreamingChangeEventSource.processMessages(PostgresStreamingChangeEventSource.java:217)at io.debezium.connector.postgresql.PostgresStreamingChangeEventSource.execute(PostgresStreamingChangeEventSource.java:179)at io.debezium.connector.postgresql.PostgresStreamingChangeEventSource.execute(PostgresStreamingChangeEventSource.java:42)at io.debezium.pipeline.ChangeEventSourceCoordinator.streamEvents(ChangeEventSourceCoordinator.java:280)at io.debezium.pipeline.ChangeEventSourceCoordinator.executeChangeEventSources(ChangeEventSourceCoordinator.java:197)at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:140)at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)at java.base/java.lang.Thread.run(Thread.java:829)Caused by: java.net.SocketException: Broken pipe (Write failed)at java.base/java.net.SocketOutputStream.socketWrite0(Native Method)at java.base/java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:110)at java.base/java.net.SocketOutputStream.write(SocketOutputStream.java:150)at java.base/java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:81)at java.base/java.io.BufferedOutputStream.flush(BufferedOutputStream.java:142)at org.postgresql.core.PGStream.flush(PGStream.java:721)at org.postgresql.core.v3.QueryExecutorImpl.flushCopy(QueryExecutorImpl.java:1174)... 17 more

      How to reproduce the issue using our tutorial deployment?

      Disconnect the streaming connection while doing a lengthy ad hoc blocking snapshot.

              Unassigned Unassigned
              james-johnston-thumbtack James Johnston (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: