Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-4520

Add backend errors among retriable for Postgres connector

    XMLWordPrintable

Details

    • Hide

      Step1: Upgrade to version 1.7

      Step2: create a postgres debezium source connector and start stream data

      Step3: Wait for a while, problem happen

      Check log below

      2022-01-04 09:32:07,648] INFO WorkerSourceTask\{id=source.postgresql_debezium.s14e_order.v1-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask)
      [2022-01-04 09:32:07,648] INFO WorkerSourceTask\{id=source.postgresql_debezium.s14e_order.v1-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask)
      [2022-01-04 09:32:07,650] INFO WorkerSourceTask\{id=source.postgresql_debezium.s14e_order.v1-0} Finished commitOffsets successfully in 2 ms (org.apache.kafka.connect.runtime.WorkerSourceTask)
      [2022-01-04 09:32:13,527] ERROR Failed to read column metadata for 'public.order_part2' (io.debezium.connector.postgresql.connection.pgoutput.PgOutputMessageDecoder)
      [2022-01-04 09:32:13,527] ERROR Producer failure (io.debezium.pipeline.ErrorHandler)
      org.postgresql.util.PSQLException: An I/O error occurred while sending to the backend.
      at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:349)
      at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:481)
      at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:401)
      at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:322)
      at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:308)
      at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:284)
      at org.postgresql.jdbc.PgStatement.executeQuery(PgStatement.java:236)
      at org.postgresql.jdbc.PgDatabaseMetaData.getColumns(PgDatabaseMetaData.java:1579)
      at io.debezium.connector.postgresql.connection.pgoutput.PgOutputMessageDecoder.getTableColumnsFromDatabase(PgOutputMessageDecoder.java:333)
      at io.debezium.connector.postgresql.connection.pgoutput.PgOutputMessageDecoder.handleRelationMessage(PgOutputMessageDecoder.java:272)
      at io.debezium.connector.postgresql.connection.pgoutput.PgOutputMessageDecoder.processNotEmptyMessage(PgOutputMessageDecoder.java:176)
      at io.debezium.connector.postgresql.connection.AbstractMessageDecoder.processMessage(AbstractMessageDecoder.java:33)
      at io.debezium.connector.postgresql.connection.PostgresReplicationConnection$1.deserializeMessages(PostgresReplicationConnection.java:493)
      at io.debezium.connector.postgresql.connection.PostgresReplicationConnection$1.readPending(PostgresReplicationConnection.java:485)
      at io.debezium.connector.postgresql.PostgresStreamingChangeEventSource.processMessages(PostgresStreamingChangeEventSource.java:205)
      at io.debezium.connector.postgresql.PostgresStreamingChangeEventSource.execute(PostgresStreamingChangeEventSource.java:167)
      at io.debezium.connector.postgresql.PostgresStreamingChangeEventSource.execute(PostgresStreamingChangeEventSource.java:40)
      at io.debezium.pipeline.ChangeEventSourceCoordinator.streamEvents(ChangeEventSourceCoordinator.java:166)
      at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:127)
      at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
      at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
      at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
      at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
      at java.base/java.lang.Thread.run(Thread.java:834)
      Caused by: java.net.SocketException: Broken pipe (Write failed)
      at java.base/java.net.SocketOutputStream.socketWrite0(Native Method)
      at java.base/java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:110)
      at java.base/java.net.SocketOutputStream.write(SocketOutputStream.java:150)
      at java.base/java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:81)
      at java.base/java.io.BufferedOutputStream.flush(BufferedOutputStream.java:142)
      at org.postgresql.core.PGStream.flush(PGStream.java:665)
      at org.postgresql.core.v3.QueryExecutorImpl.sendSync(QueryExecutorImpl.java:1469)
      at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:320)
      ... 23 more
      [2022-01-04 09:32:13,529] INFO Connection gracefully closed (io.debezium.jdbc.JdbcConnection)
      [2022-01-04 09:32:13,532] INFO Connection gracefully closed (io.debezium.jdbc.JdbcConnection)
      [2022-01-04 09:32:13,532] INFO Finished streaming (io.debezium.pipeline.ChangeEventSourceCoordinator)
      [2022-01-04 09:32:13,533] INFO Connected metrics set to 'false' (io.debezium.pipeline.metrics.StreamingChangeEventSourceMetrics)
      [2022-01-04 09:32:13,743] INFO WorkerSourceTask\{id=source.postgresql_debezium.s14e_order.v1-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask)
      [2022-01-04 09:32:13,743] INFO WorkerSourceTask\{id=source.postgresql_debezium.s14e_order.v1-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask)
      [2022-01-04 09:32:13,743] ERROR WorkerSourceTask\{id=source.postgresql_debezium.s14e_order.v1-0} Task threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask)
      org.apache.kafka.connect.errors.ConnectException: An exception occurred in the change event producer. This connector will be stopped.
      at io.debezium.pipeline.ErrorHandler.setProducerThrowable(ErrorHandler.java:42)
      at io.debezium.connector.postgresql.PostgresStreamingChangeEventSource.execute(PostgresStreamingChangeEventSource.java:170)
      at io.debezium.connector.postgresql.PostgresStreamingChangeEventSource.execute(PostgresStreamingChangeEventSource.java:40)
      at io.debezium.pipeline.ChangeEventSourceCoordinator.streamEvents(ChangeEventSourceCoordinator.java:166)
      at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:127)
      at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
      at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
      at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
      at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
      at java.base/java.lang.Thread.run(Thread.java:834)
      Caused by: org.postgresql.util.PSQLException: An I/O error occurred while sending to the backend.
      at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:349)
      at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:481)
      at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:401)
      at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:322)
      at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:308)
      at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:284)
      at org.postgresql.jdbc.PgStatement.executeQuery(PgStatement.java:236)
      at org.postgresql.jdbc.PgDatabaseMetaData.getColumns(PgDatabaseMetaData.java:1579)
      Caused by: java.net.SocketException: Broken pipe (Write failed)
      at java.base/java.net.SocketOutputStream.socketWrite0(Native Method)
      at java.base/java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:110)
      at java.base/java.net.SocketOutputStream.write(SocketOutputStream.java:150)
      at java.base/java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:81)
      at java.base/java.io.BufferedOutputStream.flush(BufferedOutputStream.java:142)
      at org.postgresql.core.PGStream.flush(PGStream.java:665)
      at org.postgresql.core.v3.QueryExecutorImpl.sendSync(QueryExecutorImpl.java:1469)
      at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:320)
      ... 23 more
      [2022-01-04 09:32:13,744] ERROR WorkerSourceTask\{id=source.postgresql_debezium.s14e_order.v1-0} Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask)
      [2022-01-04 09:32:13,744] INFO Stopping down connector (io.debezium.connector.common.BaseSourceTask)
      [2022-01-04 09:32:13,747] INFO Connection gracefully closed (io.debezium.jdbc.JdbcConnection)
      [2022-01-04 09:32:13,748] INFO Connection gracefully closed (io.debezium.jdbc.JdbcConnection)
      [2022-01-04 09:32:13,748] INFO [Producer clientId=connector-producer-source.postgresql_debezium.s14e_order.v1-0] Closing the Kafka producer with timeoutMillis = 30000 ms. (org.apache.kafka.clients.producer.KafkaProducer)
      [2022-01-04 09:32:13,750] INFO Publish thread interrupted for client_id=connector-producer-source.postgresql_debezium.s14e_order.v1-0 client_type=PRODUCER session= cluster=4-QDyBAsQemtjp7MZCIcJA (io.confluent.monitoring.clients.interceptor.MonitoringInterceptor)
      [2022-01-04 09:32:13,750] INFO Publishing Monitoring Metrics stopped for client_id=connector-producer-source.postgresql_debezium.s14e_order.v1-0 client_type=PRODUCER session= cluster=4-QDyBAsQemtjp7MZCIcJA (io.confluent.monitoring.clients.interceptor.MonitoringInterceptor)
      [2022-01-04 09:32:13,750] INFO [Producer clientId=confluent.monitoring.interceptor.connector-producer-source.postgresql_debezium.s14e_order.v1-0] Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms. (org.apache.kafka.clients.producer.KafkaProducer)
      
      Show
      Step1: Upgrade to version 1.7 Step2: create a postgres debezium source connector and start stream data Step3: Wait for a while, problem happen Check log below 2022-01-04 09:32:07,648] INFO WorkerSourceTask\{id=source.postgresql_debezium.s14e_order.v1-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask) [2022-01-04 09:32:07,648] INFO WorkerSourceTask\{id=source.postgresql_debezium.s14e_order.v1-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask) [2022-01-04 09:32:07,650] INFO WorkerSourceTask\{id=source.postgresql_debezium.s14e_order.v1-0} Finished commitOffsets successfully in 2 ms (org.apache.kafka.connect.runtime.WorkerSourceTask) [2022-01-04 09:32:13,527] ERROR Failed to read column metadata for 'public.order_part2' (io.debezium.connector.postgresql.connection.pgoutput.PgOutputMessageDecoder) [2022-01-04 09:32:13,527] ERROR Producer failure (io.debezium.pipeline.ErrorHandler) org.postgresql.util.PSQLException: An I/O error occurred while sending to the backend. at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:349) at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:481) at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:401) at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:322) at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:308) at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:284) at org.postgresql.jdbc.PgStatement.executeQuery(PgStatement.java:236) at org.postgresql.jdbc.PgDatabaseMetaData.getColumns(PgDatabaseMetaData.java:1579) at io.debezium.connector.postgresql.connection.pgoutput.PgOutputMessageDecoder.getTableColumnsFromDatabase(PgOutputMessageDecoder.java:333) at io.debezium.connector.postgresql.connection.pgoutput.PgOutputMessageDecoder.handleRelationMessage(PgOutputMessageDecoder.java:272) at io.debezium.connector.postgresql.connection.pgoutput.PgOutputMessageDecoder.processNotEmptyMessage(PgOutputMessageDecoder.java:176) at io.debezium.connector.postgresql.connection.AbstractMessageDecoder.processMessage(AbstractMessageDecoder.java:33) at io.debezium.connector.postgresql.connection.PostgresReplicationConnection$1.deserializeMessages(PostgresReplicationConnection.java:493) at io.debezium.connector.postgresql.connection.PostgresReplicationConnection$1.readPending(PostgresReplicationConnection.java:485) at io.debezium.connector.postgresql.PostgresStreamingChangeEventSource.processMessages(PostgresStreamingChangeEventSource.java:205) at io.debezium.connector.postgresql.PostgresStreamingChangeEventSource.execute(PostgresStreamingChangeEventSource.java:167) at io.debezium.connector.postgresql.PostgresStreamingChangeEventSource.execute(PostgresStreamingChangeEventSource.java:40) at io.debezium.pipeline.ChangeEventSourceCoordinator.streamEvents(ChangeEventSourceCoordinator.java:166) at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:127) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: java.net.SocketException: Broken pipe (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:110) at java.base/java.net.SocketOutputStream.write(SocketOutputStream.java:150) at java.base/java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:81) at java.base/java.io.BufferedOutputStream.flush(BufferedOutputStream.java:142) at org.postgresql.core.PGStream.flush(PGStream.java:665) at org.postgresql.core.v3.QueryExecutorImpl.sendSync(QueryExecutorImpl.java:1469) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:320) ... 23 more [2022-01-04 09:32:13,529] INFO Connection gracefully closed (io.debezium.jdbc.JdbcConnection) [2022-01-04 09:32:13,532] INFO Connection gracefully closed (io.debezium.jdbc.JdbcConnection) [2022-01-04 09:32:13,532] INFO Finished streaming (io.debezium.pipeline.ChangeEventSourceCoordinator) [2022-01-04 09:32:13,533] INFO Connected metrics set to 'false' (io.debezium.pipeline.metrics.StreamingChangeEventSourceMetrics) [2022-01-04 09:32:13,743] INFO WorkerSourceTask\{id=source.postgresql_debezium.s14e_order.v1-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask) [2022-01-04 09:32:13,743] INFO WorkerSourceTask\{id=source.postgresql_debezium.s14e_order.v1-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask) [2022-01-04 09:32:13,743] ERROR WorkerSourceTask\{id=source.postgresql_debezium.s14e_order.v1-0} Task threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask) org.apache.kafka.connect.errors.ConnectException: An exception occurred in the change event producer. This connector will be stopped. at io.debezium.pipeline.ErrorHandler.setProducerThrowable(ErrorHandler.java:42) at io.debezium.connector.postgresql.PostgresStreamingChangeEventSource.execute(PostgresStreamingChangeEventSource.java:170) at io.debezium.connector.postgresql.PostgresStreamingChangeEventSource.execute(PostgresStreamingChangeEventSource.java:40) at io.debezium.pipeline.ChangeEventSourceCoordinator.streamEvents(ChangeEventSourceCoordinator.java:166) at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:127) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: org.postgresql.util.PSQLException: An I/O error occurred while sending to the backend. at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:349) at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:481) at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:401) at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:322) at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:308) at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:284) at org.postgresql.jdbc.PgStatement.executeQuery(PgStatement.java:236) at org.postgresql.jdbc.PgDatabaseMetaData.getColumns(PgDatabaseMetaData.java:1579) Caused by: java.net.SocketException: Broken pipe (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:110) at java.base/java.net.SocketOutputStream.write(SocketOutputStream.java:150) at java.base/java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:81) at java.base/java.io.BufferedOutputStream.flush(BufferedOutputStream.java:142) at org.postgresql.core.PGStream.flush(PGStream.java:665) at org.postgresql.core.v3.QueryExecutorImpl.sendSync(QueryExecutorImpl.java:1469) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:320) ... 23 more [2022-01-04 09:32:13,744] ERROR WorkerSourceTask\{id=source.postgresql_debezium.s14e_order.v1-0} Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask) [2022-01-04 09:32:13,744] INFO Stopping down connector (io.debezium.connector.common.BaseSourceTask) [2022-01-04 09:32:13,747] INFO Connection gracefully closed (io.debezium.jdbc.JdbcConnection) [2022-01-04 09:32:13,748] INFO Connection gracefully closed (io.debezium.jdbc.JdbcConnection) [2022-01-04 09:32:13,748] INFO [Producer clientId=connector-producer-source.postgresql_debezium.s14e_order.v1-0] Closing the Kafka producer with timeoutMillis = 30000 ms. (org.apache.kafka.clients.producer.KafkaProducer) [2022-01-04 09:32:13,750] INFO Publish thread interrupted for client_id=connector-producer-source.postgresql_debezium.s14e_order.v1-0 client_type=PRODUCER session= cluster=4-QDyBAsQemtjp7MZCIcJA (io.confluent.monitoring.clients.interceptor.MonitoringInterceptor) [2022-01-04 09:32:13,750] INFO Publishing Monitoring Metrics stopped for client_id=connector-producer-source.postgresql_debezium.s14e_order.v1-0 client_type=PRODUCER session= cluster=4-QDyBAsQemtjp7MZCIcJA (io.confluent.monitoring.clients.interceptor.MonitoringInterceptor) [2022-01-04 09:32:13,750] INFO [Producer clientId=confluent.monitoring.interceptor.connector-producer-source.postgresql_debezium.s14e_order.v1-0] Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms. (org.apache.kafka.clients.producer.KafkaProducer)

    Description

      After upgrade version from 1.3 to 1.7, the following error occurs: 

      org.postgresql.util.PSQLException: An I/O error occurred while sending to the backend

      Then I revert to version 1.3 and back to normal

      plaese help me find the root cause of this
      Does anyone have any idea?

      Attachments

        Activity

          People

            jpechane Jiri Pechanec
            tin.nguyen8 Nguyen Tin (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: