Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-5097

Postgresql connector does not retry one some errors when postgres is taken offline

XMLWordPrintable

    • False
    • None
    • False
    • Hide

      While running all postgres tutorial containers, issue either `docker-compose stop postgres` or `docker-compose kill postgres`.

      Show
      While running all postgres tutorial containers, issue either `docker-compose stop postgres` or `docker-compose kill postgres`.

      Bug report

      What Debezium connector do you use and what version?

      Postgres 1.9.2.Final

      What is the connector configuration?

      {
          "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
          "database.hostname": "postgres",
          "database.port": "5432",
          "database.user": "REDACTED",
          "database.password": "REDACTED",
          "database.dbname": "REDACTED",
          "database.server.name": "REDACTED",
          "plugin.name": "pgoutput",
          "publication.autocreate.mode": "filtered",
          "table.include.list": "public.REDACTED",
          "errors.retry.timeout": -1,
          "errors.retry.delay.max.ms": 300,
          "transforms": "outbox",
          "transforms.outbox.type": "io.debezium.transforms.outbox.EventRouter",
          "transforms.outbox.route.topic.replacement": "REDACTED.${routedByValue}",
          "value.converter": "io.debezium.converters.ByteBufferConverter",
          "transforms.outbox.table.fields.additional.placement": "REDACTED"
      }

      What is the captured database version and mode of depoyment?

      Both postgres and debezium (quay.io/debezium/connect) running in docker using docker-compose.

      What behaviour do you expect?

      The connector to retry when I either `docker-compose stop` or `docker-compose kill` the container

      What behaviour do you see?

      When I stop, the following error, which is currently not treated as a retryable error: `terminating connection due to administrator command`

       

      When I kill, the following error which is currently not treated as a retryable error: `terminating connection due to unexpected postmaster exit`

      Do you see the same behaviour using the latest relesead Debezium version?

      Yes - all versions currently do not treat these errors are retryable.

      Do you have the connector logs, ideally from start till finish?

      (You might be asked later to provide DEBUG/TRACE level log)

      Example:

       

      debezium_1                      | 2022-05-05 13:27:48,717 ERROR  ||  WorkerSourceTask{id=tms-event-router-7-0} Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually restarted   [org.apache.kafka.connect.runtime.WorkerTask]
      debezium_1                      | org.apache.kafka.connect.errors.ConnectException: An exception occurred in the change event producer. This connector will be stopped.
      debezium_1                      |     at io.debezium.pipeline.ErrorHandler.setProducerThrowable(ErrorHandler.java:50)
      debezium_1                      |     at io.debezium.connector.postgresql.PostgresStreamingChangeEventSource.execute(PostgresStreamingChangeEventSource.java:172)
      debezium_1                      |     at io.debezium.connector.postgresql.PostgresStreamingChangeEventSource.execute(PostgresStreamingChangeEventSource.java:41)
      debezium_1                      |     at io.debezium.pipeline.ChangeEventSourceCoordinator.streamEvents(ChangeEventSourceCoordinator.java:174)
      debezium_1                      |     at io.debezium.pipeline.ChangeEventSourceCoordinator.executeChangeEventSources(ChangeEventSourceCoordinator.java:141)
      debezium_1                      |     at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:109)
      debezium_1                      |     at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
      debezium_1                      |     at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
      debezium_1                      |     at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
      debezium_1                      |     at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
      debezium_1                      |     at java.base/java.lang.Thread.run(Thread.java:829)
      debezium_1                      | Caused by: org.postgresql.util.PSQLException: FATAL: terminating connection due to administrator command
      debezium_1                      |     at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2675)
      debezium_1                      |     at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2365)
      debezium_1                      |     at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:355)
      debezium_1                      |     at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:490)
      debezium_1                      |     at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:408)
      debezium_1                      |     at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:166)
      debezium_1                      |     at org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:118)
      debezium_1                      |     at io.debezium.jdbc.JdbcConnection.prepareQuery(JdbcConnection.java:703)
      debezium_1                      |     at io.debezium.connector.postgresql.PostgresStreamingChangeEventSource.probeConnectionIfNeeded(PostgresStreamingChangeEventSource.java:340)
      debezium_1                      |     at io.debezium.connector.postgresql.PostgresStreamingChangeEventSource.processMessages(PostgresStreamingChangeEventSource.java:284)
      debezium_1                      |     at io.debezium.connector.postgresql.PostgresStreamingChangeEventSource.execute(PostgresStreamingChangeEventSource.java:169)
      debezium_1                      |     ... 9 more

      How to reproduce the issue using our tutorial deployment?

      While running all containers, issue either `docker-compose stop postgres` or `docker-compose kill postgres`.

              Unassigned Unassigned
              mark-allanson Mark Allanson (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: