Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-7883

Unable to recover from failed incremental snapshot attempt

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • 3.0.1.Final
    • 2.5.3.Final
    • postgresql-connector
    • None
    • False
    • None
    • False
    • Important

      In order to make your issue reports as actionable as possible, please provide the following information, depending on the issue type.

      Bug report

      For bug reports, provide this information, please:

      What Debezium connector do you use and what version?

      Postgres Debezium Connector v2.5.3

      What is the connector configuration?

         "signal.data.collection":"public.debezium_signal",

       "connector.class":"io.debezium.connector.postgresql.PostgresConnector",

         "slot.name":"<SLOT>",

         "publication.name":"<PUBLICATION>",

         "producer.override.security.protocol":"SASL_SSL",

         "tombstones.on.delete":"false",

         "topic.prefix":"cerc",

         "decimal.handling.mode":"double",

         "value.converter":"io.confluent.connect.avro.AvroConverter",

         "key.converter":"io.confluent.connect.avro.AvroConverter",

         "database.dbname":"<DBNAME>",

         "database.user":"<USER>",

         "time.precision.mode":"adaptive_time_microseconds",

         "database.port":"5432",

         "plugin.name":"pgoutput",

         "key.converter.schemas.enable":"false",

         "value.converter.schema.registry.url":"<SCHEMA-REGISTRY-URL>",

         "database.hostname":"<DBHOST>",

         "database.password":"<DBPASSWORD>",

         "schema.name.adjustment.mode":"avro",

         "name":"<NAME>",

         "value.converter.schemas.enable":"false",

         "producer.override.sasl.jaas.config":"<PATH-TO-SECRET>",

         "producer.override.sasl.mechanism":"PLAIN",

         "table.include.list":"<TABLES>..,public.debezium_signal",

         "key.converter.schema.registry.url":"<SCHEMA-REGISTRY-URL>",

         "snapshot.mode":"never",

         "field.name.adjustment.mode":"avro"

      What is the captured database version and mode of depoyment?

      AWS Aurora RDS

      What behaviour do you expect?

      For the "stop-snapshot" message (via inserting a row to the signaling table) to result in stopping the attempts to snapshot the table. In addition, to be able to configure the number of retries, or have some sensible number instead of unlimited.

      What behaviour do you see?

      Sending the "stop-snapshot" message not having any effect, and the task keeps failing and retrying because it is retrying indefinitely, so there was no way out. We ended up having to re-create the connector under a different name to circumvent the issue.

      Do you see the same behaviour using the latest relesead Debezium version?

      Didn't try.

      Do you have the connector logs, ideally from start till finish?

      WARN [<CONNECTOR>|task-0] Retry 2 of unlimited retries will be attempted (io.debezium.pipeline.ErrorHandler:125)
          ... 13 more
          at io.debezium.pipeline.source.snapshot.incremental.AbstractIncrementalSnapshotChangeEventSource.createDataEventsForTable(AbstractIncrementalSnapshotChangeEventSource.java:638)
          at org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:134)
          at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:190)
          at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:415)
          at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:498)
          at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:352)
          at org.postgresql.core.v3.SimpleParameterList.checkAllParametersSet(SimpleParameterList.java:339)
      Caused by: org.postgresql.util.PSQLException: No value specified for parameter 3.
          ... 12 more
          at io.debezium.pipeline.source.snapshot.incremental.AbstractIncrementalSnapshotChangeEventSource.readChunk(AbstractIncrementalSnapshotChangeEventSource.java:385)
          at io.debezium.pipeline.source.snapshot.incremental.AbstractIncrementalSnapshotChangeEventSource.createDataEventsForTable(AbstractIncrementalSnapshotChangeEventSource.java:682)
      Caused by: io.debezium.DebeziumException: Snapshotting of table public.<TABLE> failed
          at java.base/java.lang.Thread.run(Thread.java:829)
          at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
          at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
          at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
          at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
          at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:137)
          at io.debezium.pipeline.ChangeEventSourceCoordinator.executeChangeEventSources(ChangeEventSourceCoordinator.java:194)
          at io.debezium.pipeline.ChangeEventSourceCoordinator.streamEvents(ChangeEventSourceCoordinator.java:267)
          at io.debezium.pipeline.ChangeEventSourceCoordinator.initStreamEvents(ChangeEventSourceCoordinator.java:287)
          at java.base/java.util.Optional.ifPresent(Optional.java:183)
          at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$initStreamEvents$9(ChangeEventSourceCoordinator.java:287)
          at io.debezium.pipeline.source.snapshot.incremental.AbstractIncrementalSnapshotChangeEventSource.init(AbstractIncrementalSnapshotChangeEventSource.java:317)
          at io.debezium.pipeline.source.snapshot.incremental.AbstractIncrementalSnapshotChangeEventSource.readChunk(AbstractIncrementalSnapshotChangeEventSource.java:411)
      io.debezium.DebeziumException: Database error while executing incremental snapshot for table 'DataCollection{id=public.<TABLE>, additionalCondition=, surrogateKey=}'

      How to reproduce the issue using our tutorial deployment?

      I think what caused the error is that I included the db name in the value for the data-collections, i.e.
      {{INSERT INTO public.debezium_signal (id, type, data) VALUES (gen_random_uuid () , 'execute-snapshot', '

      {"data-collections": ["<DB>.<SCHEMA>.<TABLE>"], "type":"incremental"}

      ');}}

      Feature request or enhancement

      For feature requests or enhancements, provide this information, please:

      Which use case/requirement will be addressed by the proposed feature?

      <Your answer>

      Implementation ideas (optional)

      <Your answer>

            Unassigned Unassigned
            shauli10 Shaul Lahav (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: