Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-6851

Allow read-only incremental snapshot when snapshot.mode is initial_only

XMLWordPrintable

    • False
    • Hide

      None

      Show
      None
    • False

      What Debezium connector do you use and what version?

      MySQL connector v2.3.2.Final

      What is the connector configuration?

       

      {
          "name": "inventory-connector",
          "config": {
              "connector.class": "io.debezium.connector.mysql.MySqlConnector",
              "tasks.max": "1",
              "database.hostname": "mysql",
              "database.port": "3306",
              "database.user": "debezium",
              "database.password": "dbz",
              "database.server.id": "184054",
              "snapshot.mode": "initial_only",
              "topic.prefix": "dbserver1",
              "database.include.list": "inventory",
              "schema.history.internal.kafka.bootstrap.servers": "kafka:9092",
              "signal.kafka.bootstrap.servers": "kafka:9092",
              "schema.history.internal.kafka.topic": "schema-changes.inventory",
              "signal.kafka.topic": "debezium_signal",
              "signal.enabled.channels": "source,kafka",
              "read.only": "true"
          }
      } 

      Change snapshot.mode to default (initial) to observe incremental snapshot does work with this setup

       

      What is the captured database version and mode of depoyment?

      docker deployment provided in the tutorial deployment: https://debezium.io/documentation/reference/2.3/tutorial.html

      What behaviour do you expect?

      Incremental snapshot should work with initial_only snapshot mode

      What behaviour do you see?

      Incremental request is successful but no actual incremental snapshot is done

      Do you have the connector logs, ideally from start till finish?

      Nothing related to incremental snapshot after the following logs, full log is attached as well logs.txt:

       

      2023-08-30 15:57:51,781 INFO   ||  Requested 'INCREMENTAL' snapshot of data collections '[inventory.customers]' with additional condition 'No condition passed' and surrogate key 'PK of table will be used'   [io.debezium.pipeline.signal.actions.snapshotting.ExecuteSnapshot]
      2023-08-30 15:57:51,790 INFO   ||  Incremental snapshot for table 'inventory.customers' will end at position [1004]   [io.debezium.pipeline.source.snapshot.incremental.AbstractIncrementalSnapshotChangeEventSource]

      How to reproduce the issue using our tutorial deployment?

      Mostly follow the steps provided, some changes needed to reproduce include:

      Enable GTID for read only incremental snapshot:

      docker run -it --rm --name mysql -p 3306:3306 -e MYSQL_ROOT_PASSWORD=debezium -e MYSQL_USER=mysqluser -e MYSQL_PASSWORD=mysqlpw quay.io/debezium/example-mysql:2.3 --gtid-mode=ON --enforce-gtid-consistency=ON

       

      Trigger incremental snapshot with kafka-console-producer

      kafka-console-producer.sh --topic debezium_signal --bootstrap-server host.docker.internal:9092 --property parse.key=true --property key.separator=":"
      
      >dbserver1:{"type":"execute-snapshot","data": {"data-collections": ["inventory.customers"], "type": "INCREMENTAL"}} 

       

       

              Unassigned Unassigned
              meng.zhang Meng Zhang (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: