Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-7743

The MySQL Debezium connector is not doing the snapshot after the reset.

XMLWordPrintable

    • False
    • None
    • False
    • Critical

      In order to make your issue reports as actionable as possible, please provide the following information, depending on the issue type.

      Bug report

      The MySQL Debezium connector is not doing the snapshot after the reset

      What Debezium connector do you use and what version?

      2.6

      What is the connector configuration?

      application.properties:


      debezium.sink.type=redis
      debezium.sink.redis.message.format=extended
      debezium.sink.redis.address=${RDI_REDIS_HOST}:${RDI_REDIS_PORT}
      debezium.sink.redis.user=${RDI_REDIS_USERNAME}
      debezium.sink.redis.password=${RDI_REDIS_PASSWORD}

      debezium.source.connector.class=io.debezium.connector.mysql.MySqlConnector
      debezium.source.database.server.id=1
      debezium.source.database.include.list=redislabscdc
      debezium.source.offset.storage=io.debezium.storage.redis.offset.RedisOffsetBackingStore
      debezium.source.topic.prefix=rdi

      debezium.source.database.hostname=${HOST_IP}
      debezium.source.database.port=13000
      debezium.source.database.user=${SOURCE_DB_USERNAME}
      debezium.source.database.password=${SOURCE_DB_PASSWORD}
      debezium.source.include.schema.changes=false
      debezium.source.skip.messages.without.change=true
      debezium.source.offset.flush.interval.ms=1000
      debezium.source.tombstones.on.delete=false
      debezium.source.schema.history.internal=io.debezium.storage.redis.history.RedisSchemaHistory
      debezium.source.schema.history.internal.store.only.captured.tables.ddl=true
      debezium.source.key.converter.schemas.enable=false
      debezium.source.value.converter.schemas.enable=false
      debezium.source.decimal.handling.mode=string

      debezium.transforms=AddPrefix
      debezium.transforms.AddPrefix.type=org.apache.kafka.connect.transforms.RegexRouter
      debezium.transforms.AddPrefix.regex=.*
      debezium.transforms.AddPrefix.replacement=data:$0

      quarkus.log.min-level=TRACE
      quarkus.log.level=INFO
      quarkus.log.console.json=false
      quarkus.http.port=8085

      What is the captured database version and mode of depoyment?

       

      What behaviour do you expect?

      Snapshot should be performed after reset

      What behaviour do you see?

      In 2.6

      2024-04-04 15:19:12,333 WARN [io.deb.rel.RelationalSnapshotChangeEventSource] (debezium-mysqlconnector-rdi-change-event-source-coordinator) For table 'redislabscdc.emp' the select statement was not provided, skipping table

       

      In 2.5

      2024-04-04 16:47:08,997 INFO [io.deb.rel.RelationalSnapshotChangeEventSource] (debezium-mysqlconnector-rdi-change-event-source-coordinator) For table 'redislabscdc.emp' using select statement: 'SELECT `empno`, `fname`, `lname` FROM `redislabscdc`.`emp`'

      Do you see the same behaviour using the latest relesead Debezium version?

      Yes

      Do you have the connector logs, ideally from start till finish?

      (You might be asked later to provide DEBUG/TRACE level log)

      ... 2024-04-04 15:19:11,442 INFO [io.deb.con.mys.MySqlConnectorTask] (pool-10-thread-1) Reconnecting after finishing schema recovery 2024-04-04 15:19:11,466 INFO [io.deb.con.mys.MySqlConnectorTask] (pool-10-thread-1) No previous offset found ... 2024-04-04 15:19:11,535 INFO [io.deb.rel.RelationalSnapshotChangeEventSource] (debezium-mysqlconnector-rdi-change-event-source-coordinator) According to the connector configuration both schema and data will be snapshot. 2024-04-04 15:19:11,539 INFO [io.deb.rel.RelationalSnapshotChangeEventSource] (debezium-mysqlconnector-rdi-change-event-source-coordinator) Snapshot step 1 - Preparing 2024-04-04 15:19:11,541 INFO [io.deb.rel.RelationalSnapshotChangeEventSource] (debezium-mysqlconnector-rdi-change-event-source-coordinator) Snapshot step 2 - Determining captured tables 2024-04-04 15:19:11,541 INFO [io.deb.con.mys.MySqlSnapshotChangeEventSource] (debezium-mysqlconnector-rdi-change-event-source-coordinator) Read list of available databases 2024-04-04 15:19:11,544 INFO [io.deb.con.mys.MySqlSnapshotChangeEventSource] (debezium-mysqlconnector-rdi-change-event-source-coordinator) list of available databases is: [information_schema, mysql, performance_schema, redislabscdc, sys] 2024-04-04 15:19:11,545 INFO [io.deb.con.mys.MySqlSnapshotChangeEventSource] (debezium-mysqlconnector-rdi-change-event-source-coordinator) Read list of available tables in each database 2024-04-04 15:19:11,565 INFO [io.deb.con.mys.MySqlSnapshotChangeEventSource] (debezium-mysqlconnector-rdi-change-event-source-coordinator) snapshot continuing with database(s): [redislabscdc] 2024-04-04 15:19:11,566 INFO [io.deb.rel.RelationalSnapshotChangeEventSource] (debezium-mysqlconnector-rdi-change-event-source-coordinator) Adding table redislabscdc.emp to the list of capture schema tables 2024-04-04 15:19:11,570 INFO [io.deb.rel.RelationalSnapshotChangeEventSource] (debezium-mysqlconnector-rdi-change-event-source-coordinator) Created connection pool with 1 threads 2024-04-04 15:19:11,571 INFO [io.deb.rel.RelationalSnapshotChangeEventSource] (debezium-mysqlconnector-rdi-change-event-source-coordinator) Snapshot step 3 - Locking captured tables [redislabscdc.emp] 2024-04-04 15:19:11,578 INFO [io.deb.con.mys.MySqlSnapshotChangeEventSource] (debezium-mysqlconnector-rdi-change-event-source-coordinator) Flush and obtain global read lock to prevent writes to database 2024-04-04 15:19:11,581 INFO [io.deb.rel.RelationalSnapshotChangeEventSource] (debezium-mysqlconnector-rdi-change-event-source-coordinator) Snapshot step 4 - Determining snapshot offset 2024-04-04 15:19:11,590 INFO [io.deb.con.mys.str.mys.MySqlConnectorAdapter] (debezium-mysqlconnector-rdi-change-event-source-coordinator) Read binlog position of MySQL primary server 2024-04-04 15:19:11,595 INFO [io.deb.con.mys.str.mys.MySqlConnectorAdapter] (debezium-mysqlconnector-rdi-change-event-source-coordinator) using binlog 'binlog.000002' at position '1251' and gtid '' 2024-04-04 15:19:11,595 INFO [io.deb.rel.RelationalSnapshotChangeEventSource] (debezium-mysqlconnector-rdi-change-event-source-coordinator) Snapshot step 5 - Reading structure of captured tables 2024-04-04 15:19:11,595 INFO [io.deb.con.mys.MySqlSnapshotChangeEventSource] (debezium-mysqlconnector-rdi-change-event-source-coordinator) Only captured tables schema should be captured, capturing: [redislabscdc.emp] 2024-04-04 15:19:12,222 INFO [io.deb.con.mys.MySqlSnapshotChangeEventSource] (debezium-mysqlconnector-rdi-change-event-source-coordinator) Reading structure of database 'redislabscdc' 2024-04-04 15:19:12,299 INFO [io.deb.rel.RelationalSnapshotChangeEventSource] (debezium-mysqlconnector-rdi-change-event-source-coordinator) Snapshot step 6 - Persisting schema history 2024-04-04 15:19:12,329 INFO [io.deb.con.mys.MySqlSnapshotChangeEventSource] (debezium-mysqlconnector-rdi-change-event-source-coordinator) Releasing global read lock to enable MySQL writes 2024-04-04 15:19:12,331 INFO [io.deb.con.mys.MySqlSnapshotChangeEventSource] (debezium-mysqlconnector-rdi-change-event-source-coordinator) Writes to MySQL tables prevented for a total of 00:00:00.75 2024-04-04 15:19:12,331 INFO [io.deb.rel.RelationalSnapshotChangeEventSource] (debezium-mysqlconnector-rdi-change-event-source-coordinator) Snapshot step 7 - Snapshotting data 2024-04-04 15:19:12,332 INFO [io.deb.rel.RelationalSnapshotChangeEventSource] (debezium-mysqlconnector-rdi-change-event-source-coordinator) Creating snapshot worker pool with 1 worker thread(s) 2024-04-04 15:19:12,333 WARN [io.deb.rel.RelationalSnapshotChangeEventSource] (debezium-mysqlconnector-rdi-change-event-source-coordinator) For table 'redislabscdc.emp' the select statement was not provided, skipping table 2024-04-04 15:19:12,336 INFO [io.deb.pip.sou.AbstractSnapshotChangeEventSource] (debezium-mysqlconnector-rdi-change-event-source-coordinator) Snapshot - Final stage 2024-04-04 15:19:12,336 INFO [io.deb.pip.sou.AbstractSnapshotChangeEventSource] (debezium-mysqlconnector-rdi-change-event-source-coordinator) Snapshot completed 2024-04-04 15:19:12,374 INFO [io.deb.pip.ChangeEventSourceCoordinator] (debezium-mysqlconnector-rdi-change-event-source-coordinator) Snapshot ended with SnapshotResult [status=COMPLETED, offset=MySqlOffsetContext [sourceInfoSchema=Schema{io.debezium.connector.mysql.Source:STRUCT}, sourceInfo=SourceInfo [currentGtid=null, currentBinlogFilename=binlog.000002, currentBinlogPosition=1251, currentRowNumber=0, serverId=0, sourceTime=2024-04-04T15:19:12.315Z, threadId=-1, currentQuery=null, tableIds=[redislabscdc.emp], databaseName=redislabscdc], snapshotCompleted=true, transactionContext=TransactionContext [currentTransactionId=null, perTableEventCount={}, totalEventCount=0], restartGtidSet=null, currentGtidSet=null, restartBinlogFilename=binlog.000002, restartBinlogPosition=1251, restartRowsToSkip=0, restartEventsToSkip=0, currentEventLengthInBytes=0, inTransaction=false, transactionId=null, incrementalSnapshotContext =IncrementalSnapshotContext [windowOpened=false, chunkEndPosition=null, dataCollectionsToSnapshot=[], lastEventKeySent=null, maximumKey=null]]] ... <Your answer>

      How to reproduce the issue using our tutorial deployment?

      <Your answer>

      Feature request or enhancement

      For feature requests or enhancements, provide this information, please:

      Which use case/requirement will be addressed by the proposed feature?

      <Your answer>

      Implementation ideas (optional)

      <Your answer>

              Unassigned Unassigned
              sergey.eizner Sergey Eizner (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Created:
                Updated:
                Resolved: