Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-9428

Avoid storing irrelevant DDL statement "REPLACE INTO" in schema history topic 

XMLWordPrintable

    • False
    • Hide

      None

      Show
      None
    • False

      We’re using pt-table-checksum for replica consistency checks. It relies on the STATEMENT binlog format, but Debezium’s MySQL source connector interprets these queries as DML.
      Example:

      REPLACE INTO internal.checksums_kafka_testing SELECT test statement based AS crc FROM user WHERE ((id >= 2611069)) AND ((id <= 2771980))

      mysql connector version: 3.2.1 Final
      config:
      "topic.prefix": "cdc_mysql",
      "schema.history.internal.kafka.topic": "topic_schema_history",
      "database.server.name": "debezium_cdc_mysql",
      "database.include.list": "user",
      "connector.class": "io.debezium.connector.mysql.MySqlConnector",
      "database.allowPublicKeyRetrieval": "true",
      "transforms": "unwrap",
      "transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState",
      "transforms.unwrap.add.fields": "op,table,source.ts_ms",
      "transforms.unwrap.delete.tombstone.handling.mode": "rewrite-with-tombstone",
      "database.hostname": "<hostname>",
      "database.user": "<user_name>",
      "database.password": "<password>",
      "database.port": "3306",
      "output.data.format": "AVRO",
      "offsets.storage.topic": "_debezium_offsets",
      "tasks.max": "1",
      "internal.database.history.ddl.filter": "REPLACE INTO*",
      "schema.history.internal.store.only.captured.databases.ddl": "true",
      "schema.history.internal.skip.unparseable.ddl": "true",
      "key.converter": "io.confluent.connect.avro.AvroConverter",
      "value.converter": "io.confluent.connect.avro.AvroConverter",
      "database.connectionTimeZone": "America/Los_Angeles",
      "snapshot.mode": "recovery",
      "snapshot.locking.mode": "none"

              Unassigned Unassigned
              anton.voroniuk Anton Voroniuk (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated: