Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-7617

Incremental snapshot query doesn't honor message.key.columns order


      What Debezium connector do you use and what version?

      MySQL 2.4.0

      What is the connector configuration?


          include.schema.changes: true
          incremental.snapshot.chunk.size: 10240
          max.batch.size: 40960
          max.queue.size: 81920
          message.key.columns: redacted.my_problem_table:customer_id,field_id
          offset.flush.timeout.ms: 60000
          offset.flush.interval.ms: 10000
          read.only: true
          signal.consumer.sasl.jaas.config: "redacted"
          signal.consumer.sasl.mechanism: SCRAM-SHA-512
          signal.consumer.security.protocol: SASL_SSL
          signal.data.collection: redacted.debezium_signal
          signal.kafka.bootstrap.servers: strimzi-kafka-bootstrap.staging.redacted.com:9096
          signal.kafka.topic: debezium.sdlc-redacted-pxc.redacted.debezium_signal
          topic.creation.groups: my_problem_table
          topic.creation.my_problem_table.include: debezium.sdlc-redacted-pxc.redacted.my_problem_table
          topic.creation.my_problem_table.delete.retention.ms: 604800000 # 1 week deletion retention on tombstones
          topic.creation.my_problem_table.cleanup.policy: compact
          topic.creation.my_problem_table.partitions: 36
          topic.creation.my_problem_table.unclean.leader.election.enable: false


      What is the captured database version and mode of depoyment?

      MySQL 8 Percona XtraDB

      What behaviour do you expect?

      Ordering of the initial `SELECT` query should respect the ordering of `message.key-columns`.

      What behaviour do you see?

      The initial `SELECT` query has the keys in the opposite order.

      Do you see the same behaviour using the latest relesead Debezium version?

      2.4.0 is the latest version I've tried.

      Do you have the connector logs, ideally from start till finish?

      The logs here don't apply. The query I see in MySQL `SHOW FULL PROCESSLIST` looks like this:


      SELECT * FROM redacted.my_problem_table ORDER BY field_id DESC, customer_id DESC LIMIT 1


      Notice how the columns are not in the same order as the `message.key.columns configuration.

      How to reproduce the issue using our tutorial deployment?

      I assume you can set any two columns and set them in `message.key.columns` and then observe the behavior of the initial `LIMIT 1` query the connector uses to retrieve the latest PK of the table prior to taking the snapshot.

            rh-ee-mvitale Mario Fiore Vitale
            jonslusher Jon Slusher
            0 Vote for this issue
            5 Start watching this issue