Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-7617

Incremental snapshot query doesn't honor message.key.columns order

XMLWordPrintable

      What Debezium connector do you use and what version?

      MySQL 2.4.0

      What is the connector configuration?

      ```

      ...
          include.schema.changes: true
          incremental.snapshot.chunk.size: 10240
      ...
          max.batch.size: 40960
          max.queue.size: 81920
          message.key.columns: redacted.my_problem_table:customer_id,field_id
          offset.flush.timeout.ms: 60000
          offset.flush.interval.ms: 10000
          read.only: true
      ...
          signal.consumer.sasl.jaas.config: "redacted"
          signal.consumer.sasl.mechanism: SCRAM-SHA-512
          signal.consumer.security.protocol: SASL_SSL
          signal.data.collection: redacted.debezium_signal
          signal.kafka.bootstrap.servers: strimzi-kafka-bootstrap.staging.redacted.com:9096
          signal.kafka.topic: debezium.sdlc-redacted-pxc.redacted.debezium_signal
      ...
          topic.creation.groups: my_problem_table
          topic.creation.my_problem_table.include: debezium.sdlc-redacted-pxc.redacted.my_problem_table
          topic.creation.my_problem_table.delete.retention.ms: 604800000 # 1 week deletion retention on tombstones
          topic.creation.my_problem_table.cleanup.policy: compact
          topic.creation.my_problem_table.partitions: 36
          topic.creation.my_problem_table.unclean.leader.election.enable: false

      ```

      What is the captured database version and mode of depoyment?

      MySQL 8 Percona XtraDB

      What behaviour do you expect?

      Ordering of the initial `SELECT` query should respect the ordering of `message.key-columns`.

      What behaviour do you see?

      The initial `SELECT` query has the keys in the opposite order.

      Do you see the same behaviour using the latest relesead Debezium version?

      2.4.0 is the latest version I've tried.

      Do you have the connector logs, ideally from start till finish?

      The logs here don't apply. The query I see in MySQL `SHOW FULL PROCESSLIST` looks like this:

      ```

      SELECT * FROM redacted.my_problem_table ORDER BY field_id DESC, customer_id DESC LIMIT 1

      ```

      Notice how the columns are not in the same order as the `message.key.columns configuration.

      How to reproduce the issue using our tutorial deployment?

      I assume you can set any two columns and set them in `message.key.columns` and then observe the behavior of the initial `LIMIT 1` query the connector uses to retrieve the latest PK of the table prior to taking the snapshot.

              rh-ee-mvitale Mario Fiore Vitale
              jonslusher Jon Slusher (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated:
                Resolved: