Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-8130

Unpredicatable ordering of table rows during insertion causing foreign key error

XMLWordPrintable

    • False
    • None
    • False
    • Critical

      In order to make your issue reports as actionable as possible, please provide the following information, depending on the issue type.

      Bug report

      For bug reports, provide this information, please:

      What Debezium connector do you use and what version?

      Jdbc debezium sink connector v2.6.2.Final

      What is the connector configuration?

      ```

      {
        "name": "mysql-sink-connector",
        "connector.class": "io.debezium.connector.jdbc.JdbcSinkConnector",
        "topics": "source_topic",
        "key.converter": "io.confluent.connect.avro.AvroConverter",
        "value.converter": "io.confluent.connect.avro.AvroConverter",
        "key.converter.schema.registry.url": "http://schema-registry:8081",
        "value.converter.schema.registry.url": "http://schema-registry:8081",
        "key.converter.key.subject.name.strategy": "io.confluent.kafka.serializers.subject.TopicRecordNameStrategy",
        "value.converter.value.subject.name.strategy": "io.confluent.kafka.serializers.subject.TopicRecordNameStrategy",
        "schema.history.internal.kafka.bootstrap.servers": "kafka-1:9092",
        "schema.history.internal.kafka.topic": "source_topic-history",
        "table.name.format": "db_name.${source.table}",
        "connection.url": "jdbc:mysql://mysql:3306/db_name",
        "connection.username": "root",
        "connection.password": "password",
        "insert.mode": "upsert",
        "primary.key.mode": "record_key",
        "primary.key.fields": "Id",
        "delete.enabled": "true",
        "quote.identifiers": "true",
        "errors.log.enable": true,
        "errors.log.include.messages": true,
      }

      ```

      What is the captured database version and mode of deployment?

      MySql 8.0, MySql 8.4

      What behavior do you expect?

      Context for source : All the tables cdc changes are rerouted from source to single kafka topic. The topic has correct ordering of the messages from the source. 

      When consuming batch of records from kafka topics, the insertion/deletion order for various tables' cdc changes should be same as that in kafka records order. 

      What behavior do you see?

      Foreign Key Error as there is reordering of tables during insertion time.
      If a parent table cdc messsage is earlier than child table cdc message.
      If the batch includes both the tables, it is inserting the child table first causing the foreign key error.
      This Foreign key error is unpredicatable and will depend on tableId's hashcode created by `updateBuffer` hashmap and `deleteBuffer` hashmap.

      Do you see the same behaviour using the latest released Debezium version?

      Possibly. Haven't tested it out with v3.

      How to reproduce the issue using our tutorial deployment?

      I couldn't find any tutorial jdbc sink connectors example.

      Create two tables with name `OpenIddictApplications` and `OpenIddictAuthorizations` in SQL server. Add a foreign key relationship between both tables where parent is `OpenIddictApplications` and child is `OpenIddictAuthorizations`.

      Ingest it into one topic using `ByLogicalRouter`.

      Consume it via debezium jdbc sink connector into mysql. 

      Implementation ideas (optional)

      Using `LinkedHashMap` instead of `HashMap` for `updateBuffer` and `deleteBuffer` to preserve the ordering of tables during insertion and deletion.

            Unassigned Unassigned
            ashishbinu90@gmail.com Ashish Binu (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: