-
Bug
-
Resolution: Done
-
Major
-
2.6.2.Final, 2.7.0.Final
-
None
-
False
-
None
-
False
-
Critical
In order to make your issue reports as actionable as possible, please provide the following information, depending on the issue type.
Bug report
For bug reports, provide this information, please:
What Debezium connector do you use and what version?
Jdbc debezium sink connector v2.6.2.Final
What is the connector configuration?
```
{
"name": "mysql-sink-connector",
"connector.class": "io.debezium.connector.jdbc.JdbcSinkConnector",
"topics": "source_topic",
"key.converter": "io.confluent.connect.avro.AvroConverter",
"value.converter": "io.confluent.connect.avro.AvroConverter",
"key.converter.schema.registry.url": "http://schema-registry:8081",
"value.converter.schema.registry.url": "http://schema-registry:8081",
"key.converter.key.subject.name.strategy": "io.confluent.kafka.serializers.subject.TopicRecordNameStrategy",
"value.converter.value.subject.name.strategy": "io.confluent.kafka.serializers.subject.TopicRecordNameStrategy",
"schema.history.internal.kafka.bootstrap.servers": "kafka-1:9092",
"schema.history.internal.kafka.topic": "source_topic-history",
"table.name.format": "db_name.${source.table}",
"connection.url": "jdbc:mysql://mysql:3306/db_name",
"connection.username": "root",
"connection.password": "password",
"insert.mode": "upsert",
"primary.key.mode": "record_key",
"primary.key.fields": "Id",
"delete.enabled": "true",
"quote.identifiers": "true",
"errors.log.enable": true,
"errors.log.include.messages": true,
}
```
What is the captured database version and mode of deployment?
MySql 8.0, MySql 8.4
What behavior do you expect?
Context for source : All the tables cdc changes are rerouted from source to single kafka topic. The topic has correct ordering of the messages from the source.
When consuming batch of records from kafka topics, the insertion/deletion order for various tables' cdc changes should be same as that in kafka records order.
What behavior do you see?
Foreign Key Error as there is reordering of tables during insertion time.
If a parent table cdc messsage is earlier than child table cdc message.
If the batch includes both the tables, it is inserting the child table first causing the foreign key error.
This Foreign key error is unpredicatable and will depend on tableId's hashcode created by `updateBuffer` hashmap and `deleteBuffer` hashmap.
Do you see the same behaviour using the latest released Debezium version?
Possibly. Haven't tested it out with v3.
How to reproduce the issue using our tutorial deployment?
I couldn't find any tutorial jdbc sink connectors example.
Create two tables with name `OpenIddictApplications` and `OpenIddictAuthorizations` in SQL server. Add a foreign key relationship between both tables where parent is `OpenIddictApplications` and child is `OpenIddictAuthorizations`.
Ingest it into one topic using `ByLogicalRouter`.
Consume it via debezium jdbc sink connector into mysql.
Implementation ideas (optional)
Using `LinkedHashMap` instead of `HashMap` for `updateBuffer` and `deleteBuffer` to preserve the ordering of tables during insertion and deletion.
- is cloned by
-
DBZ-8272 Better support multi-topic writes on tables with foreign key constraints
- Open
- links to
-
RHEA-2024:139598 Red Hat build of Debezium 2.5.4 release