-
Bug
-
Resolution: Done
-
Major
-
None
-
None
-
None
-
False
-
None
-
False
Hello,
I am currently performing CDC (Change Data Capture) operations from an Oracle database to a PostgreSQL database using Debezium. However, I am facing an issue and need your assistance.
Problem Summary:
I am experiencing data loss while extracting data from the database through Debezium. I have examined the configurations, but I am having difficulty understanding the root cause of the issue.
My Settings:
{
"name": "prod_source_mbs_fatura-20240111",
"config": {
"database.hostname": "*******",
"database.port": "****",
"database.user": "*****",
"database.password": "****",
"database.history.kafka.bootstrap.servers": "*****",
"schema.history.internal.kafka.bootstrap.servers": "******",
"schema.history.internal.consumer.sasl.jaas.config": "org.apache.kafka.common.security.plain.PlainLoginModule required username=\\"******\\" password=\\"*****\\";",
"schema.history.internal.producer.sasl.jaas.config": "org.apache.kafka.common.security.plain.PlainLoginModule required username=\\"*****\\" password=\\"*****\\";", "connector.class": "io.debezium.connector.oracle.OracleConnector",
"snapshot.mode": "schema_only",
"topic.prefix": "oracle-mbs-fatura",
"topic.creation.default.replication.factor": "3",
"topic.creation.default.partitions": "7", "database.dbname": "DICLE",
"table.include.list": "DICLE.FATURA",
"schema.include.list": "DICLE",
"database.connection.adapter": "logminer",
"schema.history.internal.kafka.topic": "schema-mbs-fatura",
"schema.history.internal.producer.sasl.mechanism": "PLAIN",
"schema.history.internal.producer.security.protocol": "SASL_PLAINTEXT",
"schema.history.internal.consumer.sasl.mechanism": "PLAIN",
"schema.history.internal.consumer.security.protocol": "SASL_PLAINTEXT",
"schema.history.internal.store.only.captured.tables.ddl": "true", "errors.log.enable": "true",
"decimal.handling.mode": "double", "log.mining.strategy": "online_catalog", "transforms": "topicCase,route,changeCaseKey",
"transforms.topicCase.type": "com.github.jcustenborder.kafka.connect.transform.common.ChangeTopicCase",
"transforms.topicCase.from": "UPPER_UNDERSCORE",
"transforms.topicCase.to": "LOWER_UNDERSCORE", "transforms.route.type": "org.apache.kafka.connect.transforms.RegexRouter",
"transforms.route.regex": "([^.]+)\\\\.([^.]+)\\\\.([^.]+)",
"transforms.route.replacement": "$3", "transforms.changeCaseKey.type": "com.github.jcustenborder.kafka.connect.transform.common.ChangeCase$Key",
"transforms.changeCaseKey.from": "UPPER_UNDERSCORE",
"transforms.changeCaseKey.to": "LOWER_UNDERSCORE"
}
}
I suspect that I am missing some changes when reading from the log file. Currently, the message retention period is 24 hours, and the lag status is not visible. Even when there is no record in the Oracle table, the data counts are not equal. There is a constant data gap, and I cannot be sure that the tables are equivalent one-to-one. The missed changes could include update and delete operations.
I am using Debezium version 2.3, and Oracle version is 19c.
1. What steps should I follow to resolve this issue?
2. Could you please check if there are any missing or incorrect settings in my configuration?
3. Do you have any recommendations?
I kindly request your assistance in resolving this issue. Thank you in advance.
Best regards,
Seda
- duplicates
-
-
- Closed
-
