Details
-
Enhancement
-
Resolution: Done
-
Major
-
1.9.0.Final
-
None
-
False
-
None
-
False
Description
In order to make your issue reports as actionable as possible, please provide the following information, depending on the issue type.
Bug report
Should store event header timestamp instead of process time in HistoryRecord
What Debezium connector do you use and what version?
mysql connector
What is the connector configuration?
n/a
What is the captured database version and mode of depoyment?
aws rds
What behaviour do you expect?
<Your answer>
What behaviour do you see?
<Your answer>
Do you see the same behaviour using the latest relesead Debezium version?
(Ideally, also verify with latest Alpha/Beta/CR version)
<Your answer>
Do you have the connector logs, ideally from start till finish?
(You might be asked later to provide DEBUG/TRACE level log)
Error log for schema not found:
org.apache.kafka.connect.errors.ConnectException: An exception occurred in the change event producer. This connector will be stopped.\n\tat io.debezium.pipeline.ErrorHandler.setProducerThrowable(ErrorHandler.java:42)\n\tat io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleEvent(MySqlStreamingChangeEventSource.java:366)\n\tat io.debezium.connector.mysql.MySqlStreamingChangeEventSource.lambda$execute$25(MySqlStreamingChangeEventSource.java:857)\n\tat com.github.shyiko.mysql.binlog.BinaryLogClient.notifyEventListeners(BinaryLogClient.java:1132)\n\tat com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:974)\n\tat com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:599)\n\tat com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:857)\n\tat java.base/java.lang.Thread.run(Thread.java:830)\nCaused by: io.debezium.DebeziumException: Error processing binlog event\n\t... 7 more\nCaused by: io.debezium.DebeziumException: Encountered change event for table my_db.tb_order_offline_202107_001 whose schema isn't known to this connector\n\tat io.debezium.connector.mysql.MySqlStreamingChangeEventSource.informAboutUnknownTableIfRequired(MySqlStreamingChangeEventSource.java:652)\n\tat io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleUpdateTableMetadata(MySqlStreamingChangeEventSource.java:630)\n\tat io.debezium.connector.mysql.MySqlStreamingChangeEventSource.lambda$execute$13(MySqlStreamingChangeEventSource.java:828)\n\tat io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleEvent(MySqlStreamingChangeEventSource.java:349)\n\t... 6 more\n"
Some ddl statements from history topic:
{ "source": { "server": "my_db_cluster" }, "position": { "transaction_id": null, "ts_sec": 1643166973, "file": "mysql-bin-changelog.017832", "pos": 82903602, "server_id": 1664652013 }, "databaseName": "my_db", "ddl": "ANALYZE TABLE `my_db`.`_tb_order_offline_202107_new` /* pt-online-schema-change */", "tableChanges": []}
{ "source": { "server": "prod_shard_4_cluster" }, "position": { "transaction_id": null, "ts_sec": 1643214305, "file": "mysql-bin-changelog.017832", "pos": 82903602, "server_id": 1664652013 }, "databaseName": "my_db", "ddl": "ANALYZE TABLE `my_db`.`_tb_order_offline_202107_new` /* pt-online-schema-change */", "tableChanges": []}
Recently, we met one of our mysql connectors failed, becasue of the table schema isn't known to this connector. After investigation, we found our ops restart this connector and it re-processed the previous binlog position. After the restart, connector parsed following ddls again, and remove the table tb_order_offline_202107 from map store, caused the related table schema not found.
RENAME TABLE `my_db`.`tb_order_offline_202107` TO `my_db`.`_tb_order_offline_202107_old` RENAME TABLE `my_db`.`_tb_order_offline_202107_new` TO `my_db`.`tb_order_offline_202107`
To avoid the misunderstand, should use event header timestamp instead of process time in history record.
And we want to find a solution from debezium site to avoid processing duplicated ddl since the connect framework committed the offsets to the topic.
How to reproduce the issue using our tutorial deployment?
<Your answer>
Feature request or enhancement
For feature requests or enhancements, provide this information, please:
Which use case/requirement will be addressed by the proposed feature?
<Your answer>
Implementation ideas (optional)
<Your answer>
