XML

Word

Printable

Details

Type: Bug
Resolution: Unresolved
Priority: Major
Fix Version/s: under-triaging
Affects Version/s: 2.2.1.Final, 2.4.1.Final
Component/s: debezium-connector-mysql, mysql-connector
Labels:
None

Blocked:
False
Blocked Reason:
None
Ready:
False

Severity:
Moderate

SFDC Cases Counter:
SFDC Cases Links:

Description

Bug report

What Debezium connector do you use and what version?

I checked the above issue on 2.2.1Final & 2.4.1Final

What is the connector configuration?

relevant configuration
database.include.list: <Database_1>
tables.include.list: <Database_1>.<table_1>...<Database_1>.<table_n> inconsistent.schema.handling.mode: warn
event.processing.failure.handling.mode: warn
snapshot.mode: schema_only
schema.history.internal.store.only.captured.tables.ddl: true
schema.history.internal.store.only.captured.databases.ddl: true

What is the captured database version and mode of depoyment?

AWS MySQL RDS v8.0.33

What behaviour do you expect?

The prefer behavior is to be able to control if error will be ignore\warn\fail by event.processing.failure.handling.mode flag or other dedicated flag.

What behaviour do you see?

We had a strange situation when a developer performed an migration which included badly written 'CREATE TABLE IF NO EXIST' statement:

CREATE TABLE IF NOT EXISTS `database_2`.padding_overrides (ActivityName varchar(255), PaddingMode varchar(255), LeftPadding INTEGER, TopPadding INTEGER, RightPadding INTEGER, BottomPadding INTEGER, PRIMARY KEY (Id))'

This change was done for untracked table in an untracked database.

This table caused the connector to fail with (Where the connectors of the tracked table kept running as not :

Error processing binlog event, and propagating to Kafka Connect so it stops this connector. Future binlog events read before connector is shutdown will be ignored. (io.debezium.connector.mysql.MySqlStreamingChangeEventSource)


io.debezium.DebeziumException: Error processing binlog event
..
Caused by: java.lang.IllegalArgumentException: The column "Id" is referenced as PRIMARY KEY, but a matching column is not defined in table "detection_service.padding_overrides"!

Several issues here:

Debezium processing DDL of untracked tables. it will not fail connectors which tracking this table.
Debezium offer no way to bypass this somehow generic java.lang.IllegalArgumentException which are under the parsing step. better to have a dedicated exception for all non valid DDL queries and have the ability to treat it as a warning.
I'll be glad to have mysql connector an option to skip binlog pos. but we will leave it to a different 'feature request'.

Do you see the same behaviour using the latest relesead Debezium version?

yes.

Do you have the connector logs, ideally from start till finish?

2023-12-04 20:19:03,174 ERROR [ocr-main-streams-cdc-connector|task-0] Error during binlog processing. Last offset stored = {transaction_id=null, file=mysql-bin-changelog.057086, pos=242959, server_id=1436115291, event=1}, binlog reader near position = mysql-bin-changelog.057086/242959 (io.debezium.connector.mysql.MySqlStreamingChangeEventSource) [blc-lsports-ocr-replica.clr89j2kvsx8.eu-west-1.rds.amazonaws.com:3306]
2023-12-04 20:19:03,174 ERROR [ocr-main-streams-cdc-connector|task-0] Producer failure (io.debezium.pipeline.ErrorHandler) [blc-lsports-ocr-replica.clr89j2kvsx8.eu-west-1.rds.amazonaws.com:3306]
io.debezium.DebeziumException: Error processing binlog event
    at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleEvent(MySqlStreamingChangeEventSource.java:395)
    at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.lambda$execute$27(MySqlStreamingChangeEventSource.java:930)
    at com.github.shyiko.mysql.binlog.BinaryLogClient.notifyEventListeners(BinaryLogClient.java:1246)
    at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:1072)
    at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:631)
    at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:932)
    at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.lang.IllegalArgumentException: The column "Id" is referenced as PRIMARY KEY, but a matching column is not defined in table "detection_service.padding_overrides"!
    at io.debezium.relational.TableEditorImpl.lambda$updatePrimaryKeys$0(TableEditorImpl.java:107)
    at java.base/java.util.ArrayList.removeIf(ArrayList.java:1672)
    at java.base/java.util.ArrayList.removeIf(ArrayList.java:1660)
    at io.debezium.relational.TableEditorImpl.updatePrimaryKeys(TableEditorImpl.java:103)
    at io.debezium.relational.TableEditorImpl.create(TableEditorImpl.java:307)
    at io.debezium.connector.mysql.antlr.listener.CreateTableParserListener.lambda$exitColumnCreateTable$1(CreateTableParserListener.java:50)
    at io.debezium.connector.mysql.antlr.MySqlAntlrDdlParser.runIfNotNull(MySqlAntlrDdlParser.java:410)
    at io.debezium.connector.mysql.antlr.listener.CreateTableParserListener.exitColumnCreateTable(CreateTableParserListener.java:42)
    at io.debezium.ddl.parser.mysql.generated.MySqlParser$ColumnCreateTableContext.exitRule(MySqlParser.java:5339)
    at io.debezium.antlr.ProxyParseTreeListenerUtil.delegateExitRule(ProxyParseTreeListenerUtil.java:64)
    at io.debezium.connector.mysql.antlr.listener.MySqlAntlrDdlParserListener.exitEveryRule(MySqlAntlrDdlParserListener.java:106)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.exitRule(ParseTreeWalker.java:63)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:38)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:36)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:36)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:36)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:36)
    at io.debezium.antlr.AntlrDdlParser.parse(AntlrDdlParser.java:83)
    at io.debezium.connector.mysql.MySqlDatabaseSchema.parseDdl(MySqlDatabaseSchema.java:213)
    at io.debezium.connector.mysql.MySqlDatabaseSchema.parseStreamingDdl(MySqlDatabaseSchema.java:199)
    at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleQueryEvent(MySqlStreamingChangeEventSource.java:593)
    at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.lambda$execute$15(MySqlStreamingChangeEventSource.java:901)
    at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleEvent(MySqlStreamingChangeEventSource.java:375)
    ... 6 more
2023-12-04 20:19:03,175 INFO [ocr-main-streams-cdc-connector|task-0] Error processing binlog event, and propagating to Kafka Connect so it stops this connector. Future binlog events read before connector is shutdown will be ignored. (io.debezium.connector.mysql.MySqlStreamingChangeEventSource) [blc-lsports-ocr-replica.clr89j2kvsx8.eu-west-1.rds.amazonaws.com:3306]
2023-12-04 20:19:03,272 INFO [ocr-main-streams-cdc-connector|task-0] Keepalive thread is running (io.debezium.connector.mysql.MySqlStreamingChangeEventSource) [debezium-mysqlconnector-ocr_main_streams-change-event-source-coordinator]
2023-12-04 20:19:03,614 ERROR [ocr-main-streams-cdc-connector|task-0] WorkerSourceTask{id=ocr-main-streams-cdc-connector-0} Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask) [task-thread-ocr-main-streams-cdc-connector-0]
org.apache.kafka.connect.errors.ConnectException: An exception occurred in the change event producer. This connector will be stopped.
    at io.debezium.pipeline.ErrorHandler.setProducerThrowable(ErrorHandler.java:67)
    at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleEvent(MySqlStreamingChangeEventSource.java:395)
    at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.lambda$execute$27(MySqlStreamingChangeEventSource.java:930)
    at com.github.shyiko.mysql.binlog.BinaryLogClient.notifyEventListeners(BinaryLogClient.java:1246)
    at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:1072)
    at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:631)
    at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:932)
    at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: io.debezium.DebeziumException: Error processing binlog event
    ... 7 more
Caused by: java.lang.IllegalArgumentException: The column "Id" is referenced as PRIMARY KEY, but a matching column is not defined in table "detection_service.padding_overrides"!
    at io.debezium.relational.TableEditorImpl.lambda$updatePrimaryKeys$0(TableEditorImpl.java:107)
    at java.base/java.util.ArrayList.removeIf(ArrayList.java:1672)
    at java.base/java.util.ArrayList.removeIf(ArrayList.java:1660)
    at io.debezium.relational.TableEditorImpl.updatePrimaryKeys(TableEditorImpl.java:103)
    at io.debezium.relational.TableEditorImpl.create(TableEditorImpl.java:307)
    at io.debezium.connector.mysql.antlr.listener.CreateTableParserListener.lambda$exitColumnCreateTable$1(CreateTableParserListener.java:50)
    at io.debezium.connector.mysql.antlr.MySqlAntlrDdlParser.runIfNotNull(MySqlAntlrDdlParser.java:410)
    at io.debezium.connector.mysql.antlr.listener.CreateTableParserListener.exitColumnCreateTable(CreateTableParserListener.java:42)
    at io.debezium.ddl.parser.mysql.generated.MySqlParser$ColumnCreateTableContext.exitRule(MySqlParser.java:5339)
    at io.debezium.antlr.ProxyParseTreeListenerUtil.delegateExitRule(ProxyParseTreeListenerUtil.java:64)
    at io.debezium.connector.mysql.antlr.listener.MySqlAntlrDdlParserListener.exitEveryRule(MySqlAntlrDdlParserListener.java:106)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.exitRule(ParseTreeWalker.java:63)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:38)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:36)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:36)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:36)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:36)
    at io.debezium.antlr.AntlrDdlParser.parse(AntlrDdlParser.java:83)
    at io.debezium.connector.mysql.MySqlDatabaseSchema.parseDdl(MySqlDatabaseSchema.java:213)
    at io.debezium.connector.mysql.MySqlDatabaseSchema.parseStreamingDdl(MySqlDatabaseSchema.java:199)
    at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleQueryEvent(MySqlStreamingChangeEventSource.java:593)
    at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.lambda$execute$15(MySqlStreamingChangeEventSource.java:901)
    at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleEvent(MySqlStreamingChangeEventSource.java:375)
    ... 6 more
2023-12-04 20:19:03,614 INFO [ocr-main-streams-cdc-connector|task-0] Stopping down connector (io.debezium.connector.common.BaseSourceTask) [task-thread-ocr-main-streams-cdc-connector-0]
Dec 04, 2023 8:19:03 PM com.github.shyiko.mysql.binlog.BinaryLogClient$5 run
INFO: threadExecutor is shut down, terminating keepalive thread

How to reproduce the issue using our tutorial deployment?

create new and untracked table in an untracked database. with several column and a primary key definition which include one of the the column , for example:

CREATE TABLE `untracked_table_1` (
  `Id` int NOT NULL AUTO_INCREMENT,
  `Column1` varchar(255) NOT NULL,
  `Column2` varchar(255) NOT NULL
  PRIMARY KEY (`Id`))

execute a badly written 'CREATE TABLE IF NOT EXIST' statement on the table by removing the PK column name

CREATE TABLE `untracked_table_1` ( `Column1` varchar(255) NOT NULL, `Column2` varchar(255) NOT NULL PRIMARY KEY (`Id`))

Connector suppose to fail now.

Attachments

Activity

People

Assignee:: Unassigned

Reporter:: Matan Cohen (Inactive)

Votes:: 0 Vote for this issue

Watchers:: 2 Start watching this issue

Dates

Created:: 2023/12/05 3:13 PM

Updated:: 2023/12/07 2:13 PM