Loading...

Type: Bug
Resolution: Duplicate
Priority: Major
Fix Version/s: None
Affects Version/s: 1.5.0.Final
Component/s: mysql-connector
Labels:
None

Blocked:
False
Ready:
False
Release Note Text:
Undefined
Market:

SFDC Cases Counter:
SFDC Cases Open:
SFDC Cases Links:

Hello debezium team,
We are using debezium latest version 1.5.0.Final with the latest parser. We use the Debezium Engine to handle parsing and build our own logic based on the Customized Consumer according to this doc. Before the exception, we are using the gh-ost tool to apply a table upgrade.

The exception looks like this:

! Causing: io.debezium.text.ParsingException: DDL statement couldn't be parsed. Please open a Jira issue with the statement '`yelp`.`test_compatible_alter_table` to `yelp`.`_test_compatible_alter_table_del`'

It looks somehow missing the `RENAME TABLE` keyword.

Process done by the tool is:

CREATE TABLE source_gho LIKE source
ALTER TABLE source_gho to "whatever u want";
COPY rows between source and _source_gho (snapshot + binlog)
RENAME TABLE source TO _source_del, _source_gho TO source;

Following are related binlog events:

| mysql-bin.000003 | 12926 | Query          | 1705883804 |       13081 | use `yelp`; DROP TABLE IF EXISTS `_test_compatible_alter_table_del` /* generated by server */                                                                                                                                                                                                                                                                               |
| mysql-bin.000003 | 13081 | Gtid           | 1705883804 |       13146 | SET @@SESSION.GTID_NEXT= '1065c2ba-a7cd-11eb-a0d4-0242ac110007:33'                                                                                                                                                                                                                                                                                                          |
| mysql-bin.000003 | 13146 | Query          | 1705883804 |       13410 | use `yelp`; rename /* gh-ost */ table `yelp`.`test_compatible_alter_table` to `yelp`.`_test_compatible_alter_table_del`, `yelp`.`_test_compatible_alter_table_gho` to `yelp`.`test_compatible_alter_table`

The exception can be reproduced in our acceptance test environment.
We had a table `test_compatible_alter_table` initially setup. Then we upgraded the mysql table by using the gh-ost tool:

./gh-ost -user mysql -host source_mysql_cluster --database yelp --table test_compatible_alter_table --alter="ADD COLUMN test2 INT NOT NULL DEFAULT '0'" --allow-on-master --initially-drop-old-table -execute

There was no error at this moment and we saw three gh-ost related topics being created which means the ddl was parsed correctly and our customized consumer logic handled the new table correctly.

test_debezium_testing_mysql_source_yelp___test_compatible_alter_table_ghc__7edbf550188249c88ca16ed891ca9650
test_debezium_testing_mysql_source_yelp___test_compatible_alter_table_gho__e8650a257cfa469d95801ca56bd07e4d
test_debezium_testing_mysql_source_yelp__test_compatible_alter_table__c40f010f32b14c9dbe678a5915830f23

However, when the Debezium Engine was restarted, it failed when trying to recover from the database history topic:

ERROR [2021-04-28 03:00:13,696] io.debezium.relational.history.KafkaDatabaseHistory: Unexpected exception while processing record 'ConsumerRecord(topic = test_debezium_testing_mysql_source_debezium_database_history__source_mysql_cluster_debezium_database_history__4ad73ef2c6f9410db2d85215c2183554, partition = 0, leaderEpoch = 0, offset = 14, CreateTime = 1619578745962, serialized key size = -1, serialized value size = 1280, headers = RecordHeaders(headers = [], isReadOnly = false), key = null, value = {
  "source" : {
    "server" : "debezium_integration_test_mysql_server"
  },
  "position" : {
    "transaction_id" : null,
    "ts_sec" : 1619578745,
    "file" : "mysql-bin.000003",
    "pos" : 13410,
    "gtids" : "afa76699-a78c-11eb-b767-0242a9fe0007:1-13,1065c2ba-a7cd-11eb-a0d4-0242ac110007:1-32",
    "server_id" : 1705883804
  },
  "databaseName" : "yelp",
  "ddl" : "`yelp`.`test_compatible_alter_table` to `yelp`.`_test_compatible_alter_table_del`",
  "tableChanges" : [ {
    "type" : "ALTER",
    "id" : "\"yelp\".\"_test_compatible_alter_table_del\"",
    "table" : {
      "defaultCharsetName" : "utf8",
      "primaryKeyColumnNames" : [ "id" ],
      "columns" : [ {
        "name" : "id",
        "jdbcType" : 4,
        "typeName" : "INT",
        "typeExpression" : "INT",
        "charsetName" : null,
        "length" : 11,
        "position" : 1,
        "optional" : false,
        "autoIncremented" : true,
        "generated" : true
      }, {
        "name" : "name",
        "jdbcType" : 12,
        "typeName" : "VARCHAR",
        "typeExpression" : "VARCHAR",
        "charsetName" : "utf8",
        "length" : 64,
        "position" : 2,
        "optional" : true,
        "autoIncremented" : false,
        "generated" : false
      } ]
    }
  } ]
})'
! org.antlr.v4.runtime.InputMismatchException: null
! at org.antlr.v4.runtime.DefaultErrorStrategy.sync(DefaultErrorStrategy.java:270)
! at io.debezium.ddl.parser.mysql.generated.MySqlParser.root(MySqlParser.java:880)
! ... 15 common frames omitted
! Causing: io.debezium.text.ParsingException: DDL statement couldn't be parsed. Please open a Jira issue with the statement '`yelp`.`test_compatible_alter_table` to `yelp`.`_test_compatible_alter_table_del`'
! mismatched input '`yelp`' expecting {<EOF>, 'ALTER', 'ANALYZE', 'CALL', 'CHANGE', 'CHECK', 'CREATE', 'DELETE', 'DESC', 'DESCRIBE', 'DROP', 'EXPLAIN', 'GET', 'GRANT', 'INSERT', 'KILL', 'LOAD', 'LOCK', 'OPTIMIZE', 'PURGE', 'RELEASE', 'RENAME', 'REPLACE', 'RESIGNAL', 'REVOKE', 'SELECT', 'SET', 'SHOW', 'SIGNAL', 'UNLOCK', 'UPDATE', 'USE', 'BEGIN', 'BINLOG', 'CACHE', 'CHECKSUM', 'COMMIT', 'DEALLOCATE', 'DO', 'FLUSH', 'HANDLER', 'HELP', 'INSTALL', 'PREPARE', 'REPAIR', 'RESET', 'ROLLBACK', 'SAVEPOINT', 'START', 'STOP', 'TRUNCATE', 'UNINSTALL', 'XA', 'EXECUTE', 'SHUTDOWN', '--', '(', ';'}
! at io.debezium.antlr.ParsingErrorListener.syntaxError(ParsingErrorListener.java:43)
! at org.antlr.v4.runtime.ProxyErrorListener.syntaxError(ProxyErrorListener.java:41)
! at org.antlr.v4.runtime.Parser.notifyErrorListeners(Parser.java:544)
! at org.antlr.v4.runtime.DefaultErrorStrategy.reportInputMismatch(DefaultErrorStrategy.java:327)
! at org.antlr.v4.runtime.DefaultErrorStrategy.reportError(DefaultErrorStrategy.java:139)
! at io.debezium.ddl.parser.mysql.generated.MySqlParser.root(MySqlParser.java:905)
! at io.debezium.connector.mysql.antlr.MySqlAntlrDdlParser.parseTree(MySqlAntlrDdlParser.java:72)
! at io.debezium.connector.mysql.antlr.MySqlAntlrDdlParser.parseTree(MySqlAntlrDdlParser.java:45)
! at io.debezium.antlr.AntlrDdlParser.parse(AntlrDdlParser.java:80)
! at io.debezium.relational.history.AbstractDatabaseHistory.lambda$recover$1(AbstractDatabaseHistory.java:134)
! at io.debezium.relational.history.KafkaDatabaseHistory.recoverRecords(KafkaDatabaseHistory.java:307)
! at io.debezium.relational.history.AbstractDatabaseHistory.recover(AbstractDatabaseHistory.java:101)
! at io.debezium.relational.HistorizedRelationalDatabaseSchema.recover(HistorizedRelationalDatabaseSchema.java:49)
! at io.debezium.connector.mysql.MySqlConnectorTask.validateAndLoadDatabaseHistory(MySqlConnectorTask.java:311)
! at io.debezium.connector.mysql.MySqlConnectorTask.start(MySqlConnectorTask.java:96)
! at io.debezium.connector.common.BaseSourceTask.start(BaseSourceTask.java:130)
! at io.debezium.embedded.EmbeddedEngine.run(EmbeddedEngine.java:759)
! at io.debezium.embedded.ConvertingEngineBuilder$2.run(ConvertingEngineBuilder.java:188)
! at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
! at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
! at java.base/java.lang.Thread.run(Thread.java:834)

duplicates

DBZ-3399 Rename table stores only a fragment of DDL in schema history

Closed

is duplicated by

DBZ-3487 DDL statement couldn't be parsed

Closed

Details

Description

Attachments

Issue Links

Easy Agile Planning Poker

Activity

People

Dates