Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-3481

RENAME TABLE DDL cannot be parsed

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Major
    • None
    • 1.5.0.Final
    • mysql-connector
    • None
    • False
    • False
    • Undefined

    Description

      Hello debezium team,
      We are using debezium latest version 1.5.0.Final with the latest parser. We use the Debezium Engine to handle parsing and build our own logic based on the Customized Consumer according to this doc. Before the exception, we are using the gh-ost tool to apply a table upgrade.

      The exception looks like this:

      ! Causing: io.debezium.text.ParsingException: DDL statement couldn't be parsed. Please open a Jira issue with the statement '`yelp`.`test_compatible_alter_table` to `yelp`.`_test_compatible_alter_table_del`'

      It looks somehow missing the `RENAME TABLE` keyword.

      Process done by the tool is:

      • CREATE TABLE source_gho LIKE source
      • ALTER TABLE source_gho to "whatever u want";
      • COPY rows between source and _source_gho (snapshot + binlog)
      • RENAME TABLE source TO _source_del, _source_gho TO source;

      Following are related binlog events:

      | mysql-bin.000003 | 12926 | Query          | 1705883804 |       13081 | use `yelp`; DROP TABLE IF EXISTS `_test_compatible_alter_table_del` /* generated by server */                                                                                                                                                                                                                                                                               |
| mysql-bin.000003 | 13081 | Gtid           | 1705883804 |       13146 | SET @@SESSION.GTID_NEXT= '1065c2ba-a7cd-11eb-a0d4-0242ac110007:33'                                                                                                                                                                                                                                                                                                          |
| mysql-bin.000003 | 13146 | Query          | 1705883804 |       13410 | use `yelp`; rename /* gh-ost */ table `yelp`.`test_compatible_alter_table` to `yelp`.`_test_compatible_alter_table_del`, `yelp`.`_test_compatible_alter_table_gho` to `yelp`.`test_compatible_alter_table`

      The exception can be reproduced in our acceptance test environment.
      We had a table `test_compatible_alter_table` initially setup. Then we upgraded the mysql table by using the gh-ost tool:

      ./gh-ost -user mysql -host source_mysql_cluster --database yelp --table test_compatible_alter_table --alter="ADD COLUMN test2 INT NOT NULL DEFAULT '0'" --allow-on-master --initially-drop-old-table -execute

      There was no error at this moment and we saw three gh-ost related topics being created which means the ddl was parsed correctly and our customized consumer logic handled the new table correctly.

      test_debezium_testing_mysql_source_yelp___test_compatible_alter_table_ghc__7edbf550188249c88ca16ed891ca9650
test_debezium_testing_mysql_source_yelp___test_compatible_alter_table_gho__e8650a257cfa469d95801ca56bd07e4d
test_debezium_testing_mysql_source_yelp__test_compatible_alter_table__c40f010f32b14c9dbe678a5915830f23

      However, when the Debezium Engine was restarted, it failed when trying to recover from the database history topic:

      ERROR [2021-04-28 03:00:13,696] io.debezium.relational.history.KafkaDatabaseHistory: Unexpected exception while processing record 'ConsumerRecord(topic = test_debezium_testing_mysql_source_debezium_database_history__source_mysql_cluster_debezium_database_history__4ad73ef2c6f9410db2d85215c2183554, partition = 0, leaderEpoch = 0, offset = 14, CreateTime = 1619578745962, serialized key size = -1, serialized value size = 1280, headers = RecordHeaders(headers = [], isReadOnly = false), key = null, value = {
  "source" : {
    "server" : "debezium_integration_test_mysql_server"
  },
  "position" : {
    "transaction_id" : null,
    "ts_sec" : 1619578745,
    "file" : "mysql-bin.000003",
    "pos" : 13410,
    "gtids" : "afa76699-a78c-11eb-b767-0242a9fe0007:1-13,1065c2ba-a7cd-11eb-a0d4-0242ac110007:1-32",
    "server_id" : 1705883804
  },
  "databaseName" : "yelp",
  "ddl" : "`yelp`.`test_compatible_alter_table` to `yelp`.`_test_compatible_alter_table_del`",
  "tableChanges" : [ {
    "type" : "ALTER",
    "id" : "\"yelp\".\"_test_compatible_alter_table_del\"",
    "table" : {
      "defaultCharsetName" : "utf8",
      "primaryKeyColumnNames" : [ "id" ],
      "columns" : [ {
        "name" : "id",
        "jdbcType" : 4,
        "typeName" : "INT",
        "typeExpression" : "INT",
        "charsetName" : null,
        "length" : 11,
        "position" : 1,
        "optional" : false,
        "autoIncremented" : true,
        "generated" : true
      }, {
        "name" : "name",
        "jdbcType" : 12,
        "typeName" : "VARCHAR",
        "typeExpression" : "VARCHAR",
        "charsetName" : "utf8",
        "length" : 64,
        "position" : 2,
        "optional" : true,
        "autoIncremented" : false,
        "generated" : false
      } ]
    }
  } ]
})'
! org.antlr.v4.runtime.InputMismatchException: null
! at org.antlr.v4.runtime.DefaultErrorStrategy.sync(DefaultErrorStrategy.java:270)
! at io.debezium.ddl.parser.mysql.generated.MySqlParser.root(MySqlParser.java:880)
! ... 15 common frames omitted
! Causing: io.debezium.text.ParsingException: DDL statement couldn't be parsed. Please open a Jira issue with the statement '`yelp`.`test_compatible_alter_table` to `yelp`.`_test_compatible_alter_table_del`'
! mismatched input '`yelp`' expecting {<EOF>, 'ALTER', 'ANALYZE', 'CALL', 'CHANGE', 'CHECK', 'CREATE', 'DELETE', 'DESC', 'DESCRIBE', 'DROP', 'EXPLAIN', 'GET', 'GRANT', 'INSERT', 'KILL', 'LOAD', 'LOCK', 'OPTIMIZE', 'PURGE', 'RELEASE', 'RENAME', 'REPLACE', 'RESIGNAL', 'REVOKE', 'SELECT', 'SET', 'SHOW', 'SIGNAL', 'UNLOCK', 'UPDATE', 'USE', 'BEGIN', 'BINLOG', 'CACHE', 'CHECKSUM', 'COMMIT', 'DEALLOCATE', 'DO', 'FLUSH', 'HANDLER', 'HELP', 'INSTALL', 'PREPARE', 'REPAIR', 'RESET', 'ROLLBACK', 'SAVEPOINT', 'START', 'STOP', 'TRUNCATE', 'UNINSTALL', 'XA', 'EXECUTE', 'SHUTDOWN', '--', '(', ';'}
! at io.debezium.antlr.ParsingErrorListener.syntaxError(ParsingErrorListener.java:43)
! at org.antlr.v4.runtime.ProxyErrorListener.syntaxError(ProxyErrorListener.java:41)
! at org.antlr.v4.runtime.Parser.notifyErrorListeners(Parser.java:544)
! at org.antlr.v4.runtime.DefaultErrorStrategy.reportInputMismatch(DefaultErrorStrategy.java:327)
! at org.antlr.v4.runtime.DefaultErrorStrategy.reportError(DefaultErrorStrategy.java:139)
! at io.debezium.ddl.parser.mysql.generated.MySqlParser.root(MySqlParser.java:905)
! at io.debezium.connector.mysql.antlr.MySqlAntlrDdlParser.parseTree(MySqlAntlrDdlParser.java:72)
! at io.debezium.connector.mysql.antlr.MySqlAntlrDdlParser.parseTree(MySqlAntlrDdlParser.java:45)
! at io.debezium.antlr.AntlrDdlParser.parse(AntlrDdlParser.java:80)
! at io.debezium.relational.history.AbstractDatabaseHistory.lambda$recover$1(AbstractDatabaseHistory.java:134)
! at io.debezium.relational.history.KafkaDatabaseHistory.recoverRecords(KafkaDatabaseHistory.java:307)
! at io.debezium.relational.history.AbstractDatabaseHistory.recover(AbstractDatabaseHistory.java:101)
! at io.debezium.relational.HistorizedRelationalDatabaseSchema.recover(HistorizedRelationalDatabaseSchema.java:49)
! at io.debezium.connector.mysql.MySqlConnectorTask.validateAndLoadDatabaseHistory(MySqlConnectorTask.java:311)
! at io.debezium.connector.mysql.MySqlConnectorTask.start(MySqlConnectorTask.java:96)
! at io.debezium.connector.common.BaseSourceTask.start(BaseSourceTask.java:130)
! at io.debezium.embedded.EmbeddedEngine.run(EmbeddedEngine.java:759)
! at io.debezium.embedded.ConvertingEngineBuilder$2.run(ConvertingEngineBuilder.java:188)
! at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
! at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
! at java.base/java.lang.Thread.run(Thread.java:834)

      Attachments

        Issue Links

          duplicates

          Bug - A problem that impairs or prevents the functions of the product. DBZ-3399 Rename table stores only a fragment of DDL in schema history

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed
          is duplicated by

          Bug - A problem that impairs or prevents the functions of the product. DBZ-3487 DDL statement couldn't be parsed

          • Minor - To be worked after urgent, high, and medium priorities are resolved. This term is chosen when the severity of the issue is low, or the complexity or effort to correct it may be higher, relatively speaking. For low priority issues, known workarounds exist or are not needed due to the trivial effort needed to address the issue.
          • Closed

          Activity

            People

              Unassigned Unassigned
              yongzhou1 Yongzhou Gu (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: