Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-3481

RENAME TABLE DDL cannot be parsed

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Duplicate Issue
    • 1.5.0.Final
    • None
    • mysql-connector
    • None

    Description

      Hello debezium team,
      We are using debezium latest version 1.5.0.Final with the latest parser. We use the Debezium Engine to handle parsing and build our own logic based on the Customized Consumer according to this doc. Before the exception, we are using the gh-ost tool to apply a table upgrade.

      The exception looks like this:

      ! Causing: io.debezium.text.ParsingException: DDL statement couldn't be parsed. Please open a Jira issue with the statement '`yelp`.`test_compatible_alter_table` to `yelp`.`_test_compatible_alter_table_del`'
      

      It looks somehow missing the `RENAME TABLE` keyword.

      Process done by the tool is:

      • CREATE TABLE source_gho LIKE source
      • ALTER TABLE source_gho to¬†"whatever u want";
      • COPY rows between source and _source_gho (snapshot + binlog)
      • RENAME TABLE source TO _source_del, _source_gho TO source;

      Following are related binlog events:

      | mysql-bin.000003 | 12926 | Query          | 1705883804 |       13081 | use `yelp`; DROP TABLE IF EXISTS `_test_compatible_alter_table_del` /* generated by server */                                                                                                                                                                                                                                                                               |
      | mysql-bin.000003 | 13081 | Gtid           | 1705883804 |       13146 | SET @@SESSION.GTID_NEXT= '1065c2ba-a7cd-11eb-a0d4-0242ac110007:33'                                                                                                                                                                                                                                                                                                          |
      | mysql-bin.000003 | 13146 | Query          | 1705883804 |       13410 | use `yelp`; rename /* gh-ost */ table `yelp`.`test_compatible_alter_table` to `yelp`.`_test_compatible_alter_table_del`, `yelp`.`_test_compatible_alter_table_gho` to `yelp`.`test_compatible_alter_table`
      

      The exception can be reproduced in our acceptance test environment.
      We had a table `test_compatible_alter_table` initially setup. Then we upgraded the mysql table by using the gh-ost tool:

      ./gh-ost -user mysql -host source_mysql_cluster --database yelp --table test_compatible_alter_table --alter="ADD COLUMN test2 INT NOT NULL DEFAULT '0'" --allow-on-master --initially-drop-old-table -execute
      

      There was no error at this moment and we saw three gh-ost related topics being created which means the ddl was parsed correctly and our customized consumer logic handled the new table correctly.

      test_debezium_testing_mysql_source_yelp___test_compatible_alter_table_ghc__7edbf550188249c88ca16ed891ca9650
      test_debezium_testing_mysql_source_yelp___test_compatible_alter_table_gho__e8650a257cfa469d95801ca56bd07e4d
      test_debezium_testing_mysql_source_yelp__test_compatible_alter_table__c40f010f32b14c9dbe678a5915830f23
      

      However, when the Debezium Engine was restarted, it failed when trying to recover from the database history topic:

      ERROR [2021-04-28 03:00:13,696] io.debezium.relational.history.KafkaDatabaseHistory: Unexpected exception while processing record 'ConsumerRecord(topic = test_debezium_testing_mysql_source_debezium_database_history__source_mysql_cluster_debezium_database_history__4ad73ef2c6f9410db2d85215c2183554, partition = 0, leaderEpoch = 0, offset = 14, CreateTime = 1619578745962, serialized key size = -1, serialized value size = 1280, headers = RecordHeaders(headers = [], isReadOnly = false), key = null, value = {
        "source" : {
          "server" : "debezium_integration_test_mysql_server"
        },
        "position" : {
          "transaction_id" : null,
          "ts_sec" : 1619578745,
          "file" : "mysql-bin.000003",
          "pos" : 13410,
          "gtids" : "afa76699-a78c-11eb-b767-0242a9fe0007:1-13,1065c2ba-a7cd-11eb-a0d4-0242ac110007:1-32",
          "server_id" : 1705883804
        },
        "databaseName" : "yelp",
        "ddl" : "`yelp`.`test_compatible_alter_table` to `yelp`.`_test_compatible_alter_table_del`",
        "tableChanges" : [ {
          "type" : "ALTER",
          "id" : "\"yelp\".\"_test_compatible_alter_table_del\"",
          "table" : {
            "defaultCharsetName" : "utf8",
            "primaryKeyColumnNames" : [ "id" ],
            "columns" : [ {
              "name" : "id",
              "jdbcType" : 4,
              "typeName" : "INT",
              "typeExpression" : "INT",
              "charsetName" : null,
              "length" : 11,
              "position" : 1,
              "optional" : false,
              "autoIncremented" : true,
              "generated" : true
            }, {
              "name" : "name",
              "jdbcType" : 12,
              "typeName" : "VARCHAR",
              "typeExpression" : "VARCHAR",
              "charsetName" : "utf8",
              "length" : 64,
              "position" : 2,
              "optional" : true,
              "autoIncremented" : false,
              "generated" : false
            } ]
          }
        } ]
      })'
      ! org.antlr.v4.runtime.InputMismatchException: null
      ! at org.antlr.v4.runtime.DefaultErrorStrategy.sync(DefaultErrorStrategy.java:270)
      ! at io.debezium.ddl.parser.mysql.generated.MySqlParser.root(MySqlParser.java:880)
      ! ... 15 common frames omitted
      ! Causing: io.debezium.text.ParsingException: DDL statement couldn't be parsed. Please open a Jira issue with the statement '`yelp`.`test_compatible_alter_table` to `yelp`.`_test_compatible_alter_table_del`'
      ! mismatched input '`yelp`' expecting {<EOF>, 'ALTER', 'ANALYZE', 'CALL', 'CHANGE', 'CHECK', 'CREATE', 'DELETE', 'DESC', 'DESCRIBE', 'DROP', 'EXPLAIN', 'GET', 'GRANT', 'INSERT', 'KILL', 'LOAD', 'LOCK', 'OPTIMIZE', 'PURGE', 'RELEASE', 'RENAME', 'REPLACE', 'RESIGNAL', 'REVOKE', 'SELECT', 'SET', 'SHOW', 'SIGNAL', 'UNLOCK', 'UPDATE', 'USE', 'BEGIN', 'BINLOG', 'CACHE', 'CHECKSUM', 'COMMIT', 'DEALLOCATE', 'DO', 'FLUSH', 'HANDLER', 'HELP', 'INSTALL', 'PREPARE', 'REPAIR', 'RESET', 'ROLLBACK', 'SAVEPOINT', 'START', 'STOP', 'TRUNCATE', 'UNINSTALL', 'XA', 'EXECUTE', 'SHUTDOWN', '--', '(', ';'}
      ! at io.debezium.antlr.ParsingErrorListener.syntaxError(ParsingErrorListener.java:43)
      ! at org.antlr.v4.runtime.ProxyErrorListener.syntaxError(ProxyErrorListener.java:41)
      ! at org.antlr.v4.runtime.Parser.notifyErrorListeners(Parser.java:544)
      ! at org.antlr.v4.runtime.DefaultErrorStrategy.reportInputMismatch(DefaultErrorStrategy.java:327)
      ! at org.antlr.v4.runtime.DefaultErrorStrategy.reportError(DefaultErrorStrategy.java:139)
      ! at io.debezium.ddl.parser.mysql.generated.MySqlParser.root(MySqlParser.java:905)
      ! at io.debezium.connector.mysql.antlr.MySqlAntlrDdlParser.parseTree(MySqlAntlrDdlParser.java:72)
      ! at io.debezium.connector.mysql.antlr.MySqlAntlrDdlParser.parseTree(MySqlAntlrDdlParser.java:45)
      ! at io.debezium.antlr.AntlrDdlParser.parse(AntlrDdlParser.java:80)
      ! at io.debezium.relational.history.AbstractDatabaseHistory.lambda$recover$1(AbstractDatabaseHistory.java:134)
      ! at io.debezium.relational.history.KafkaDatabaseHistory.recoverRecords(KafkaDatabaseHistory.java:307)
      ! at io.debezium.relational.history.AbstractDatabaseHistory.recover(AbstractDatabaseHistory.java:101)
      ! at io.debezium.relational.HistorizedRelationalDatabaseSchema.recover(HistorizedRelationalDatabaseSchema.java:49)
      ! at io.debezium.connector.mysql.MySqlConnectorTask.validateAndLoadDatabaseHistory(MySqlConnectorTask.java:311)
      ! at io.debezium.connector.mysql.MySqlConnectorTask.start(MySqlConnectorTask.java:96)
      ! at io.debezium.connector.common.BaseSourceTask.start(BaseSourceTask.java:130)
      ! at io.debezium.embedded.EmbeddedEngine.run(EmbeddedEngine.java:759)
      ! at io.debezium.embedded.ConvertingEngineBuilder$2.run(ConvertingEngineBuilder.java:188)
      ! at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
      ! at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
      ! at java.base/java.lang.Thread.run(Thread.java:834)
      

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              yongzhou1 Yongzhou Gu (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: