Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-4583

Can't process column definition with length exceeding Integer.MAX_VALUE

    XMLWordPrintable

Details

    Description

      The Bin log statement is 

      use `stage-1`; ALTER TABLE `ArrowEntity` MODIFY data BLOB(4294967295) NOT NULL;  

      The debezium gets stuck in this line, and does not make progress. Excluding Database or table seems to not help

       

          include.schema.changes: false
          event.deserialization.failure.handling.mode: warn
          snapshot.locking.mode: none
          database.history.store.only.monitored.tables.ddl: true
          database.history.skip.unparseable.ddl: true

       

      2022-01-21 03:54:00,894 INFO Error processing binlog event, and propagating to Kafka Connect so it stops this connector. Future binlog events read before connector is shutdown will be ignored. (io.debezium.connector.mysql.MySqlStreamingChangeEventSource) [blc-affirm1-us-east-1.cluster-cmmbjjhd4nzh.us-east-1.rds.amazonaws.com:3306]
      2022-01-21 03:54:00,900 INFO WorkerSourceTask{id=debezium-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask) [task-thread-debezium-0]
      2022-01-21 03:54:00,900 ERROR WorkerSourceTask{id=debezium-0} Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask) [task-thread-debezium-0]
      org.apache.kafka.connect.errors.ConnectException: An exception occurred in the change event producer. This connector will be stopped.
      	at io.debezium.pipeline.ErrorHandler.setProducerThrowable(ErrorHandler.java:42)
      	at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleEvent(MySqlStreamingChangeEventSource.java:369)
      	at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.lambda$execute$25(MySqlStreamingChangeEventSource.java:860)
      	at com.github.shyiko.mysql.binlog.BinaryLogClient.notifyEventListeners(BinaryLogClient.java:1125)
      	at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:973)
      	at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:599)
      	at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:857)
      	at java.base/java.lang.Thread.run(Thread.java:829)
      Caused by: io.debezium.DebeziumException: Error processing binlog event
      	... 7 more
      Caused by: java.lang.NumberFormatException: For input string: "4294967295"
      	at java.base/java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
      	at java.base/java.lang.Integer.parseInt(Integer.java:652)
      	at java.base/java.lang.Integer.valueOf(Integer.java:983)
      	at io.debezium.connector.mysql.antlr.listener.ColumnDefinitionParserListener.resolveColumnDataType(ColumnDefinitionParserListener.java:184)
      	at io.debezium.connector.mysql.antlr.listener.ColumnDefinitionParserListener.enterColumnDefinition(ColumnDefinitionParserListener.java:71)
      	at io.debezium.ddl.parser.mysql.generated.MySqlParser$ColumnDefinitionContext.enterRule(MySqlParser.java:8762)
      	at io.debezium.antlr.ProxyParseTreeListenerUtil.delegateEnterRule(ProxyParseTreeListenerUtil.java:46)
      	at io.debezium.connector.mysql.antlr.listener.MySqlAntlrDdlParserListener.enterEveryRule(MySqlAntlrDdlParserListener.java:89)
      	at org.antlr.v4.runtime.tree.ParseTreeWalker.enterRule(ParseTreeWalker.java:41)
      	at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:25)
      	at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28)
      	at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28)
      	at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28)
      	at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28)
      	at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28)
      	at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28)
      	at io.debezium.antlr.AntlrDdlParser.parse(AntlrDdlParser.java:87)
      	at io.debezium.connector.mysql.MySqlDatabaseSchema.parseDdl(MySqlDatabaseSchema.java:223)
      	at io.debezium.connector.mysql.MySqlDatabaseSchema.parseStreamingDdl(MySqlDatabaseSchema.java:209)
      	at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleQueryEvent(MySqlStreamingChangeEventSource.java:576)
      	at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.lambda$execute$14(MySqlStreamingChangeEventSource.java:832)
      	at io.debezi 

      Anyone encountered this? Any way to skip this event or parse it?

      Attachments

        Activity

          People

            jpechane Jiri Pechanec
            anand.swaminathan Anand Swaminathan (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: