Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-7037

Fake ROTATE event on connection restart cleans metadata

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major Major
    • 2.5.0.Alpha1
    • 2.1.4.Final
    • mysql-connector
    • None
    • False
    • None
    • False

      In order to make your issue reports as actionable as possible, please provide the following information, depending on the issue type.

      Bug report

      For bug reports, provide this information, please:

      What Debezium connector do you use and what version?

      mysql connector.

      Debezium version 2.1

      What is the captured database version and mode of depoyment?

      RDS latest version

      What behaviour do you see?

      Our spring boot 3.1 app which is a wrapper around Debezium gets these exceptions after running fine for 30 minutes:

      2023-10-05 07:36:54 [pool-4-thread-1] INFO  io.debezium.embedded.EmbeddedEngine - Retriable exception thrown, connector will be restarted; errors.max.retries=-1
      org.apache.kafka.connect.errors.RetriableException: An exception occurred in the change event producer. This connector will be restarted.
              at io.debezium.pipeline.ErrorHandler.setProducerThrowable(ErrorHandler.java:49)
              at io.debezium.connector.mysql.MySqlStreamingChangeEventSource$ReaderThreadLifecycleListener.onEventDeserializationFailure(MySqlStreamingChangeEventSource.java:1299)
              at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:1101)
              at com.github.shyiko.mysql.binlog.BinaryLogClient._connect(BinaryLogClient.java:620)
              at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:525)
              at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:932)
              at java.base/java.lang.Thread.run(Thread.java:833)
      Caused by: io.debezium.DebeziumException: Failed to deserialize data of EventHeaderV4{timestamp=1696426677000, eventType=EXT_DELETE_ROWS, serverId=73360770, headerLength=19, dataLength=8155, nextPosition=116155347, flags=0}
              at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.wrap(MySqlStreamingChangeEventSource.java:1246)
              ... 6 common frames omitted
      Caused by: com.github.shyiko.mysql.binlog.event.deserialization.EventDataDeserializationException: Failed to deserialize data of EventHeaderV4{timestamp=1696426677000, eventType=EXT_DELETE_ROWS, serverId=73360770, headerLength=19, dataLength=8155, nextPosition=116155347, flags=0}
              at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.deserializeEventData(EventDeserializer.java:341)
              at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.nextEvent(EventDeserializer.java:244)
              at io.debezium.connector.mysql.MySqlStreamingChangeEventSource$1.nextEvent(MySqlStreamingChangeEventSource.java:236)
              at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:1088)
              ... 4 common frames omitted
      Caused by: com.github.shyiko.mysql.binlog.event.deserialization.MissingTableMapEventException: No TableMapEventData has been found for table id:137. Usually that means that you have started reading binary log 'within the logical event group' (e.g. from WRITE_ROWS and not proceeding TABLE_MAP
              at com.github.shyiko.mysql.binlog.event.deserialization.AbstractRowsEventDataDeserializer.deserializeRow(AbstractRowsEventDataDeserializer.java:109)
              at com.github.shyiko.mysql.binlog.event.deserialization.DeleteRowsEventDataDeserializer.deserializeRows(DeleteRowsEventDataDeserializer.java:64)
              at com.github.shyiko.mysql.binlog.event.deserialization.DeleteRowsEventDataDeserializer.deserialize(DeleteRowsEventDataDeserializer.java:56)
              at com.github.shyiko.mysql.binlog.event.deserialization.DeleteRowsEventDataDeserializer.deserialize(DeleteRowsEventDataDeserializer.java:32)
              at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.deserializeEventData(EventDeserializer.java:335)
              ... 7 common frames omitted

       

      and then it gets stuck at:

       

      2023-10-05 07:36:59 [pool-4-thread-1] ERROR io.debezium.embedded.EmbeddedEngine - Error while trying to run connector class 'io.debezium.connector.mysql.MySqlConnector'
      org.apache.kafka.connect.errors.ConnectException: An exception occurred in the change event producer. This connector will be stopped.
              at io.debezium.pipeline.ErrorHandler.setProducerThrowable(ErrorHandler.java:53)
              at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleEvent(MySqlStreamingChangeEventSource.java:396)
              at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.lambda$execute$27(MySqlStreamingChangeEventSource.java:926)
              at com.github.shyiko.mysql.binlog.BinaryLogClient.notifyEventListeners(BinaryLogClient.java:1283)
              at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:1109)
              at com.github.shyiko.mysql.binlog.BinaryLogClient._connect(BinaryLogClient.java:620)
              at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:525)
              at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:932)
              at java.base/java.lang.Thread.run(Thread.java:833)
      Caused by: io.debezium.DebeziumException: Error processing binlog event
              ... 8 common frames omitted
      Caused by: io.debezium.DebeziumException: Encountered change event for table events_qa.events2 whose schema isn't known to this connector
              at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.informAboutUnknownTableIfRequired(MySqlStreamingChangeEventSource.java:701)
              at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.informAboutUnknownTableIfRequired(MySqlStreamingChangeEventSource.java:742)
              at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleUpdateTableMetadata(MySqlStreamingChangeEventSource.java:657)
              at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.lambda$execute$14(MySqlStreamingChangeEventSource.java:896)
              at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleEvent(MySqlStreamingChangeEventSource.java:376)
              ... 7 common frames omitted

      Do you see the same behaviour using the latest relesead Debezium version?

      No

              Unassigned Unassigned
              a50c68ae Anon Anon (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated:
                Resolved: