-
Bug
-
Resolution: Duplicate
-
Major
-
None
-
None
-
None
-
None
-
False
-
None
-
False
I was able to reproduce a MySQL binlog deserialization error that we have run into when using the Debezium MySQL connector and which I have seen mentioned in a few places (one example in this ticket here: https://issues.redhat.com/browse/DBZ-5333).
It seems that there are two important conditions to reproduce: 1) a SMT with some latency and 2) a number of large bulk insert queries. This seems to cause the loss of the connection with MySQL which we see with:
Nov 28, 2022 10:47:09 PM com.github.shyiko.mysql.binlog.BinaryLogClient$5 run INFO: Keepalive: Trying to restore lost connection to mysql:3306
and then the deserialization error follows soon thereafter:
org.apache.kafka.connect.errors.ConnectException: An exception occurred in the change event producer. This connector will be stopped. at io.debezium.pipeline.ErrorHandler.setProducerThrowable(ErrorHandler.java:50) at io.debezium.connector.mysql.MySqlStreamingChangeEventSource$ReaderThreadLifecycleListener.onEventDeserializationFailure(MySqlStreamingChangeEventSource.java:1247) at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:1064) at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:631) at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:932) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: io.debezium.DebeziumException: Failed to deserialize data of EventHeaderV4{timestamp=1669665091000, eventType=EXT_WRITE_ROWS, serverId=1, headerLength=19, dataLength=8112, nextPosition=21550035, flags=0} at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.wrap(MySqlStreamingChangeEventSource.java:1194) ... 5 more Caused by: com.github.shyiko.mysql.binlog.event.deserialization.EventDataDeserializationException: Failed to deserialize data of EventHeaderV4{timestamp=1669665091000, eventType=EXT_WRITE_ROWS, serverId=1, headerLength=19, dataLength=8112, nextPosition=21550035, flags=0} at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.deserializeEventData(EventDeserializer.java:341) at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.nextEvent(EventDeserializer.java:244) at io.debezium.connector.mysql.MySqlStreamingChangeEventSource$1.nextEvent(MySqlStreamingChangeEventSource.java:230) at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:1051) ... 3 more Caused by: com.github.shyiko.mysql.binlog.event.deserialization.MissingTableMapEventException: No TableMapEventData has been found for table id:116. Usually that means that you have started reading binary log 'within the logical event group' (e.g. from WRITE_ROWS and not proceeding TABLE_MAP at com.github.shyiko.mysql.binlog.event.deserialization.AbstractRowsEventDataDeserializer.deserializeRow(AbstractRowsEventDataDeserializer.java:109) at com.github.shyiko.mysql.binlog.event.deserialization.WriteRowsEventDataDeserializer.deserializeRows(WriteRowsEventDataDeserializer.java:64) at com.github.shyiko.mysql.binlog.event.deserialization.WriteRowsEventDataDeserializer.deserialize(WriteRowsEventDataDeserializer.java:56) at com.github.shyiko.mysql.binlog.event.deserialization.WriteRowsEventDataDeserializer.deserialize(WriteRowsEventDataDeserializer.java:32) at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.deserializeEventData(EventDeserializer.java:335) ... 6 more
We have been able to avoid this error by fixing some inefficient code in our SMT but still are concerned that this issue could resurface. I have created a repo with instructions on how to reproduce this error locally (takes 5 mins to get up and running and to reproduce the error):
https://github.com/nadflinn/reproduce-debezium-deserialization-error
In the example I am running Kafka Connect 2.6 and Debezium MySQL connector 1.9.7 and MySQL 5.7.
If anyone with more knowledge on how the binlog client works has time to take a look it would be great to better understand what is going on beneath the hood and, if possible, fix the issue or provide guidance on how to avoid it. Thank you!
