Details
-
Bug
-
Resolution: Unresolved
-
Major
-
1.0.2.Final
-
None
-
- Stop a running connector, and start it
- The MySQL server must have GTID enabled
Description
I am facing the following error when I stop and start the Kafka Connect process running debezium mysql connector. I have many MySQL database servers that debezium reads from, and my observation is that this happens particularly in MySQL instance having GTID enabled.
Once failed with this error, the connector won't start for 24 hours and will keep throwing the same error on each restart attempt. After 24 hours I am able to successfully start the connector though.
This is impacting our production pipelines.
org.apache.kafka.connect.errors.ConnectException: com.github.shyiko.mysql.binlog.event.deserialization.EventDataDeserializationException: Failed to deserialize data of EventHeaderV4{timestamp=1596224623000, eventType=EXT_UPDATE_ROWS, serverId=1861118255, headerLength=19, dataLength=41149, nextPosition=83347748, flags=0}
at io.debezium.connector.mysql.AbstractReader.wrap(AbstractReader.java:230)
at io.debezium.connector.mysql.AbstractReader.failed(AbstractReader.java:207)
at io.debezium.connector.mysql.BinlogReader.handleEvent(BinlogReader.java:536)
at com.github.shyiko.mysql.binlog.BinaryLogClient.notifyEventListeners(BinaryLogClient.java:1095)
at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:943)
at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:580)
at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:825)
at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.RuntimeException: com.github.shyiko.mysql.binlog.event.deserialization.EventDataDeserializationException: Failed to deserialize data of EventHeaderV4{timestamp=1596224623000, eventType=EXT_UPDATE_ROWS, serverId=1861118255, headerLength=19, dataLength=41149, nextPosition=83347748, flags=0}
at io.debezium.connector.mysql.BinlogReader.handleServerIncident(BinlogReader.java:604)
at io.debezium.connector.mysql.BinlogReader.handleEvent(BinlogReader.java:519)
... 5 more
Caused by: com.github.shyiko.mysql.binlog.event.deserialization.EventDataDeserializationException: Failed to deserialize data of EventHeaderV4{timestamp=1596224623000, eventType=EXT_UPDATE_ROWS, serverId=1861118255, headerLength=19, dataLength=41149, nextPosition=83347748, flags=0}
at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.deserializeEventData(EventDeserializer.java:300)
at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.nextEvent(EventDeserializer.java:223)
at io.debezium.connector.mysql.BinlogReader$1.nextEvent(BinlogReader.java:239)
at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:922)
... 3 more
Caused by: java.io.EOFException
at com.github.shyiko.mysql.binlog.io.ByteArrayInputStream.read(ByteArrayInputStream.java:190)
at java.io.InputStream.read(Unknown Source)
at java.io.InputStream.skip(Unknown Source)
at com.github.shyiko.mysql.binlog.io.ByteArrayInputStream.skipToTheEndOfTheBlock(ByteArrayInputStream.java:216)
at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.deserializeEventData(EventDeserializer.java:296)
... 6 more