Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-2208

Support compressed binlog events from MariaDB

XMLWordPrintable

    • Icon: Feature Request Feature Request
    • Resolution: Unresolved
    • Icon: Blocker Blocker
    • 1.9-backlog
    • 1.7.0.CR1
    • mysql-connector
    • None

      We are facing this issue whenever binlog compression is enabled on the database side. Database is: mariadb-10.3.21

      Logs show the following: Unknown event type 117

      ERROR Error during binlog processing. Last offset stored = {ts_sec=1591629226, file=mysql-bin.033898, pos=758324334, row=1, server_id=15, event=3}, binlog reader near position = mysql-bin.033898/3498342761 (io.debezium.connector.mysql.BinlogReader)
      ERROR Error during binlog processing. Last offset stored = {ts_sec=1591629226, file=mysql-bin.033898, pos=758324334, row=1, server_id=15, event=3}, binlog reader near position = mysql-bin.033898/3498342761 (io.debezium.connector.mysql.BinlogReader)
      INFO WorkerSourceTask{id=connector-masterdb-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask)
      INFO WorkerSourceTask{id=connector-masterdb-0} Finished commitOffsets successfully in 2 ms (org.apache.kafka.connect.runtime.WorkerSourceTask)
      	at java.lang.Thread.run(Thread.java:748)
      	at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:922)
      	at io.debezium.connector.mysql.BinlogReader$1.nextEvent(BinlogReader.java:237)
      	at com.github.shyiko.mysql.binlog.event.deserialization.EventHeaderV4Deserializer.deserialize(EventHeaderV4Deserializer.java:35)
      	at io.debezium.connector.mysql.AbstractReader.failed(AbstractReader.java:197)
      Caused by: java.io.IOException: Unknown event type 117
      	at io.debezium.connector.mysql.BinlogReader$ReaderThreadLifecycleListener.onEventDeserializationFailure(BinlogReader.java:1049)
      	at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:825)
      	at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.nextEvent(EventDeserializer.java:212)
      org.apache.kafka.connect.errors.ConnectException: Unknown event type 117
      	at io.debezium.connector.mysql.AbstractReader.wrap(AbstractReader.java:230)
      	... 3 more
      [2020-06-08 15:13:46,884] INFO ChainedReader: Stopping the binlog reader (io.debezium.connector.mysql.ChainedReader)
      	at com.github.shyiko.mysql.binlog.event.deserialization.EventHeaderV4Deserializer.deserialize(EventHeaderV4Deserializer.java:27)
      	at com.github.shyiko.mysql.binlog.event.deserialization.EventHeaderV4Deserializer.getEventType(EventHeaderV4Deserializer.java:45)
      	at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:580)
      [2020-06-08 15:13:46,884] INFO Stopping MySQL connector task (io.debezium.connector.mysql.MySqlConnectorTask)
      [2020-06-08 15:13:46,884] ERROR WorkerSourceTask{id=connector-masterdb-0} Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask)
      [2020-06-08 15:13:46,881] ERROR WorkerSourceTask{id=connector-masterdb-0} Task threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask)
      	at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:935)
      [2020-06-08 15:13:46,884] INFO Discarding 0 unsent record(s) due to the connector shutting down (io.debezium.connector.mysql.BinlogReader)
      [2020-06-08 15:13:46,885] INFO Discarding 0 unsent record(s) due to the connector shutting down (io.debezium.connector.mysql.BinlogReader)
      [2020-06-08 15:13:46,885] INFO Stopped reading binlog after 254962 events, last recorded offset: {ts_sec=1591629226, file=mysql-bin.033898, pos=758324334, row=1, server_id=15, event=3} (io.debezium.connector.mysql.BinlogReader)
      

      using mysqlbinlog, the problematic entry looks something like this:

      Event{ header=EventHeaderV4{ timestamp=1592003319000, eventType=QUERY, serverId=5, headerLength=19, dataLength=280, nextPosition=753402908, flags=8 }, data=QueryEventData{ threadId=0, executionTime=0, errorCode=0, database='', sql='# Dummy event replacing event type 160 that slave cannot handle. ' } }

              Unassigned Unassigned
              akhawatrah Abdallah Alkhawatrah (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated: