Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-3949

Log DML replication events instead of throwing an error

    XMLWordPrintable

Details

    Description

      I am running a debezium connector against an amazon MYSQL RDS with MYSQL version 5.7.26. Here is my debezium configuration:

      database.server.id=6013
      database.history.kafka.bootstrap.servers=KAFKA:9092
      database.history.kafka.topic=dev-test-debezium-history
      transforms=unwrap
      database.server.name=dev-test
      snapshot.delay.ms=0
      database.hostname=host
      database.password=pass
      database.history.skip.unparseable.ddl=true
      transforms.unwrap.type=io.debezium.transforms.ExtractNewRecordState
      table.include.list=bbt.test_table
      binlog.buffer.size=0
      include.query=false
      database.include.list=bbt
      snapshot.mode=schema_only
      

      When I run the connector, I get the below error:

      [2021-08-31 13:35:51,066] ERROR Error during binlog processing. Last offset stored = null, binlog reader near position = mysql-bin-changelog.531476/647 (io.debezium.connect
      or.mysql.MySqlStreamingChangeEventSource:1054)
      [2021-08-31 13:35:51,067] ERROR Producer failure (io.debezium.pipeline.ErrorHandler:31)
      io.debezium.DebeziumException: Error processing binlog event
              at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleEvent(MySqlStreamingChangeEventSource.java:366)
              at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.lambda$execute$25(MySqlStreamingChangeEventSource.java:842)
              at com.github.shyiko.mysql.binlog.BinaryLogClient.notifyEventListeners(BinaryLogClient.java:1118)
              at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:966)
              at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:606)
              at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:850)
              at java.lang.Thread.run(Thread.java:748)
      Caused by: io.debezium.DebeziumException: Received DML 'DELETE FROM `ecshop`.`ecs_sessions`' for processing, binlog probably contains events generated with statement or mix
      ed based replication format
              at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleQueryEvent(MySqlStreamingChangeEventSource.java:553)
              at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.lambda$execute$14(MySqlStreamingChangeEventSource.java:814)
              at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleEvent(MySqlStreamingChangeEventSource.java:349)
              ... 6 more
      [2021-08-31 13:35:51,067] INFO Error processing binlog event, and propagating to Kafka Connect so it stops this connector. Future binlog events read before connector is shu
      tdown will be ignored. (io.debezium.connector.mysql.MySqlStreamingChangeEventSource:371)
      [2021-08-31 13:35:51,535] INFO WorkerSourceTask{id=dev-order-connector-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:398)
      [2021-08-31 13:35:51,535] INFO WorkerSourceTask{id=dev-order-connector-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTa
      sk:415)
      [2021-08-31 13:35:51,535] ERROR WorkerSourceTask{id=dev-order-connector-0} Task threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask:179)
      org.apache.kafka.connect.errors.ConnectException: An exception occurred in the change event producer. This connector will be stopped.
              at io.debezium.pipeline.ErrorHandler.setProducerThrowable(ErrorHandler.java:42)
              at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleEvent(MySqlStreamingChangeEventSource.java:366)
              at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.lambda$execute$25(MySqlStreamingChangeEventSource.java:842)
              at com.github.shyiko.mysql.binlog.BinaryLogClient.notifyEventListeners(BinaryLogClient.java:1118)
              at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:966)
              at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:606)
              at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:850)
              at java.lang.Thread.run(Thread.java:748)
      Caused by: io.debezium.DebeziumException: Error processing binlog event
              ... 7 more
      

      my own connector jars files:

      antlr4-runtime-4.8.jar
      debezium-api-1.6.1.Final.jar               
      debezium-connector-mysql-1.6.1.Final.jar   
      debezium-core-1.6.1.Final.jar              
      debezium-ddl-parser-1.6.1.Final.jar        
      mysql-binlog-connector-java-0.25.1.jar
      mysql-connector-java-8.0.21.jar
      

      Attachments

        Activity

          People

            vjuranek@redhat.com Vojtech Juranek
            xxm404 404 xxm (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: