Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-674

Replication error with MySql HA setup

XMLWordPrintable

      Architecture overview:

      Debezium is configured to access Mysql cluster with master-slave replication mode through HAProxy, GTID enabled.

      At first run, everything works as expected.
      When Mysql instance where Debezium and underlying mysql-binlog-connector-java are connected crashes, mysql binlog client will try to reconnect, connection will be directed to new instance but instance will have different position inside binary log and task will be terminated.

      Mysql replication error code 1236 was occurred, here’s the log.

      When kafka connect task is restarted, only GTID info is set to binary log client, without binlog filename and position info and everything works

      connect | 2018-03-26 13:44:02,384 INFO MySQL|dbserver1|binlog Stopped reading binlog after 12 events, last recorded offset:

      Unknown macro: {ts_sec=1522071785, file=binlogfilename.000001, pos=12560, gtids=c197da1d-30f9-11e8-9aa3-0242ac1a0008}

      [io.debezium.connector.mysql.BinlogReader]
      connect | 2018-03-26 13:44:40,637 INFO || WorkerSourceTask

      Unknown macro: {id=ride-connector-0}

      Committing offsets [org.apache.kafka.connect.runtime.WorkerSourceTask]
      connect | 2018-03-26 13:44:40,637 INFO || WorkerSourceTask

      flushing 0 outstanding messages for offset commit [org.apache.kafka.connect.runtime.WorkerSourceTask]
      connect | Mar 26, 2018 1:44:41 PM com.github.shyiko.mysql.binlog.BinaryLogClient$5 run
      connect | INFO: Trying to restore lost connection to haproxy:3306
      connect | 2018-03-26 13:44:41,887 INFO MySQL|dbserver1|binlog Creating thread debezium-mysqlconnector-dbserver1-binlog-client [io.debezium.util.Threads]
      connect | Mar 26, 2018 1:44:41 PM com.github.shyiko.mysql.binlog.BinaryLogClient connect
      connect | INFO: Connected to haproxy:3306 at c197da1d-30f9-11e8-9aa3-0242ac1a0008:1-37 (sid:184054, cid:361)
      connect | 2018-03-26 13:44:41,987 INFO MySQL|dbserver1|binlog Connected to MySQL binlog at haproxy:3306, starting at GTIDs c197da1d-30f9-11e8-9aa3-0242ac1a0008:1-37 and binlog file 'binlogfilename.000001', pos=12867, skipping 1 events plus 0 rows [io.debezium.connector.mysql.BinlogReader]
      connect | 2018-03-26 13:44:42,033 ERROR MySQL|dbserver1|binlog Error during binlog processing. Last offset stored =

      Unknown macro: {ts_sec=1522071785, file=binlogfilename.000001, pos=12560, gtids=c197da1d-30f9-11e8-9aa3-0242ac1a0008}

      , binlog reader near position = binlogfilename.000001/123 [io.debezium.connector.mysql.BinlogReader]
      connect | 2018-03-26 13:44:42,035 INFO || WorkerSourceTask

      Unknown macro: {id=ride-connector-0}

      Committing offsets [org.apache.kafka.connect.runtime.WorkerSourceTask]
      connect | 2018-03-26 13:44:42,035 INFO MySQL|dbserver1|binlog Stopped reading binlog after 12 events, last recorded offset:

      Unknown macro: {ts_sec=1522071785, file=binlogfilename.000001, pos=12560, gtids=c197da1d-30f9-11e8-9aa3-0242ac1a0008}

      [io.debezium.connector.mysql.BinlogReader]

              Unassigned Unassigned
              vuckooo Ivan Vucina (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated: