Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-572

Carry over SourceInfo.restartEventsToSkip to next binlog file handling cause binlog events are not written to kafka

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Blocker Blocker
    • 0.7.3
    • 0.7.1
    • mysql-connector
    • None
    • Hide

      1. Assume SourceInfo.restartEventsToSkip = 10 after handling a ROTATE event of binlog file mysql-bin.000002;

      2. Continue handle first 5 events in next binlog file (mysql-bin.000003) and SourceInfo.restartEventsToSkip will be 15;

      3. debezium_offsets topic is updated with 'event' field = 15, for instance:

      {"ts_sec":1516725113,"file":"mysql-bin.000004","pos":334,"server_id":184054,"event":15}

      4. A restart of debezium service happens and the first 15 events of mysql-bin.000003 gonna be skipped even though you only consumed 5 events in the mysql-bin.000003 binlog file.

      Show
      1. Assume SourceInfo.restartEventsToSkip = 10 after handling a ROTATE event of binlog file mysql-bin.000002; 2. Continue handle first 5 events in next binlog file (mysql-bin.000003) and SourceInfo.restartEventsToSkip will be 15; 3. debezium_offsets topic is updated with 'event' field = 15, for instance: { "ts_sec" :1516725113, "file" : "mysql-bin.000004" , "pos" :334, "server_id" :184054, "event" :15} 4. A restart of debezium service happens and the first 15 events of mysql-bin.000003 gonna be skipped even though you only consumed 5 events in the mysql-bin.000003 binlog file.

      When meet a FORMAT_DESCRIPTION event, SourceInfo.restartEventsToSkip should be reset to 0. Otherwise a restart of the service is likely cause binlog events are skipped somewhat.

              jpechane Jiri Pechanec
              pigwen_jira Cole0110224 Wen (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: