Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-1033

Mysql binlog reader lost data if restart task when last binlog event is QUERY event.

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major Major
    • 0.9.0.Beta2
    • 0.8.3.Final
    • mysql-connector
    • None
    • Hide

      Reproduce procedure:

      1. create table `test` (id int, val varchar(20));
      2. start dbz connector to sync for table test.
      3. alter table `test` comment 'test1'/'test2' for N times.
      4. restart the task.
      5. insert into `test` VALUES(1, 'test')

      The binlog event for insert will be skipped.

      Show
      Reproduce procedure: 1. create table `test` (id int, val varchar(20)); 2. start dbz connector to sync for table test. 3. alter table `test` comment 'test1'/'test2' for N times. 4. restart the task. 5. insert into `test` VALUES(1, 'test') The binlog event for insert will be skipped.

      Binlog reader increased the count for restartEventsToSkip each time after it processed binlog event. That count would be reset when there is a 'BEGIN' sql or a 'COMMIT' sql. However, if the task was restarted after processing a DDL event, it would increase the restartEventsToSkip count as well as record the latest binlog position in offset, and the offset will be updated to kafka offset topic by the DDL record. And then, if the task was restarted, it will skip the first N events after restart which should not be skipped.

              Unassigned Unassigned
              ooooorz Hanlin Liu (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: