Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-1033

Mysql binlog reader lost data if restart task when last binlog event is QUERY event.

    XMLWordPrintable

Details

    • Hide

      Reproduce procedure:

      1. create table `test` (id int, val varchar(20));
      2. start dbz connector to sync for table test.
      3. alter table `test` comment 'test1'/'test2' for N times.
      4. restart the task.
      5. insert into `test` VALUES(1, 'test')

      The binlog event for insert will be skipped.

      Show
      Reproduce procedure: 1. create table `test` (id int, val varchar(20)); 2. start dbz connector to sync for table test. 3. alter table `test` comment 'test1'/'test2' for N times. 4. restart the task. 5. insert into `test` VALUES(1, 'test') The binlog event for insert will be skipped.

    Description

      Binlog reader increased the count for restartEventsToSkip each time after it processed binlog event. That count would be reset when there is a 'BEGIN' sql or a 'COMMIT' sql. However, if the task was restarted after processing a DDL event, it would increase the restartEventsToSkip count as well as record the latest binlog position in offset, and the offset will be updated to kafka offset topic by the DDL record. And then, if the task was restarted, it will skip the first N events after restart which should not be skipped.

      Attachments

        Activity

          People

            Unassigned Unassigned
            ooooorz Hanlin Liu (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: