Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-2329

Change events lost when connnector is restarted while processing transaction with PK update

    Details

    • Steps to Reproduce:
      Hide

      connection.execute("INSERT INTO tableb (id, colb) values (1,'1')");
      connection.execute("INSERT INTO tableb (id, colb) values (2,'2')");
      connection.execute("INSERT INTO tableb (id, colb) values (3,'3')");
      connection.execute("INSERT INTO tableb (id, colb) values (4,'4')");
      connection.execute("INSERT INTO tableb (id, colb) values (5,'5')");
      connection.execute("UPDATE tableb set id = colb + 300");
      connection.execute("UPDATE tableb set id = colb + 300");

      select CONVERT(VARCHAR(1000), [__$start_lsn], 2) as start_lsn, CONVERT(VARCHAR(1000), [__$seqval], 2) as seqval, [__$operation],[__$command_id],id,colb from testDb.cdc.dbo_tableb_CT
      

       

      commit offset and restart connector on 19th row.
      after starting connector it will skip not processed records: 21, 22, 23 

       

       

      Show
      connection.execute("INSERT INTO tableb (id, colb) values (1,'1')"); connection.execute("INSERT INTO tableb (id, colb) values (2,'2')"); connection.execute("INSERT INTO tableb (id, colb) values (3,'3')"); connection.execute("INSERT INTO tableb (id, colb) values (4,'4')"); connection.execute("INSERT INTO tableb (id, colb) values (5,'5')"); connection.execute("UPDATE tableb set id = colb + 300"); connection.execute("UPDATE tableb set id = colb + 300"); select CONVERT(VARCHAR(1000), [__$start_lsn], 2) as start_lsn, CONVERT(VARCHAR(1000), [__$seqval], 2) as seqval, [__$operation],[__$command_id],id,colb from testDb.cdc.dbo_tableb_CT   commit offset and restart connector on 19th row. after starting connector it will skip not processed records: 21, 22, 23     

      Description

      This Bug is the same as  DBZ-1152 

      Sometimes during update of multiple primary keys order of change_lsn is not monotonic.

      Basically, fix for above issue works only for one record with ChangePosition equal to commited offset 

       

      It seems, that event_serial_no should be calculated on common start_lsn not only on records with duplicated change_lsns

       

      I will create PR with tests to reproduce

      Also I have a solution in mind which does not require changes in implementation of event_serial_no

       

        Gliffy Diagrams

          Attachments

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                korzenek Łukasz Korzeniowski
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: