Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-2591

db2 connector doesn't allow to reprocess messages

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Critical Critical
    • 1.3.0.Final
    • None
    • db2-connector
    • None
    • False
    • False
    • Undefined
    • Hide

      1) start new db2 connector with one table whitelisted

      2) delete/pause connector

      3) get last message from connect-offsets topic for connector; change commit_lsn,change_lsn to commit_lsn,change_lsn which were done earlier got from source db2 (CDC table) and produce new message into connect-offsets topic

      4) start/resume the connector

      5) only last commit_lsn from table got processed into kafka topic

      Show
      1) start new db2 connector with one table whitelisted 2) delete/pause connector 3) get last message from connect-offsets topic for connector; change commit_lsn,change_lsn to commit_lsn,change_lsn which were done earlier got from source db2 (CDC table) and produce new message into connect-offsets topic 4) start/resume the connector 5) only last commit_lsn from table got processed into kafka topic

      After restart of the connector/kafka-connect/task(s) for connector or trying to reprocess events from source db by changing commit_lsn stored in connect-offsets internal topic only latest commit from CDC table got processed into kafka topic.

      Problem caused by logic for calculation fromLsn which gets getStartLsn as a result which makes impossible to reprocess any messages by connector.

      https://github.com/debezium/debezium-incubator/blob/master/debezium-connector-db2/src/main/java/io/debezium/connector/db2/Db2Connection.java

      final Lsn fromLsn = changeTable.getStartLsn().compareTo(intervalFromLsn) > 0 ? changeTable.getStartLsn() : intervalFromLsn;

       

      00000000:41772d1b:00000df02ee28e5a - commit stored in connect-offsets

      00000000:4177fffa:00000df02f37f443 - commit stored in REGISTER table for CDC_IDDELA_RGETS30S table

      00000000:418d3f4b:00000df0358988c2 - global commit from REGISTER table

       

      Sep 23 14:04:23kafka-connect-6b5bc7f6f-2gqz2kafka-connectINFO|| Last position recorded in offsets is 00000000:41772d1b:00000df02ee28e5a(00000000:00000000:00002e1fbf54decc)[1] [io.debezium.connector.db2.Db2StreamingChangeEventSource]
      Sep 23 14:04:23kafka-connect-6b5bc7f6f-2gqz2kafka-connectTRACE| Current maximum lsn is 00000000:418d3f4b:00000df0358988c2 [io.debezium.connector.db2.Db2Connection]
      Sep 23 14:04:23kafka-connect-6b5bc7f6f-2gqz2kafka-connectTRACE| Getting changes for table Capture instance "CDC_IDDELA_RGETS30S" [sourceTableId=IDDELA.RGETS30S, changeTableId=ASNCDC.CDC_IDDELA_RGETS30S, startLsn=00000000:4177fffa:00000df02f37f443, changeTableObjectId=329492, stopLsn=NULL] in range[00000000:4177fffa:00000df02f37f443, 00000000:418d3f4b:00000df0358988c2] [io.debezium.connector.db2.Db2Connection]

              lga@zurich.ibm.com Luis Garcés-Erice (Inactive)
              G-d042b168-59c9-42f8-9fe4-41515651c4a7 Victar Malinouski (Inactive)
              Luis Garcés-Erice (Inactive)
              Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: