Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-8214

duplicate change events on ibmi connector

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • 3.0.3.Final
    • None
    • ibmi-connector
    • None
    • False
    • None
    • False
    • Low

      Bug report

      What Debezium connector do you use and what version?

      debezium-connector-ibmi-3.0.0.Alpha1

      What is the connector configuration?

      <Your answer>

      What is the captured database version and mode of deployment?

      (E.g. on-premises, with a specific cloud provider, etc.)

      <Your answer>

      What behavior do you expect?

      updating a single row should result in one kafka event

      What behavior do you see?

      duplicate events for the same entry the second one missing the before image

      Do you see the same behaviour using the latest released Debezium version?

      yes 3.0.0.Alpha1

      Do you have the connector logs, ideally from start till finish?

      Looking at the logs:

       

      ```

      2024-09-03 13:24:31,711 DEBUG [io.deb.con.db2.As400StreamingChangeEventSource] (debezium-as400rpcconnector-db2-change-event-source-coordinator) next event: 2024-09-03T13:24:38.543824Z - 543187362 type: UB table: TEST_TABLE
      2024-09-03 13:24:31,711 DEBUG [io.deb.con.db2.As400StreamingChangeEventSource] (debezium-as400rpcconnector-db2-change-event-source-coordinator) next event: 2024-09-03T13:24:38.543824Z - 543187363 type: UP table: TEST_TABLE
      2024-09-03 13:24:31,711 DEBUG [io.deb.con.db2.As400StreamingChangeEventSource] (debezium-as400rpcconnector-db2-change-event-source-coordinator) found before image for DEBEZIUM-TEST_TABLE
      2024-09-03 13:24:31,711 DEBUG [io.deb.con.db2.As400StreamingChangeEventSource] (debezium-as400rpcconnector-db2-change-event-source-coordinator) update event id 543187363 tx 0 table B7031210.DEBEZIUM.TEST_TABLE
      2024-09-03 13:24:45,208 DEBUG [io.deb.con.db2.As400StreamingChangeEventSource] (debezium-as400rpcconnector-db2-change-event-source-coordinator) next event: 2024-09-03T13:24:38.543824Z - 543187363 type: UP table: TEST_TABLE
      2024-09-03 13:24:45,208 DEBUG [io.deb.con.db2.As400StreamingChangeEventSource] (debezium-as400rpcconnector-db2-change-event-source-coordinator) before image not found for DEBEZIUM-TEST_TABLE

      ```

       

       

      it looks like we see the before image of the change offset 543187362

      then the after image offset 543187363 and we tie the two events together: found before image for...
      Then we find the after image again at the same offset 543187363 and this time we don't find the before image (which is correct as we deleted it the first time we found it)

      I think this is a bug around marking the entry as processed. The API gives us a continuation offset - where to start next if there is more data or it's blank if there is no more data.
      Our start position reflects this it's either set to the continuation offset or if we don't get one it's the last entry we processed.
      If there is no continuation offset it should flag the last offset as processed so we skip it on the next call. I guess this isn't happening.

       

      How to reproduce the issue using our tutorial deployment?

      <Your answer>

      Feature request or enhancement

      For feature requests or enhancements, provide this information, please:

      Which use case/requirement will be addressed by the proposed feature?

      <Your answer>

      Implementation ideas (optional)

      almost certainly a bug in the `is processed` flag

      I wonder if we should remove the handling of the continuation offset and always have the processed flag set to true

              Unassigned Unassigned
              msillencefnz Martin Sillence
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated: