Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-9022

Add logic to `ExtractNewRecordState` to convert all deletes to tombstone records

XMLWordPrintable

    • Icon: Enhancement Enhancement
    • Resolution: Done
    • Icon: Major Major
    • 3.2.0.Beta1
    • None
    • core-library
    • None
    • Important

      Bug report

      The code in EventDispatcher can cause a data loss for the tombstone records. Assuming the pseudocode looks like the following:

      ```

      queue.enqueue(record);

       

      if (shouldEmitTombstones)

      {{{}}

        queue.enqueue(tombstoneRecord);

      }

       

      ```

      What Debezium connector do you use and what version?

      Debezium 2.5.2.Final but the issue is present across all versions.

      What behavior do you expect?

      Tombstone records should not be lost in transit.

      What behavior do you see?

      Loss of tombstone records.

      Do you see the same behaviour using the latest released Debezium version?

      Yes.

      Do you have the connector logs, ideally from start till finish?

      No.

      How to reproduce the issue using our tutorial deployment?

      We will need to modify this block a little to introduce a delay to simulate the situation.

      Implementation ideas

      As discussed in the thread #community-general > Potential data loss of tombstone records @ 💬 , we can have a SMT which will convert all the delete records to tombstone records and drop all the tombstone records so that deletes are processed for the downstream application.

       

      A suggestion to add is that instead of a new SMT, `ExtractNewRecordState` could be modified itself to introduce a new configuration property, which, when enabled, will convert the deletes to tombstones. Note that this should be used in conjunction with the `drop.tombstones=true`.

      Finalised approach

      Upon discussion in the community, we concluded that the way to go for this change would be to add another enum option delete.tombstone.handling.mode which will convert delete records to tombstone records and drop all tombstone records.

              Unassigned Unassigned
              vaibhav-yb Vaibhav Kushwaha (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: