Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-2397

Transaction data loss when process restarted

    XMLWordPrintable

Details

    Description

      After upgrading from postgresql 9.6, decoderbufs/debezium 0.7.5 to postgresql 12.3, decoderbufs/debezium 1.1.2.Final we began to notice data loss when the debezium process was restarted while a transaction was open. For example:

      1. Start debezium process
      2. Begin transaction T
      3. Insert record A inside transaction T
      4. Insert record X outside transaction T
      5. Stop debezium process
      6. Insert record B inside transaction T
      7. Insert record Y outside transaction T
      8. Start debezium process
      9. Insert record C inside transaction T
      10. Insert record Z outside transaction T
      11. Commit transaction T
      12. Observe that record A has been dropped from the CDC stream.

      NOTE: `shouldMessageBeSkipped` method in `io.debezium.connector.postgresql.connection.AbstractMessageDecoder` will log and return true when it sees the message for record A.

       

      NOTE: We’ve also been able to confirm that the bug was introduced sometime on or before decoderbufs/debezium 0.9.5.Final

      Attachments

        1. diff
          2 kB
          Stephen Webber

        Issue Links

          Activity

            People

              jpechane Jiri Pechanec
              atongen@gmail.com Andrew Tongen (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: