Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-5329

Transaction IDs in PostgreSQL transaction metadata topics subject to wraparound

    XMLWordPrintable

Details

    Description

      Over at Materialize we realized that we cannot deduplicate Debezium events in constant space—this time with the transaction metadata topic. For historical context, see DBZ-2911, which covered the case of O(1) space deduplication for data topics.

      The problem is that the transaction ID for PostgreSQL databases is a 32-bit integer that wraps around. That means we cannot simply track the highest transaction ID that we've seen, and discard any new transactions with a smaller transaction ID. 

      Instead, we'd need the transactional metadata topic to expose the end LSN of the transaction. The end LSN is a 64-bit integer and so it never wraps around.

      Is that something that would be possible to add? 

      cc jpechane 

      Attachments

        Activity

          People

            vjuranek@redhat.com Vojtech Juranek
            benesch Nikhil Benesch (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: