Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-521

Possibility of commit LSN before record is consumed/notified

    XMLWordPrintable

    Details

      Description

      Version: 0.7.1
      Modules: postgres connector, EmbeddedEngine
      Postgres Version: AWS RDS 9.6.5

      There may be chance that LSN committed back to Postgres can be later than what written to offset store. task.commit() on postgres connector (commitRecord() is not implemented) seems to use latest lsn received (e.g. record it puts into queue). Offset (LSN) that is written to the store is the latest record consumed by the consumer which may not be the latest one from the queue.

      I believe postgres may truncate / archive WAL block once LSN is committed?

      To reproduce:
      1) Make consumer (notify callback) slow so that changes are queue up
      2) Start EmbeddedEngine
      3) Make DB changes continuously until commit happens
      4) Check offset store for LSN and postgres committed LSN

        Attachments

          Activity

            People

            Assignee:
            jpechanec Jiri Pechanec
            Reporter:
            thon Thon Mekathikom (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: