Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-521

Possibility of commit LSN before record is consumed/notified

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major Major
    • 0.7.2
    • 0.7.1
    • postgresql-connector
    • None

      Version: 0.7.1
      Modules: postgres connector, EmbeddedEngine
      Postgres Version: AWS RDS 9.6.5

      There may be chance that LSN committed back to Postgres can be later than what written to offset store. task.commit() on postgres connector (commitRecord() is not implemented) seems to use latest lsn received (e.g. record it puts into queue). Offset (LSN) that is written to the store is the latest record consumed by the consumer which may not be the latest one from the queue.

      I believe postgres may truncate / archive WAL block once LSN is committed?

      To reproduce:
      1) Make consumer (notify callback) slow so that changes are queue up
      2) Start EmbeddedEngine
      3) Make DB changes continuously until commit happens
      4) Check offset store for LSN and postgres committed LSN

              jpechane Jiri Pechanec
              thon_jira Thon Mekathikom (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: