Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-2288

Postgres connector may skip events during snapshot-streaming transition

XMLWordPrintable

    • Hide
      1. start connector
      2. while it's starting up and still doing its snapshop pump tons of rows into the table it monitors
      3. check what it publishes vs. what went in
      4. every row should be there after a while, either from snapshot or for streaming
      Show
      start connector while it's starting up and still doing its snapshop pump tons of rows into the table it monitors check what it publishes vs. what went in every row should be there after a while, either from snapshot or for streaming

       I'm using the PostgreSQL connector (version 1.1.0) and am currently investigating data loss occurring when the connector transitions from snapshot to WAL/replication. 
       
      The scenario is like this:

      1. I start the connector to monitor one table with some data in it
      2. immediately as the connector starts I insert 2000+ rows into that table (problem doesn't seem to really happen with smaller amounts of data)
      3. I check if the connector ends up publishing all the data that was in the table + what I've inserted
         

      Almost always there is a ton of rows/events missing. I've tried various combinations of the `snapshot.mode` parameter, both with letting the connector to create the replication slot itself or creating it myself, before starting the connector. 
       
      The only version which almost works is `snapshot.mode=export` and letting the connector create the replication slot itself. With this combination I always end up with a single lost row, regardless if I insert 2000, or 10000 or however many rows. The missing row seems exactly the wal event where snapshotting ends and streaming from the WAL starts. Seems awfully lot like an off-by-one bug in the connector... I don't know what causes the massive loss with other snapshotting modes.

              jpechane Jiri Pechanec
              bjozsef-1 Bartok Jozsef (Inactive)
              Votes:
              1 Vote for this issue
              Watchers:
              6 Start watching this issue

                Created:
                Updated:
                Resolved: