I'm using the PostgreSQL connector (version 1.1.0) and am currently investigating data loss occurring when the connector transitions from snapshot to WAL/replication.
The scenario is like this:
- I start the connector to monitor one table with some data in it
- immediately as the connector starts I insert 2000+ rows into that table (problem doesn't seem to really happen with smaller amounts of data)
- I check if the connector ends up publishing all the data that was in the table + what I've inserted
Almost always there is a ton of rows/events missing. I've tried various combinations of the `snapshot.mode` parameter, both with letting the connector to create the replication slot itself or creating it myself, before starting the connector.
The only version which almost works is `snapshot.mode=export` and letting the connector create the replication slot itself. With this combination I always end up with a single lost row, regardless if I insert 2000, or 10000 or however many rows. The missing row seems exactly the wal event where snapshotting ends and streaming from the WAL starts. Seems awfully lot like an off-by-one bug in the connector... I don't know what causes the massive loss with other snapshotting modes.