The Debezium Postgres connector can result in data loss if the replication slot is re-created for some reason. There should be an option to fail the connector instead of silent data loss when this happens.
Steps to reproduce:
- Configure Debezium Postgres Connector with the default snapshot.mode `initial`
- Run the connector and let it perform the snapshot, sync tables and update the offset
- Stop the connector
- On the Postgres instance
- Drop the `debezium` replication slot
- Insert new rows to the table (say, r1, r2)
- Create a new replication slot with `debezium`
- Start the connector
- Insert some more rows (say r3, r4)
- Observe that the the connector starts syncing from rows r3, r4 and missing the rows r1 and r2. This causes silent data loss. There should be an option to fail the connector when this happens.
What Debezium connector do you use and what version?
Postgres Connector, version 1.9
What is the connector configuration?
What is the captured database version and mode of depoyment?
Postgres 11, Docker
What behaviour do you expect?
Connector to fail
What behaviour do you see?
Connector continues to work, but with data loss
Do you see the same behaviour using the latest relesead Debezium version?
Haven't tested, but looking at the code it has the same problem
Do you have the connector logs, ideally from start till finish?
The integration test demonstrates the issue.
How to reproduce the issue using our tutorial deployment?
Feature request or enhancement
For feature requests or enhancements, provide this information, please:
Which use case/requirement will be addressed by the proposed feature?
Implementation ideas (optional)