Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-5739

Postgres connector results in silent data loss if replication slot is recreated

XMLWordPrintable

    • False
    • None
    • False

       

      Bug report

      The Debezium Postgres connector can result in data loss if the replication slot is re-created for some reason. There should be an option to fail the connector instead of silent data loss when this happens.

       

      Steps to reproduce:

       

      • Configure Debezium Postgres Connector with the default snapshot.mode `initial`
      • Run the connector and let it perform the snapshot, sync tables and update the offset
      • Stop the connector
      • On the Postgres instance
        • Drop the `debezium` replication slot
        • Insert new rows to the table (say, r1, r2)
        • Create a new replication slot with `debezium`
      • Start the connector
      • Insert some more rows  (say r3, r4)
        • Observe that the the connector starts syncing from rows r3, r4 and missing the rows r1 and r2. This causes silent data loss. There should be an option to fail the connector when this happens.

       

      What Debezium connector do you use and what version?

      Postgres Connector, version 1.9

      What is the connector configuration?

      default configuration

      What is the captured database version and mode of depoyment?

      Postgres 11, Docker

      What behaviour do you expect?

      Connector to fail

      What behaviour do you see?

      Connector continues to work, but with data loss

      Do you see the same behaviour using the latest relesead Debezium version?

      Haven't tested, but looking at the code it has the same problem

      Do you have the connector logs, ideally from start till finish?

      The integration test demonstrates the issue.

      How to reproduce the issue using our tutorial deployment?

      integration test 

      Feature request or enhancement

      For feature requests or enhancements, provide this information, please:

      Which use case/requirement will be addressed by the proposed feature?

      <Your answer>

      Implementation ideas (optional)

      <Your answer>

              Unassigned Unassigned
              prburgu Praveen Burgu
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: