Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-5811

Postgres: Disable LSN confirmation to database

    XMLWordPrintable

Details

    Description

      Feature request or enhancement

      For feature requests or enhancements, provide this information, please:

      Which use case/requirement will be addressed by the proposed feature?

      I don't want Debezium to confirm LSN of processed events to the database i.e. Debezium should not update the `pg_replication_slots`. Currently debezium flushes LSN in the source postgres database as it reads records from the WAL logs. We send these records downstream in data warehouses. In case of a downstream failure (while writing the data into the warehouse) we need to be able to start from previous point but since debezium commits LSN as it reads this is not possible cause the WAL log is deleted by the postgres database.

      Implementation ideas (optional)

      I think this option can be disabled via a new config property. Based on the code that I read,
      we could do this inside the `PostgresReplicationConnection` class in the `flushLsn` method. The class has access to `private final PostgresConnectorConfig connectorConfig;` so we could change the `flushLsn` method to be something like
      ```

        @Override
        public void flushLsn(Lsn lsn) throws SQLException {
          if (connectorConfig.shouldFlushLsn())

      { //check in the connector config if flushing is required       doFlushLsn(lsn);     }

        }

        private void doFlushLsn(Lsn lsn) throws SQLException

      {     stream.setFlushedLSN(lsn.asLogSequenceNumber());     stream.setAppliedLSN(lsn.asLogSequenceNumber());     stream.forceUpdateStatus();   }

      ```

      We will have to add a new property in the postgres connector but overall this seems doable.

      We can also add a separate warning that if the user has the set the property to skip the LSN flush on the source database, a warning is emitted saying something like `Debezium is not going to commit the LSN for the replication slot which would result in building of WAL logs and disk space issues would arise. It is expected of the user to handle it separately`

      {{}}

      1) Verify the warning is not emitted when flush is disabled
      2) Make sure a warning is emitted when this option is enabled
      3) Have a test to verify the fetaure
      4) Have the new option documented

      {{}}

      {{}}

      Also For the item 1) - it is not necessary to change the code, first check if the code is even needed

      Ref : https://debezium.zulipchat.com/#narrow/stream/302529-users/topic/Postgres.3A.20Disable.20LSN.20confirmation.20to.20database

      Attachments

        Activity

          People

            Unassigned Unassigned
            subodh1810 Subodh Chaturvedi (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: