Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-9180

Make debezium prune the cdc tables for db2

XMLWordPrintable

    • False
    • Hide

      None

      Show
      None
    • False

      The CDC tables for db2 will fill up until the disk is full. There should be a way to set the connector or asncap process to prune these tables to prevent this from happening

      Based off the following conversation from debezium.zulipchat.com

      Daniel Silverman: Does anyone have a good way of cleaning out the old data in the CDC tables? Currently using
      DELETE FROM ASNCDC.CDC_TABLE
      WHERE IBMSNAP_COMMITSEQ < (
      SELECT min(RESTART_SEQ) FROM ASNCDC.IBMSNAP_CAPMON);
      to delete old data but that could delete data not read yet by the Kafka connector and if the asncap process hasn't been restarted in awhile then it won't delete any data

      Chris Cranford: I am not aware of any @Daniel silverman but I do wonder if we could perhaps use a similar hook that we have for PG and Oracle XStream where based on the Kafka commit positions, we could either:

      • Prune the tables automatically
      • Write a value into another metadata table that you could use to know whats committed easily

      Chris Cranford: Here's the PG example of what I am referring to here.

      Chris Cranford: It might be great to raise a Jira enhancement request, https://issues.redhat.com/projects/DBZ.

      What Debezium connector do you use and what version?

      Debezium server 3.0.0.Final Docker container 

      What is the connector configuration?

      DEBEZIUM_SINK_TYPE: eventhubs
      DEBEZIUM_FORMAT_VALUE: json
      DEBEZIUM_SOURCE_CONNECTOR_CLASS: io.debezium.connector.db2.Db2Connector
      DEBEZIUM_SOURCE_SCHEMA_HISTORY_INTERNAL: io.debezium.relational.history.MemorySchemaHistory

      What is the captured database version and mode of deployment?

      on-premises IBM DB2 v11.1.4.6 on Solaris Sparc

      What behavior do you expect?

      The CDC tables should not fill infinitely, Debezium should prune the tables the same way it prunes IBMSNAP_CAPTRACE & IBMSNAP_CAPMON tables

      What behavior do you see?

      There is nothing that prunes the CDC tables for old data and must be manually removed

       

      Which use case/requirement will be addressed by the proposed feature?

      Normal usage so that the disk does not fill up infinitely 

      Implementation ideas (optional)

      Perhaps a setting for how many records it should preserve or something similar

              Unassigned Unassigned
              daniel.silverman Daniel Silverman (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: