-
Enhancement
-
Resolution: Obsolete
-
Major
-
None
-
False
-
-
False
The CDC tables for db2 will fill up until the disk is full. There should be a way to set the connector or asncap process to prune these tables to prevent this from happening
Based off the following conversation from debezium.zulipchat.com
Daniel Silverman: Does anyone have a good way of cleaning out the old data in the CDC tables? Currently using
DELETE FROM ASNCDC.CDC_TABLE
WHERE IBMSNAP_COMMITSEQ < (
SELECT min(RESTART_SEQ) FROM ASNCDC.IBMSNAP_CAPMON);
to delete old data but that could delete data not read yet by the Kafka connector and if the asncap process hasn't been restarted in awhile then it won't delete any data
Chris Cranford: I am not aware of any @Daniel silverman but I do wonder if we could perhaps use a similar hook that we have for PG and Oracle XStream where based on the Kafka commit positions, we could either:
- Prune the tables automatically
- Write a value into another metadata table that you could use to know whats committed easily
Chris Cranford: Here's the PG example of what I am referring to here.
Chris Cranford: It might be great to raise a Jira enhancement request, https://issues.redhat.com/projects/DBZ.
What Debezium connector do you use and what version?
Debezium server 3.0.0.Final Docker container
What is the connector configuration?
DEBEZIUM_SINK_TYPE: eventhubs
DEBEZIUM_FORMAT_VALUE: json
DEBEZIUM_SOURCE_CONNECTOR_CLASS: io.debezium.connector.db2.Db2Connector
DEBEZIUM_SOURCE_SCHEMA_HISTORY_INTERNAL: io.debezium.relational.history.MemorySchemaHistory
What is the captured database version and mode of deployment?
on-premises IBM DB2 v11.1.4.6 on Solaris Sparc
What behavior do you expect?
The CDC tables should not fill infinitely, Debezium should prune the tables the same way it prunes IBMSNAP_CAPTRACE & IBMSNAP_CAPMON tables
What behavior do you see?
There is nothing that prunes the CDC tables for old data and must be manually removed
Which use case/requirement will be addressed by the proposed feature?
Normal usage so that the disk does not fill up infinitely
Implementation ideas (optional)
Perhaps a setting for how many records it should preserve or something similar