Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-9158

Introduce publish.via.partition.root flag in PostgresCDC Connector

XMLWordPrintable

      In order to make your issue reports as actionable as possible, please provide the following information, depending on the issue type.

      Feature request or enhancement

      publish.via.partition.root addition in connector config

      Which use case/requirement will be addressed by the proposed feature?

      When using the PostgreSQL CDC connector with partitioned tables, CDC behavior depends on how the logical publication is created and whether the publish_via_partition_root flag is set:

      With publish_via_partition_root = true: The connector captures both snapshots and streaming changes for root partitioned tables as well as non-partitioned tables. All operations on the root partitioned table and its partitions are reliably tracked.
      Without publish_via_partition_root: The publication lists each partition separately and streaming changes on the root table after connector creation are not tracked, even when the root table is present in the include list. In this scenario, the connector only captures a snapshot of the root table before connector creation and streaming changes for sub-partition tables if explicitly included, but not via the main topic for the root table. This leads to missing CDC events for operations on partitioned tables, creating inconsistency and data loss risk.
      Solution Approach
      To address this issue and simplify CDC for partitioned tables, the following changes are proposed:

      New connector config flag: Introduce a boolean parameter publish.via.partition.root (default: false) in the connector configuration. When enabled, the connector will create the publication with publish_via_partition_root = true automatically during publication auto-creation.
      Code changes: Update the connector logic to honour the new flag in publication creation, ensuring streaming and snapshot CDC work as expected for both root partitioned tables and standard tables.
      Documentation: Add guidance on the use and effect of the new configuration option, including its impact on partitioned table replication and any migration considerations.
      Testing: Validate the updated connector in all relevant scenarios (partitioned and regular tables, auto-create publication, CDC event delivery).
      This approach enables reliable and automated CDC for partitioned tables without requiring manual intervention or advanced PostgreSQL operations.

      Implementation ideas (optional)

       

              Unassigned Unassigned
              skalla Shubham Kalla (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: