Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-1891

SQL Server - Snapshot of newly enabled CDC tables

    XMLWordPrintable

Details

    • Enhancement
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Duplicate Issue
    • 1.0.3.Final
    • None
    • sqlserver-connector
    • None

    Description

      Hello!

      First thank you for the nice product. Great Job!

      In the case of Sql Server, it is nice that, while running, Debezium can detect new tables enabled for CDC. However, Debezium doesn't take a snapshot of those newly enabled tables and only push changes. Attached is a small procedure to reproduce this.

      This is problematic for us because :
      (1) we plan to deploy Kafka domain by domain, not all the tables in one shot,
      (2) new tables are regularly created and may or may not be enabled for CDC in a latter time so that, we can't enable CDC by default for all newly created tables, and we need the snapshot to occur when the CDC is effectively enabled.

      This Enhancement request is to ask for having the possbility to specify if snapshot are performed or not on newly CDC enabled tables. (may be a new parameter or a new value for the snapshot.mode property ?). This capability is very critical for our use case.

      According to the log message we got, this is the code triggered when a new table is detected:
      -> SqlServerStreamingChangeEventSource.java :
      line 330: if (schema.tableFor(currentTable.getSourceTableId()) == null) {
      line 331: LOGGER.info("Table {} is new to be monitored by capture instance {}", [...]
      -> Log
      [2020-03-20 18:24:23,810] INFO Table SandBox.dbo.TableC is new to be monitored by capture instance dbo_TableC (io.debezium.connector.sqlserver.SqlServerStreamingChangeEventSource:328)
      [2020-03-20 18:24:25,007] INFO Already applied 6 database changes (io.debezium.relational.history.DatabaseHistoryMetrics:135)
      [2020-03-20 18:24:25,007] INFO Schema will be changed for Capture instance "dbo_TableC" [sourceTableId=SandBox.dbo.TableC, changeTableId=SandBox.cdc.dbo_TableC_CT, startLsn=000001b7:00018fa7:002c, changeTableObjectId=1390627997, stopLsn=NULL] (io.debezium.connector.sqlserver.SqlServerStreamingChangeEventSource:154)

      Thank you for your consideration.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              sarye Sarye HADDADI (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: