Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-7697

After the initial deployment of Debezium, if a new table is added to MSSQL, its schema is was captured

XMLWordPrintable

      What Debezium connector do you use and what version?

       - Debezium connector for SQL Server version 2.5.3 Final

      What is the connector configuration?

       

      {
        "name": "debezium-Source-Connector-MsSql-test_internalTopic-002",
        "config": {
        "connector.class":
        "io.debezium.connector.sqlserver.SqlServerConnector",
      
        "database.encrypt": "false",
        "database.hostname": "mssql.xyz",
        "database.port": "1433",
        "database.names": "BAY_POC",
        "database.user": "user",
        "database.password": "password",
        "table.include.list": "dbo.WorkProfile",
      
        "topic.prefix": "test_internalTopic-002",
        "schema.history.internal.kafka.bootstrap.servers": "broker.xyz:9092", 
        "schema.history.internal.kafka.topic": "test_internalTopic-002_schemahistory",
      
        "tombstones.on.delete": "false",  
        "time.precision.mode": "connect",
      
        "key.converter": "io.confluent.connect.avro.AvroConverter",
        "value.converter" : "io.confluent.connect.avro.AvroConverter",
        "value.converter.schema.registry.url" : "http://schema.xyz:8081",
        "key.converter.schema.registry.url" : "http://schema.xyz:8081",  
      
        "transforms": "unwrap",
        "transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState",
        "transforms.unwrap.delete.handling.mode": "rewrite"  
       }
      } 

       

      What is the captured database version?

       - Microsoft SQL Server 2019

      What behaviour do you expect?

       - After the initial deployment of Debezium, if a new table is added to MSSQL, its schema is not captured.

      What behaviour do you see?

       let me explain what I've tested:

        1. On MSSQL, I have tables WorkProfile and WorkProfile_2, both with CDC enabled.
        2. I deployed Debezium with table.include.list set to table WorkProfile only and left schema.history.internal.store.only.captured.tables.ddl at its default value, which is false.
        3. After deploying Debezium in step 2, I found that the schema history topic had two record, which described the schema of table WorkProfile and WorkProfile_2 (Which is the correct behavior according to what we want.)
        4. I created a new table WorkProfile_3 on MSSQL and enabled CDC.
        5. I updated table.include.list to include tables WorkProfile and WorkProfile_3 and then re-deployed Debezium.
        6. After re-deploying Debezium in step 5, I found that the schema history topic had one more record, which described the schema of table WorkProfile_3. So, at this point, the schema history topic had a total of 3 records.

       

       The bugs I encountered were:

        - After the initial deployment of Debezium, if a new table is added to MSSQL, its schema is not captured. However, in the case I encountered, the schema of the newly added table was captured.

      Note : From 5., if I don't add WorkProfile_3 to the table.include.list, Debezium won't capture the schema of the WorkProfile_3 table. This means that in 6., the schema history topic won't have a new record added. That is, there will be only 2 records in the schema history topic, which describe the schema of WorkProfile and WorkProfile_2.

      Do you see the same behaviour using the latest relesead Debezium version?

       -  Yes, I encountered this behavior in the Debezium connector for SQL Server version 2.5.3 Final, which is the latest stable version.

      For further information

      https://debezium.zulipchat.com/#narrow/stream/348252-community-sqlserver/topic/Adding.20a.20table.20.20after.20deploy.20debezium

              Unassigned Unassigned
              monpunsak Punsak Incham (Inactive)
              Punsak Incham (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: