Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-2716

Dropped columns cause NPE in SqlServerConnector

    XMLWordPrintable

Details

    • False
    • False
    • Undefined
    • Hide
      1. Create table with a bigint ID column PK and some random other fields
      2. Enable CDC for this table
      3. Look at the capture instance for a field that exists to the left of ID (i.e. between the first 5 CDC columns and the ID column)
      4. Drop that column from the source table and don't re-create the capture instance
      5. Spin up a sql server connector against this table with a "column.include.list" set to just the ID column.
      6. Insert a row into the table to receive the NPE in the connector
      Show
      Create table with a bigint ID column PK and some random other fields Enable CDC for this table Look at the capture instance for a field that exists to the left of ID (i.e. between the first 5 CDC columns and the ID column) Drop that column from the source table and don't re-create the capture instance Spin up a sql server connector against this table with a "column.include.list" set to just the ID column. Insert a row into the table to receive the NPE in the connector

    Description

      We use the debezium sql server connector to pull just the primary key from the tables we care about. Sadly our legacy systems do not allow for easy rolling of capture instances on schema changes, so pulling the ID from the table and having downstream processes query for the current state of the row is the best we can do. 

      We have one table that is in the unfortunate situation of having deleted fields in the source table schema that still exist in the capture instance and that field exists before the ID field we are trying to extract. This triggers an NPE from the following line: https://github.com/debezium/debezium/blob/v1.3.0.Final/debezium-connector-sqlserver/src/main/java/io/debezium/connector/sqlserver/SqlServerChangeTablePointer.java#L136. 

       

      We know the solution is to roll the capture instance, but I would hope the connector could support this situation since dropping a column and then waiting to roll the capture instance will be a common situation for us. 

       

      2020-10-30 15:17:45,565 ERROR WorkerSourceTask{id=******} Task threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask) [task-thread-********]
org.apache.kafka.connect.errors.ConnectException: An exception occurred in the change event producer. This connector will be stopped.
	at io.debezium.pipeline.ErrorHandler.setProducerThrowable(ErrorHandler.java:42)
	at io.debezium.connector.sqlserver.SqlServerStreamingChangeEventSource.execute(SqlServerStreamingChangeEventSource.java:283)
	at io.debezium.pipeline.ChangeEventSourceCoordinator.streamEvents(ChangeEventSourceCoordinator.java:140)
	at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:113)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
	at io.debezium.connector.sqlserver.SqlServerChangeTablePointer$IndicesMapping.<init>(SqlServerChangeTablePointer.java:136)
	at io.debezium.connector.sqlserver.SqlServerChangeTablePointer.createResultSetMapper(SqlServerChangeTablePointer.java:108)
	at io.debezium.connector.sqlserver.SqlServerChangeTablePointer.getData(SqlServerChangeTablePointer.java:87)
	at io.debezium.connector.sqlserver.SqlServerStreamingChangeEventSource.lambda$execute$1(SqlServerStreamingChangeEventSource.java:241)
	at io.debezium.jdbc.JdbcConnection.prepareQuery(JdbcConnection.java:588)
	at io.debezium.connector.sqlserver.SqlServerConnection.getChangesForTables(SqlServerConnection.java:217)
	at io.debezium.connector.sqlserver.SqlServerStreamingChangeEventSource.execute(SqlServerStreamingChangeEventSource.java:164)
	... 7 more

      Attachments

        Activity

          People

            jpechane Jiri Pechanec
            jgray1206 John Gray (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: