Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-7797

Incremental snapshot fails with NPE if surrogate key doesn't exist

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major Major
    • 2.7.2.Final, 3.0.0.CR1
    • 2.5.4.Final, 2.6.1.Final
    • core-library
    • None

      Bug report

      An incremental snapshot will fail with a NullPointerException if the provided surrogate key ("surrogate-key") doesn't exist in the provided tables ("data-collections").

       

      io.debezium.DebeziumException: Database error while executing incremental snapshot for table 'DataCollection{id=schemaname.tablename, additionalCondition=, surrogateKey=SURROGATE_KEY}'
      Caused by: java.lang.NullPointerException
              at io.debezium.pipeline.source.snapshot.incremental.AbstractChunkQueryBuilder.lambda$buildMaxPrimaryKeyQuery$3(AbstractChunkQueryBuilder.java:225)

       

       

      https://debezium.io/blog/2023/04/20/debezium-2-2-final-released/#new-incremental-snapshots-surrogate-key

      What Debezium connector do you use and what version?

      Debezium for MySQL 2.6.1.Final

      What is the connector configuration?

      -

      What is the captured database version and mode of depoyment?

      Database: Amazon RDS for MariaDB 10.5.21

      Deployment of Debezium: MSK Connect

      What behaviour do you expect?

      The snapshot process should emit a proper error message and not start if the provided surrogate key doesn't exist.

      What behaviour do you see?

      The snapshot process fails with a NullPointerException if the provided surrogate key doesn't exist.

      Do you see the same behaviour using the latest released Debezium version?

      Yes, also happens on main branch.

      Do you have the connector logs, ideally from start till finish?

       

      io.debezium.DebeziumException: Database error while executing incremental snapshot for table 'DataCollection{id=schemaname.tablename, additionalCondition=, surrogateKey=SURROGATE_KEY}'
          at io.debezium.pipeline.source.snapshot.incremental.AbstractIncrementalSnapshotChangeEventSource.readChunk(AbstractIncrementalSnapshotChangeEventSource.java:337)
          at io.debezium.pipeline.source.snapshot.incremental.AbstractIncrementalSnapshotChangeEventSource.addDataCollectionNamesToSnapshot(AbstractIncrementalSnapshotChangeEventSource.java:451)
          at io.debezium.pipeline.signal.actions.snapshotting.ExecuteSnapshot.arrived(ExecuteSnapshot.java:78)
          at io.debezium.pipeline.signal.SignalProcessor.processSignal(SignalProcessor.java:191)
          at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
          at java.base/java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(AbstractList.java:720)
          at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658)
          at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:274)
          at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
          at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177)
          at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655)
          at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
          at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
          at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
          at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
          at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
          at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)
          at io.debezium.pipeline.signal.SignalProcessor.lambda$processSourceSignal$4(SignalProcessor.java:155)
          at io.debezium.pipeline.signal.SignalProcessor.executeWithSemaphore(SignalProcessor.java:165)
          at io.debezium.pipeline.signal.SignalProcessor.processSourceSignal(SignalProcessor.java:149)
          at io.debezium.pipeline.EventDispatcher$2.changeRecord(EventDispatcher.java:290)
          at io.debezium.relational.RelationalChangeRecordEmitter.emitCreateRecord(RelationalChangeRecordEmitter.java:79)
          at io.debezium.relational.RelationalChangeRecordEmitter.emitChangeRecords(RelationalChangeRecordEmitter.java:47)
          at io.debezium.pipeline.EventDispatcher.dispatchDataChangeEvent(EventDispatcher.java:271)
          at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.lambda$handleInsert$5(MySqlStreamingChangeEventSource.java:713)
          at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleChange(MySqlStreamingChangeEventSource.java:781)
          at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleInsert(MySqlStreamingChangeEventSource.java:710)
          at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.lambda$execute$20(MySqlStreamingChangeEventSource.java:854)
          at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleEvent(MySqlStreamingChangeEventSource.java:285)
          at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.lambda$execute$30(MySqlStreamingChangeEventSource.java:880)
          at com.github.shyiko.mysql.binlog.BinaryLogClient.notifyEventListeners(BinaryLogClient.java:1263)
          at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:1089)
          at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:648)
          at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:949)
          at java.base/java.lang.Thread.run(Thread.java:829)
      Caused by: java.lang.NullPointerException
          at io.debezium.pipeline.source.snapshot.incremental.AbstractChunkQueryBuilder.lambda$buildMaxPrimaryKeyQuery$3(AbstractChunkQueryBuilder.java:225)
          at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
          at java.base/java.util.Collections$2.tryAdvance(Collections.java:4747)
          at java.base/java.util.Collections$2.forEachRemaining(Collections.java:4755)
          at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
          at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
          at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
          at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
          at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)
      eyQuery(AbstractChunkQueryBuilder.java:226)
          at io.debezium.pipeline.source.snapshot.incremental.AbstractIncrementalSnapshotChangeEventSource.readChunk(AbstractIncrementalSnapshotChangeEventSource.java:276)
          ... 34 more 

       

       

            Unassigned Unassigned
            jochen-redhat Jochen Schalanda (Inactive)
            Votes:
            1 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: