Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-2116

MongoDB connector snapshot NPE in case of document field named "op"

    Details

      Description

      I've started initial sync for all collections except a few blacklisted ones.

      At some point task fails with the following exception:

      org.apache.kafka.connect.errors.ConnectException: An exception occurred in the change event producer. This connector will be stopped.
      	at io.debezium.pipeline.ErrorHandler.setProducerThrowable(ErrorHandler.java:42)
      	at io.debezium.connector.mongodb.MongoDbSnapshotChangeEventSource.lambda$null$0(MongoDbSnapshotChangeEventSource.java:126)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      Caused by: org.apache.kafka.connect.errors.ConnectException: Error while attempting to sync 'rs0.paler_service_sandbox.system.profile'
      	at io.debezium.connector.mongodb.MongoDbSnapshotChangeEventSource.lambda$establishConnectionToPrimary$3(MongoDbSnapshotChangeEventSource.java:235)
      	at io.debezium.connector.mongodb.ConnectionContext$MongoPrimary.executeBlocking(ConnectionContext.java:346)
      	at io.debezium.connector.mongodb.MongoDbSnapshotChangeEventSource.createDataEventsForCollection(MongoDbSnapshotChangeEventSource.java:439)
      	at io.debezium.connector.mongodb.MongoDbSnapshotChangeEventSource.createDataEventsForReplicaSet(MongoDbSnapshotChangeEventSource.java:419)
      	at io.debezium.connector.mongodb.MongoDbSnapshotChangeEventSource.createDataEvents(MongoDbSnapshotChangeEventSource.java:318)
      	at io.debezium.connector.mongodb.MongoDbSnapshotChangeEventSource.snapshotReplicaSet(MongoDbSnapshotChangeEventSource.java:217)
      	at io.debezium.connector.mongodb.MongoDbSnapshotChangeEventSource.lambda$null$0(MongoDbSnapshotChangeEventSource.java:115)
      	... 5 more
      Caused by: java.lang.NullPointerException
      	at io.debezium.pipeline.AbstractChangeRecordEmitter.emitChangeRecords(AbstractChangeRecordEmitter.java:33)
      	at io.debezium.pipeline.EventDispatcher.dispatchSnapshotEvent(EventDispatcher.java:123)
      	at io.debezium.connector.mongodb.MongoDbSnapshotChangeEventSource.lambda$createDataEventsForCollection$8(MongoDbSnapshotChangeEventSource.java:463)
      	at io.debezium.connector.mongodb.ConnectionContext$MongoPrimary.executeBlocking(ConnectionContext.java:339)
      	... 10 more
      

      I don't have the record itself which caused the issue.
      However, I found another JIRA with the similar symptoms:
      https://issues.redhat.com/browse/DBZ-1464

      Looking at the code:

      public class MongoDbChangeRecordEmitter {
      ...
      // it could return null here
          protected Operation getOperation() {
              if (oplogEvent.getString("op") == null) {
                  return Operation.READ;
              }
              return OPERATION_LITERALS.get(oplogEvent.getString("op"));
          }
      
      public abstract class AbstractChangeRecordEmitter {
      ...
          public void emitChangeRecords(DataCollectionSchema schema, Receiver receiver) throws InterruptedException {
              Operation operation = getOperation();
             // operation could be null and switch will throw NPE
              switch (operation) {
      

      I'll do a bit of a digging with DEBUG logs enabled to find out what is the record payload.

        Gliffy Diagrams

          Attachments

            Activity

              People

              • Assignee:
                gunnar.morling Gunnar Morling
                Reporter:
                serge.travin Serge Travin
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: