Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-2597

ConcurrentModificationException during exporting data for a mongodb collection in a sharded cluster

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Critical
    • 1.4.0.Alpha1
    • 1.2.1.Final, 1.2.5.Final
    • mongodb-connector
    • None

    Description

      Settings,

      mongodb.hosts: "host and port for mongos"
      mongodb.name: "internal.int.v1"
      mongodb.user: "username"
      mongodb.password: "password"
      mongodb.ssl.enabled: "false"
      database.whitelist: "database"
      collection.whitelist: "database.collection"
      mongodb.members.auto.discover: "true"
      

       

      Looks like multiple events are trying simultaneously to modify https://github.com/debezium/debezium/blob/v1.2.5.Final/debezium-connector-mongodb/src/main/java/io/debezium/connector/mongodb/MongoDbSchema.java#L59 that is not synchronised.

       

      Connector throws out java.util.ConcurrentModificationException when setting up tasks

      2020-09-24 15:52:51,451 INFO     Exporting data for collection 'mongodb-sharded-shard-0.database.collection' (io.debezium.connector.mongodb.MongoDbSnapshotChangeEventSource) [debezium-mongodbconnector-internal.int.v1-replicator-snapshot-0]
      2020-09-24 15:52:51,494 INFO     Exporting data for collection 'mongodb-sharded-shard-1.database.collection' (io.debezium.connector.mongodb.MongoDbSnapshotChangeEventSource) [debezium-mongodbconnector-internal.int.v1-replicator-snapshot-1]
      2020-09-24 15:52:51,660 ERROR Error while attempting to sync 'mongodb-sharded-shard-0.database.collection':  (io.debezium.connector.mongodb.MongoDbSnapshotChangeEventSource) [debezium-mongodbconnector-internal.int.v1-replicator-snapshot-0]
      java.util.ConcurrentModificationException
              at java.base/java.util.HashMap.computeIfAbsent(HashMap.java:1134)
              at io.debezium.connector.mongodb.MongoDbSchema.schemaFor(MongoDbSchema.java:74)
              at io.debezium.connector.mongodb.MongoDbSchema.schemaFor(MongoDbSchema.java:39)
              at io.debezium.pipeline.EventDispatcher.dispatchSnapshotEvent(EventDispatcher.java:139)
              at io.debezium.connector.mongodb.MongoDbSnapshotChangeEventSource.lambda$createDataEventsForCollection$8(MongoDbSnapshotChangeEventSource.java:466)
              at io.debezium.connector.mongodb.ConnectionContext$MongoPrimary.executeBlocking(ConnectionContext.java:339)
              at io.debezium.connector.mongodb.MongoDbSnapshotChangeEventSource.createDataEventsForCollection(MongoDbSnapshotChangeEventSource.java:442)
              at io.debezium.connector.mongodb.MongoDbSnapshotChangeEventSource.createDataEventsForReplicaSet(MongoDbSnapshotChangeEventSource.java:422)
              at io.debezium.connector.mongodb.MongoDbSnapshotChangeEventSource.createDataEvents(MongoDbSnapshotChangeEventSource.java:321)
              at io.debezium.connector.mongodb.MongoDbSnapshotChangeEventSource.snapshotReplicaSet(MongoDbSnapshotChangeEventSource.java:219)
              at io.debezium.connector.mongodb.MongoDbSnapshotChangeEventSource.lambda$null$0(MongoDbSnapshotChangeEventSource.java:115)
              at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
              at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
              at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
              at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
              at java.base/java.lang.Thread.run(Thread.java:834)
      2020-09-24 15:52:51,761 ERROR Snapshot for replica set mongodb-sharded-shard-0 failed (io.debezium.connector.mongodb.MongoDbSnapshotChangeEventSource) [debezium-mongodbconnector-internal.int.v1-replicator-snapshot-0]
      org.apache.kafka.connect.errors.ConnectException: Error while attempting to sync 'mongodb-sharded-shard-0.database.collection'
              at io.debezium.connector.mongodb.MongoDbSnapshotChangeEventSource.lambda$establishConnectionToPrimary$3(MongoDbSnapshotChangeEventSource.java:238)
              at io.debezium.connector.mongodb.ConnectionContext$MongoPrimary.executeBlocking(ConnectionContext.java:346)
              at io.debezium.connector.mongodb.MongoDbSnapshotChangeEventSource.createDataEventsForCollection(MongoDbSnapshotChangeEventSource.java:442)
              at io.debezium.connector.mongodb.MongoDbSnapshotChangeEventSource.createDataEventsForReplicaSet(MongoDbSnapshotChangeEventSource.java:422)
              at io.debezium.connector.mongodb.MongoDbSnapshotChangeEventSource.createDataEvents(MongoDbSnapshotChangeEventSource.java:321)
              at io.debezium.connector.mongodb.MongoDbSnapshotChangeEventSource.snapshotReplicaSet(MongoDbSnapshotChangeEventSource.java:219)
              at io.debezium.connector.mongodb.MongoDbSnapshotChangeEventSource.lambda$null$0(MongoDbSnapshotChangeEventSource.java:115)
              at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
              at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
              at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
              at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
              at java.base/java.lang.Thread.run(Thread.java:834)
      Caused by: java.util.ConcurrentModificationException
              at java.base/java.util.HashMap.computeIfAbsent(HashMap.java:1134)
              at io.debezium.connector.mongodb.MongoDbSchema.schemaFor(MongoDbSchema.java:74)
              at io.debezium.connector.mongodb.MongoDbSchema.schemaFor(MongoDbSchema.java:39)
              at io.debezium.pipeline.EventDispatcher.dispatchSnapshotEvent(EventDispatcher.java:139)
              at io.debezium.connector.mongodb.MongoDbSnapshotChangeEventSource.lambda$createDataEventsForCollection$8(MongoDbSnapshotChangeEventSource.java:466)
              at io.debezium.connector.mongodb.ConnectionContext$MongoPrimary.executeBlocking(ConnectionContext.java:339)
              ... 10 more

       

      Attachments

        Activity

          People

            ccranfor@redhat.com Chris Cranford
            rodrigofkuntzer@gmail.com Rodrigo Fior Kuntzer (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: