Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-9232

Mongodb incremental snapshot is not honoring additional conditions

XMLWordPrintable

      What Debezium connector do you use and what version?

      Debezium mongodb connector 2.7.4.Final

      What is the connector configuration?

       

       

      {
          "name": "mongodb-connector",
          "config": {
              "connector.class" : "io.debezium.connector.mongodb.MongoDbConnector",
              "tasks.max" : "1",
              "mongodb.connection.string" : "mongodb://dbz_mongo:27017/?replicaSet=rs0",
              "topic.prefix" : "dbserver1",
              "mongodb.user" : "debezium",
              "mongodb.password" : "dbz",
              "collection.include.list": "mydb.bugs",
              "capture.mode": "change_streams_update_full_with_pre_image",
              "capture.mode.full.update.type": "post_image",
              "snapshot.mode": "no_data",
              "signal.enabled.channels": "source",
              "signal.data.collection": "mydb.debezium_signal",
              "transforms": "unwrap,renamekeyid",
              "transforms.unwrap.type": "io.debezium.connector.mongodb.transforms.ExtractNewDocumentState",
              "transforms.unwrap.delete.tombstone.handling.mode": "rewrite",
              "transforms.unwrap.delete.tombstone.handling.mode.rewrite-with-id": "true",
              "transforms.unwrap.add.fields": "op,ts_ms,source.ts_ms,source.snapshot,source.db",
              "transforms.unwrap.add.fields.prefix": "__md.",
              "transforms.unwrap.flatten.struct": "true",
              "transforms.unwrap.flatten.struct.delimiter": "_",
              "transforms.renamekeyid.type": "org.apache.kafka.connect.transforms.ReplaceField$Key",
              "transforms.renamekeyid.renames": "id:_id"
          }
      } 

       

       

       

      What is the captured database version and mode of deployment?

      Locally hosted mongodb 6.0

      What behavior do you expect?

      Filter conditions specified in `additional-conditions` are honored during incremental snapshot. 

      What behavior do you see?

      Below is the signal I am using. Observe that filter specifies a specific bug id, but the events sent to topic by debezium incudes documents which do match the filter predicate. The filter value being passed is not being applied to initial chunk it tries to fetch. I think its due to startKey being null for first chunk here - https://github.com/debezium/debezium/blob/main/debezium-connector-mongodb/src/main/java/io/debezium/connector/mongodb/snapshot/MongoDbIncrementalSnapshotChangeEventSource.java#L629

      {
          "type": "execute-snapshot",
          "data": {
              "type": "incremental",
              "data-collections": ["mydb.bugs"],
              "additional-conditions": [{
                  "data-collection": "mydb.bugs",
                  "filter": "{\"bug_id\": 3484500}"
              }]
          }
      } 

       

       

      Do you see the same behaviour using the latest released Debezium version?

      Have not tried with newer version.

      Do you have the connector logs, ideally from start till finish?

      Debug logs from connect for queries it is making to source collection.

      ...
      2025-07-17 10:36:49,675 DEBUG  MongoDB|dbserver1|streaming  	 For collection 'mydb.bugs' using query: '{"_id": {"$lte": {"$oid": "67ffe6aff42055750e35d474"}}}', key: 'null', maximum key: '[67ffe6aff42055750e35d474]' to get all _id fields   [io.debezium.connector.mongodb.snapshot.MongoDbIncrementalSnapshotChangeEventSource]
      ...
      2025-07-17 10:36:49,791 DEBUG  MongoDB|dbserver1|streaming  	 For collection 'mydb.bugs' using query: '{"$and": [{"_id": {"$gt": {"$oid": "67ffe6aff42055750e35d474"}}}, {"_id": {"$lte": {"$oid": "67ffe6aff42055750e35d474"}}}, {"bug_id": 3484500}]}', key: '[67ffe6aff42055750e35d474]', maximum key: '[67ffe6aff42055750e35d474]' to get all _id fields   [io.debezium.connector.mongodb.snapshot.MongoDbIncrementalSnapshotChangeEventSource]
       

       

       

      How to reproduce the issue using our tutorial deployment?

      • Provide filter value in signal's `additional-conditions` for incremental snapshot.

       

       

              ccranfor@redhat.com Chris Cranford
              suchak-dwivedi Suchak Dwivedi (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: