Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-8311

Debezium ChangeStream Size Errors from unincluded collections

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Obsolete
    • Icon: Major Major
    • None
    • None
    • mongodb-connector
    • None
    • False
    • None
    • False

      In order to make your issue reports as actionable as possible, please provide the following information, depending on the issue type.

      Bug report

      For bug reports, provide this information, please:

      What Debezium connector do you use and what version?

      MongoDB, 2.4.0

      What is the connector configuration?

      We observe the same behavior with capture scope and capture target defined or not included. 

      ```

      {             "connector.class": "io.debezium.connector.mongodb.MongoDbConnector",             "collection.include.list": "db.expectedCollection",             "mongodb.connection.mode": "sharded",             "topic.creation.default.partitions": "-1",             "mongodb.connection.string": "...",             "mongodb.password": "...",             "topic.creation.default.cleanup.policy": "compact",             "tasks.max": "1",             "mongodb.user": "...",             "capture.mode": "change_streams_update_full",             "source.struct.version": "v2",             "mongodb.name": "...",             "topic.creation.enable": "false",             "tombstones.on.delete": "false",             "topic.prefix": "...",             "capture.scope": "database",             "capture.target": "...",             "name": "...",             "topic.creation.default.replication.factor": "-1",             "snapshot.mode": "never"       }

      ```

      What is the captured database version and mode of deployment?

      (E.g. on-premises, with a specific cloud provider, etc.)

      MongoDB, 5.0

      What behavior do you expect?

      Continued streaming of changes from "db.expectedCollection". This occurs until a document that throws 'BSONObjectTooLarge' from an unrelated collection. I expect that `collections.include.list` applies a filter to the changestream so this isn't possible. We also are not running snapshots against the database, at all.

      Is debezium self-filtering for collections off of an unfiltered changestream implementation from the Mongo driver and therefore receiving these exceptions? 

      What behavior do you see?

      A 'BSONObjectTooLarge' exception is thrown from an unrelated collection. I am familiar with this exception as we have collections on this DB that will throw similar exceptions. However, we do not care or want to stream changes from those collections. We have too many collections to swap include to an exclude property, and am unsure if the same error would occur.

       

      We can confirm this error is not occurring in actually streaming the data - no topic is created or streamed for any collections besides the expected in the included collections list. 

      Do you see the same behaviour using the latest released Debezium version?

      (Ideally, also verify with latest Alpha/Beta/CR version)

      I cannot use the latest version of debezium in this implementation

      Do you have the connector logs, ideally from start till finish?

      (You might be asked later to provide DEBUG/TRACE level log)

      ```

      io.debezium.DebeziumException: Error while attempting to read from change stream on 'cluster' at io.debezium.connector.mongodb.MongoDbChangeEventSourceFactory.lambda$connectionErrorHandler$1(MongoDbChangeEventSourceFactory.java:126) at io.debezium.connector.mongodb.connection.MongoDbConnection.execute(MongoDbConnection.java:116) at io.debezium.connector.mongodb.connection.MongoDbConnection.execute(MongoDbConnection.java:93) at io.debezium.connector.mongodb.MongoDbStreamingChangeEventSource.streamChangesForReplicaSet(MongoDbStreamingChangeEventSource.java:98) at io.debezium.connector.mongodb.MongoDbStreamingChangeEventSource.execute(MongoDbStreamingChangeEventSource.java:82) at io.debezium.connector.mongodb.MongoDbStreamingChangeEventSource.execute(MongoDbStreamingChangeEventSource.java:41) at io.debezium.pipeline.ChangeEventSourceCoordinator.streamEvents(ChangeEventSourceCoordinator.java:272) at io.debezium.pipeline.ChangeEventSourceCoordinator.executeChangeEventSources(ChangeEventSourceCoordinator.java:197) at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:137) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: com.mongodb.MongoQueryException: Query failed with error code 10334 with name 'BSONObjectTooLarge' and error message 'BSONObj size: 17146559 (0x105A2BF) is invalid. Size must be between 0 and 16793600(16MB) First element: event:

      ```

      Implementation ideas (optional)

      <Your answer>

              Unassigned Unassigned
              agierczak Antoni Gierczak (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: