Loading...

Type: Bug
Resolution: Won't Do
Priority: Major
Fix Version/s: 2.2-backlog
Affects Version/s: 2.2.0.CR1
Component/s: mongodb-connector
Labels:
None

Blocked:
False
Blocked Reason:
None
Ready:
False
Intelligence Requested:
Market:

Severity:
Moderate

SFDC Cases Counter:
SFDC Cases Open:
SFDC Cases Links:

When running a snapshot only test with 1000 documents and chunk size of 10, the test takes about 10s for ReplicaSet topology. The same test takes about 16 minutes for sharded topology (number of shards and the fact whether the captured collection is sharded has little to no effect). Increasing the chunk size to 100 when running against sharded cluster loweres the execution time to about 2 minutes.

The following long show the processing of single snapshot window

2023-04-15 14:27:58,617 TRACE  MongoDB|mongo1|streaming  Emitting open window for chunk = 'b411b3b5-4d14-4f2c-80c0-67cb6bbcd913'   [io.debezium.connector.mongodb.MongoDbIncrementalSnapshotChangeEventSource]
2023-04-15 14:27:58,650 INFO   MongoDB|mongo1|streaming  Incremental snapshot for collection 'cluster.dbA.c1' will end at position [2147484647]   [io.debezium.connector.mongodb.MongoDbIncrementalSnapshotChangeEventSource]
2023-04-15 14:27:58,651 DEBUG  MongoDB|mongo1|streaming  Exporting data chunk from collection 'cluster.dbA.c1' (total 1 collections)   [io.debezium.connector.mongodb.MongoDbIncrementalSnapshotChangeEventSource]
2023-04-15 14:27:58,652 DEBUG  MongoDB|mongo1|streaming  	 For collection 'cluster.dbA.c1' using query: '{"_id": {"$lte": 2147484647}}', key: 'null', maximum key: '[2147484647]'   [io.debezium.connector.mongodb.MongoDbIncrementalSnapshotChangeEventSource]
2023-04-15 14:27:58,658 DEBUG  MongoDB|mongo1|streaming  	 Next window will resume from [2147483657]   [io.debezium.connector.mongodb.MongoDbIncrementalSnapshotChangeEventSource]
2023-04-15 14:27:58,658 DEBUG  MongoDB|mongo1|streaming  	 Finished exporting 10 records for window of collection 'cluster.dbA.c1'; total duration '00:00:00.008'   [io.debezium.connector.mongodb.MongoDbIncrementalSnapshotChangeEventSource]
2023-04-15 14:27:58,660 TRACE  MongoDB|mongo1|streaming  Emitting close window for chunk = 'b411b3b5-4d14-4f2c-80c0-67cb6bbcd913'   [io.debezium.connector.mongodb.MongoDbIncrementalSnapshotChangeEventSource]
2023-04-15 14:27:58,671 TRACE  MongoDB|mongo1|streaming  Checking window for table 'cluster.dbA.signals', key 'Struct{id={"$oid": "643a9842c2f9b423a7a0b393"}}', window contains '{Struct{id=2147483648}=[Ljava.lang.Object;@67ebfea0, Struct{id=2147483649}=[Ljava.lang.Object;@78310ace, Struct{id=2147483650}=[Ljava.lang.Object;@410a7b71, Struct{id=2147483651}=[Ljava.lang.Object;@7056f649, Struct{id=2147483652}=[Ljava.lang.Object;@4870181d, Struct{id=2147483653}=[Ljava.lang.Object;@38e70ca7, Struct{id=2147483654}=[Ljava.lang.Object;@2a243939, Struct{id=2147483655}=[Ljava.lang.Object;@6d07768a, Struct{id=2147483656}=[Ljava.lang.Object;@257f3bc7, Struct{id=2147483657}=[Ljava.lang.Object;@257d99fb}'   [io.debezium.connector.mongodb.MongoDbIncrementalSnapshotChangeEventSource]
2023-04-15 14:27:59,053 INFO   ShardedIncrementalSnapshotIT||engine  1 records sent during previous 00:00:17.672, last recorded offset of {rs=cluster, server_id=mongo1} partition is {sec=1681561666, ord=3, transaction_id=null, resume_token=82643A9842000000032B022C0100296E5A1004596D049A67164583A59732300152246546645F69640064643A9842C2F9B423A7A0B3930004, incremental_snapshot_maximum_key=aced0005757200135b4c6a6176612e6c616e672e4f626a6563743b90ce589f1073296c0200007870000000017372000e6a6176612e6c616e672e4c6f6e673b8be490cc8f23df0200014a000576616c7565787200106a6176612e6c616e672e4e756d62657286ac951d0b94e08b020000787000000000800003e7, incremental_snapshot_collections=[{"incremental_snapshot_collections_id":"cluster.dbA.c1","incremental_snapshot_collections_additional_condition":null}], incremental_snapshot_primary_key=aced000570}   [io.debezium.connector.common.BaseSourceTask]
2023-04-15 14:28:08,603 TRACE  MongoDB|mongo1|streaming  Checking window for table 'cluster.dbA.signals', key 'Struct{id="b411b3b5-4d14-4f2c-80c0-67cb6bbcd913-open"}', window contains '{Struct{id=2147483648}=[Ljava.lang.Object;@67ebfea0, Struct{id=2147483649}=[Ljava.lang.Object;@78310ace, Struct{id=2147483650}=[Ljava.lang.Object;@410a7b71, Struct{id=2147483651}=[Ljava.lang.Object;@7056f649, Struct{id=2147483652}=[Ljava.lang.Object;@4870181d, Struct{id=2147483653}=[Ljava.lang.Object;@38e70ca7, Struct{id=2147483654}=[Ljava.lang.Object;@2a243939, Struct{id=2147483655}=[Ljava.lang.Object;@6d07768a, Struct{id=2147483656}=[Ljava.lang.Object;@257f3bc7, Struct{id=2147483657}=[Ljava.lang.Object;@257d99fb}'   [io.debezium.connector.mongodb.MongoDbIncrementalSnapshotChangeEventSource]
2023-04-15 14:28:08,611 DEBUG  MongoDB|mongo1|streaming  Sending 10 events from window buffer   [io.debezium.connector.mongodb.MongoDbIncrementalSnapshotChangeEventSource]
2023-04-15 14:28:08,626 TRACE  MongoDB|mongo1|streaming  Emitting open window for chunk = '43c9d115-92a6-4168-b71b-b5c8acb6d34e'   [io.debezium.connector.mongodb.MongoDbIncrementalSnapshotChangeEventSource]
2023-04-15 14:28:08,635 DEBUG  MongoDB|mongo1|streaming  Exporting data chunk from collection 'cluster.dbA.c1' (total 1 collections)   [io.debezium.connector.mongodb.MongoDbIncrementalSnapshotChangeEventSource]
2023-04-15 14:28:08,636 DEBUG  MongoDB|mongo1|streaming  	 For collection 'cluster.dbA.c1' using query: '{"$and": [{"_id": {"$gt": 2147483657}}, {"_id": {"$lte": 2147484647}}]}', key: '[2147483657]', maximum key: '[2147484647]'   [io.debezium.connector.mongodb.MongoDbIncrementalSnapshotChangeEventSource]
2023-04-15 14:28:08,642 DEBUG  MongoDB|mongo1|streaming  	 Next window will resume from [2147483667]   [io.debezium.connector.mongodb.MongoDbIncrementalSnapshotChangeEventSource]
2023-04-15 14:28:08,642 DEBUG  MongoDB|mongo1|streaming  	 Finished exporting 10 records for window of collection 'cluster.dbA.c1'; total duration '00:00:00.007'   [io.debezium.connector.mongodb.MongoDbIncrementalSnapshotChangeEventSource]
2023-04-15 14:28:08,644 TRACE  MongoDB|mongo1|streaming  Emitting close window for chunk = '43c9d115-92a6-4168-b71b-b5c8acb6d34e'   [io.debezium.connector.mongodb.MongoDbIncrementalSnapshotChangeEventSource]
2023-04-15 14:28:08,658 TRACE  MongoDB|mongo1|streaming  Checking window for table 'cluster.dbA.signals', key 'Struct{id="b411b3b5-4d14-4f2c-80c0-67cb6bbcd913-close"}', window contains '{Struct{id=2147483658}=[Ljava.lang.Object;@75f35f80, Struct{id=2147483659}=[Ljava.lang.Object;@69b48acf, Struct{id=2147483660}=[Ljava.lang.Object;@2e28d3c5, Struct{id=2147483661}=[Ljava.lang.Object;@2bf11095, Struct{id=2147483662}=[Ljava.lang.Object;@44114534, Struct{id=2147483663}=[Ljava.lang.Object;@93e4d8b, Struct{id=2147483664}=[Ljava.lang.Object;@61600192, Struct{id=2147483665}=[Ljava.lang.Object;@42b6d093, Struct{id=2147483666}=[Ljava.lang.Object;@cc00514, Struct{id=2147483667}=[Ljava.lang.Object;@77694312}'   [io.debezium.connector.mongodb.MongoDbIncrementalSnapshotChangeEventSource]
2023-04-15 14:28:18,640 TRACE  MongoDB|mongo1|streaming  Checking window for table 'cluster.dbA.signals', key 'Struct{id="43c9d115-92a6-4168-b71b-b5c8acb6d34e-open"}', window contains '{Struct{id=2147483658}=[Ljava.lang.Object;@75f35f80, Struct{id=2147483659}=[Ljava.lang.Object;@69b48acf, Struct{id=2147483660}=[Ljava.lang.Object;@2e28d3c5, Struct{id=2147483661}=[Ljava.lang.Object;@2bf11095, Struct{id=2147483662}=[Ljava.lang.Object;@44114534, Struct{id=2147483663}=[Ljava.lang.Object;@93e4d8b, Struct{id=2147483664}=[Ljava.lang.Object;@61600192, Struct{id=2147483665}=[Ljava.lang.Object;@42b6d093, Struct{id=2147483666}=[Ljava.lang.Object;@cc00514, Struct{id=2147483667}=[Ljava.lang.Object;@77694312}'   [io.debezium.connector.mongodb.MongoDbIncrementalSnapshotChangeEventSource]

split from

DBZ-4427 Complete MongoDB incremental snapshotting implementation

Closed

Details

Description

Attachments

Issue Links

Easy Agile Planning Poker

Activity

People

Dates