Details
-
Bug
-
Resolution: Not a Bug
-
Major
-
None
-
2.1.2.Final, 2.2.0.Alpha2
-
None
-
False
-
None
-
False
-
Critical
Description
5.0.15
Bug report
For bug reports, provide this information, please:
What Debezium connector do you use and what version?
I am using Debezium (embedded) engine with MongoDB connector - 2.1.2-Final
What is the connector configuration?
public Configuration mongodbConnector() throws IOException { File offsetStorageTempFile = File.createTempFile("offsets_", ".dat"); return Configuration.create() // engine properties .with("name", "sbd-mongodb") .with("connector.class", "io.debezium.connector.mongodb.MongoDbConnector") .with("offset.storage", "org.apache.kafka.connect.storage.FileOffsetBackingStore") .with("offset.storage.file.filename", offsetStorageTempFile.getAbsolutePath()) .with("offset.flush.interval.ms", "60000") // connector specific properties .with("mongodb.connection.string", "mongodb+srv://sb-debezium.1ewsyzd.mongodb.net") .with("topic.prefix", "sbd-mongodb-connector") .with("mongodb.user", "bwgjoseph") .with("mongodb.password", "<redacted>") .with("mongodb.ssl.enabled", "true") // default false .with("database.include.list", "source") // default empty .with("snapshot.delay.ms", "100") .with("errors.log.include.messages", "true") .build(); }
What is the captured database version and mode of deployment?
I am using MongoDB Atlas running on 5.0.15
What behaviour do you expect?
Using the default capture.mode (change_streams_update_full) should work, or any other mode, actually.
What behaviour do you see?
When an update occurs, I suspect the same for delete as well (or anywhere that uses document.getFullDocumentBeforeChange()) will throw an exception (see log below)
Do you see the same behaviour using the latest released Debezium version?
Yes, tested with 2.2.0-Alpha2.
Do you have the connector logs, ideally from start till finish?
(You might be asked later to provide DEBUG/TRACE level log)
Caused by: org.apache.kafka.connect.errors.ConnectException: Error while attempting to read from change stream on 'atlas-b15fi5-shard-0/ac-3jwzkhz-shard-00-00.1ewsyzd.mongodb.net:27017,ac-3jwzkhz-shard-00-01.1ewsyzd.mongodb.net:27017,ac-3jwzkhz-shard-00-02.1ewsyzd.mongodb.net:27017' at io.debezium.connector.mongodb.MongoDbStreamingChangeEventSource.lambda$establishConnection$3(MongoDbStreamingChangeEventSource.java:170) ~[debezium-connector-mongodb-2.1.2.Final.jar:2.1.2.Final] at io.debezium.connector.mongodb.ConnectionContext$MongoPreferredNode.execute(ConnectionContext.java:381) ~[debezium-connector-mongodb-2.1.2.Final.jar:2.1.2.Final] at io.debezium.connector.mongodb.MongoDbStreamingChangeEventSource.streamChangesForReplicaSet(MongoDbStreamingChangeEventSource.java:115) ~[debezium-connector-mongodb-2.1.2.Final.jar:2.1.2.Final] ... 10 common frames omitted Caused by: java.lang.NoSuchMethodError: 'java.lang.Object com.mongodb.client.model.changestream.ChangeStreamDocument.getFullDocumentBeforeChange()' at io.debezium.connector.mongodb.MongoDbCollectionSchema.valueFromDocumentChangeStream(MongoDbCollectionSchema.java:114) ~[debezium-connector-mongodb-2.1.2.Final.jar:2.1.2.Final] at io.debezium.connector.mongodb.MongoDbChangeRecordEmitter.createAndEmitChangeRecord(MongoDbChangeRecordEmitter.java:84) ~[debezium-connector-mongodb-2.1.2.Final.jar:2.1.2.Final] at io.debezium.connector.mongodb.MongoDbChangeRecordEmitter.emitUpdateRecord(MongoDbChangeRecordEmitter.java:72) ~[debezium-connector-mongodb-2.1.2.Final.jar:2.1.2.Final] at io.debezium.connector.mongodb.MongoDbChangeRecordEmitter.emitUpdateRecord(MongoDbChangeRecordEmitter.java:30) ~[debezium-connector-mongodb-2.1.2.Final.jar:2.1.2.Final] at io.debezium.pipeline.AbstractChangeRecordEmitter.emitChangeRecords(AbstractChangeRecordEmitter.java:45) ~[debezium-core-2.1.2.Final.jar:2.1.2.Final] at io.debezium.pipeline.EventDispatcher.dispatchDataChangeEvent(EventDispatcher.java:224) ~[debezium-core-2.1.2.Final.jar:2.1.2.Final] at io.debezium.connector.mongodb.MongoDbStreamingChangeEventSource.readChangeStream(MongoDbStreamingChangeEventSource.java:261) ~[debezium-connector-mongodb-2.1.2.Final.jar:2.1.2.Final] at io.debezium.connector.mongodb.MongoDbStreamingChangeEventSource.lambda$streamChangesForReplicaSet$0(MongoDbStreamingChangeEventSource.java:116) ~[debezium-connector-mongodb-2.1.2.Final.jar:2.1.2.Final] at io.debezium.connector.mongodb.ConnectionContext$MongoPreferredNode.execute(ConnectionContext.java:377) ~[debezium-connector-mongodb-2.1.2.Final.jar:2.1.2.Final] ... 11 common frames omitted
How to reproduce the issue using our tutorial deployment?
I have created a reproduce at https://github.com/bwgjoseph/spring-boot-debezium-mongodb-es
I believe the underlying cause is that
debezium-connector-mongodb:2.1.2-Final brings in mongodb-driver-sync;4.6.1 which brings in mongodb-driver-core:4.6.1 where getFullDocumentBeforeChange does not exist in ChangeStreamDocument until 4.7.
If I switched to use 4.7.x, it seems to work
<dependencyManagement> <dependencies> <dependency> <groupId>org.mongodb</groupId> <artifactId>mongodb-driver-sync</artifactId> <version>4.7.2</version> </dependency> <dependency> <groupId>org.mongodb</groupId> <artifactId>mongodb-driver-core</artifactId> <version>4.7.2</version> </dependency> <dependency> <groupId>org.mongodb</groupId> <artifactId>bson</artifactId> <version>4.7.2</version> </dependency> <dependency> <groupId>org.mongodb</groupId> <artifactId>bson-record-codec</artifactId> <version>4.7.2</version> </dependency> </dependencies> </dependencyManagement>
Thanks
