-
Bug
-
Resolution: Done
-
Major
-
2.3.2.Final
-
None
-
False
-
None
-
False
-
Important
In order to make your issue reports as actionable as possible, please provide the following information, depending on the issue type.
Bug report
For bug reports, provide this information, please:
What Debezium connector do you use and what version?
debezium-connector-mongodb:2.3.2.Final
What is the connector configuration?
This is what I think is the relevant subset of the config
- name: debezium.source.key.converter value: org.apache.kafka.connect.json.JsonConverter - name: debezium.source.key.converter.schemas.enable value: "false" - name: debezium.source.value.converter value: org.apache.kafka.connect.storage.StringConverter - name: debezium.source.value.converter.schemas.enable value: "false" - name: debezium.source.header.converter.schemas.enable value: "false" - name: debezium.source.connector.class value: io.debezium.connector.mongodb.MongoDbConnector - name: debezium.transforms value: outbox - name: debezium.transforms.outbox.type value: io.debezium.connector.mongodb.transforms.outbox.MongoEventRouter - name: debezium.transforms.outbox.collection.field.event.key value: aggregateId - name: debezium.transforms.outbox.route.by.field value: aggregateType - name: debezium.transforms.outbox.collection.fields.additional.placement value: eventType:envelope,aggregateType:envelope,aggregateId:envelope,createdAt:envelope,instanaT:header:x-instana-t,instanaS:header:x-instana-s,instanaL:header:x-instana-l - name: debezium.transforms.outbox.collection.expand.json.payload value: "true"
What is the captured database version and mode of deployment?
Atlas MongoDB 5.0.19 (AWS)
What behaviour do you expect?
Outbox records should be correctly processed irrespective of payload, failures should not lead to unrecoverable errors.
What behaviour do you see?
When `debezium.transforms.outbox.collection.expand.json.payload` is set to true, the connector crashes on certain JSON payloads and is unable to recover. A sample JSON payload is this:
{
"additionalContacts": [
{
"firstName": "John",
"lastName": "Doe",
"comment": null
},
{
"firstName": "Jane",
"lastName": "Doe",
"comment": "A comment"
}
]
}
Apparently the issue lies with the JSON expansion. The docs say that "In the converted string, the companion schema is deduced from the original JSON document."
Here it appears as if the type of the "comment" property is deduced to be "struct" from the null value in the first array element, and the string in the second element then causes the conversion to fail and subsequently the connector to crash, see log excerpt below.
In general I feel like the conversion should be more lenient and not require conforming to any schema as long as the payload is syntactically correct JSON, and even non-JSON payloads should at least be tolerated (maybe kept as the original string value).
Do you see the same behaviour using the latest relesead Debezium version?
Tested with latest stable version 2.3.2.Final
Do you have the connector logs, ideally from start till finish?
Aug 4, 2023 @ 09:45:49.986 ERROR Connector completed: success = 'false', message = 'Stopping connector after error in the application's handler method: Only Struct objects supported for [Read Outbox Event], found: java.lang.String', error = 'org.apache.kafka.connect.errors.DataException: Only Struct objects supported for [Read Outbox Event], found: java.lang.String' - Aug 4, 2023 @ 09:45:49.885 WARN Failed to expand after field: Field comment of schema io.debezium.data.Json.payload is not the same type for all documents in the array. Check option 'struct' of parameter 'array.encoding' -
How to reproduce the issue using our tutorial deployment?
- links to
-
RHEA-2023:120698
Red Hat build of Debezium 2.3.4 release
