-
Feature Request
-
Resolution: Done
-
Minor
-
1.8.0.Final
-
None
-
False
-
False
-
- The Goal
Implement the outbox pattern on a Spring Boot/Data Java service using MongoDB, Debezium and Kafka
- Given
An object to seralise, with a known-working Avro schema (we can serialise to and from the schema fine in tests).
A local Kafka/Zookeeper running in Docker, with a replicaset of 2x MongoDB instances
A Debezium Connector with "correct" best-guess config, (that at least successfully plucks the payload from the Outbox and places it on a topic)
- When
We manually serialise our object with KavkaAvroSerializer to a byte[] in Java
We set this into the `byte[] payload` field on our OutboxEntity
We use a spring data MongoRepository to save this entity to file
- Actual
The document appears successfully in MongoDB
The payload field appears as a binary representation - e.g.
"payload": { "$binary": "AAAAAAJIZWU5MjljNDQtZDVjNi00NjNjLTg2NTAtYjU5MTAxOGVhZTg2CkhlbGxvAgIEAgYCCKbZ59LPXw==", "$type": "0" }
This appears to be the Base64 encoded of the previously generated Avro byte array.
Debezium picks up the change and copies the contents of the payload field - verbatim - placing a Base64 encode of the desired content onto the topic.
Kafka consumers fail to deserialise the message payload, as it no longer conforms to the Avro schema.
- Desired
This behaviour seems to be a side-effect of MongoDB's internal mechanisms for storing Binary data as a Base64 encode.
As such it's not really a bug with Debezium - but it would be helpful (we think) to anyone attempting this for Debezium's behaviour to be configurable to cope.
Could a converter be introduced to support this use of the outbox pattern, where the binary data from the payload is auto-decoded before being placed on the topic?
- relates to
-
DBZ-4598 Unable to register a Custom Converter with MongoDB Connector
- Closed