Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-4600

Allow mongodb-connector to decode Binary payloads


    • False
    • False

      1. The Goal
        Implement the outbox pattern on a Spring Boot/Data Java service using MongoDB, Debezium and Kafka
      1. Given
        An object to seralise, with a known-working Avro schema (we can serialise to and from the schema fine in tests).

      A local Kafka/Zookeeper running in Docker, with a replicaset of 2x MongoDB instances

      A Debezium Connector with "correct" best-guess config, (that at least successfully plucks the payload from the Outbox and places it on a topic)

      1. When 
        We manually serialise our object with KavkaAvroSerializer to a byte[] in Java
        We set this into the `byte[] payload` field on our OutboxEntity
        We use a spring data MongoRepository to save this entity to file
      1. Actual
        The document appears successfully in MongoDB

      The payload field appears as a binary representation - e.g.

       "payload": {
          "$binary": "AAAAAAJIZWU5MjljNDQtZDVjNi00NjNjLTg2NTAtYjU5MTAxOGVhZTg2CkhlbGxvAgIEAgYCCKbZ59LPXw==",
          "$type": "0"

      This appears to be the Base64 encoded of the previously generated Avro byte array.

      Debezium picks up the change and copies the contents of the payload field - verbatim - placing a Base64 encode of the desired content onto the topic.

      Kafka consumers fail to deserialise the message payload, as it no longer conforms to the Avro schema. 

      1. Desired
        This behaviour seems to be a side-effect of MongoDB's internal mechanisms for storing Binary data as a Base64 encode.
        As such it's not really a bug with Debezium - but it would be helpful (we think) to anyone attempting this for Debezium's behaviour to be configurable to cope.

      Could a converter be introduced to support this use of the outbox pattern, where the binary data from the payload is auto-decoded before being placed on the topic?

            Unassigned Unassigned
            nathan.bradshaw Nathan Bradshaw (Inactive)
            0 Vote for this issue
            5 Start watching this issue