Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-971

Convey original operation type when using flattening SMTs

XMLWordPrintable

    • Icon: Feature Request Feature Request
    • Resolution: Done
    • Icon: Minor Minor
    • 0.9.0.Beta2
    • 0.9.0.Alpha2
    • mongodb-connector
    • None

      The problem

      Currently there's no way to inform whether the message operation is, this information is lost along the way through the `UnwrapFromMongoDbEnvelope`, we can only distinguish between deletions or not.

      The scenario

      I want to be able to tell whether the message is an update or insert thus easing logic within the Sink Connector.
      The rationale is about optimizing the Sink in a way it doesn't have to look for an item before knowing whether it should insert or update, this can potentially save a great number of queries in the Sink side.
      Basically avoid `saveOrUpdate` situations by knowing exactly which operation to run.

      Past proposed solution (rejected)

      Currently the value schema has a metadata name which informs the server, db and collection, I want to add on it the operation as in `io.debezium.data.Envelope.Operation`
      The current value looks like: `serverX.dbA.c1`
      Proposed value would be:

      • `serverX.dbA.c1.r` for initial read
      • `serverX.dbA.c1.c` for creation
      • `serverX.dbA.c1.u` for updates

      This is very similar to the initialSync key in the non cdc way.

      Proposed solution

      Add a new header into the kafka record, finally being available in the message itself, this header will be optional via a transformer flag (MONGO_OPERATION_HEADER) and will respect the `io.debezium.data.Envelope.Operation` enum.
      The proposed header name is `application/debezium-operation`

      Implementation details

      This can be an optional feature within the transformer thus not breaking BC

              renatomefi Renato Mefi (Inactive)
              renatomefi Renato Mefi (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: