Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-971

Convey original operation type when using flattening SMTs

    XMLWordPrintable

Details

    • Feature Request
    • Resolution: Done
    • Minor
    • 0.9.0.Beta2
    • 0.9.0.Alpha2
    • mongodb-connector
    • None
    • 0
    • 0% 0%

    Description

      The problem

      Currently there's no way to inform whether the message operation is, this information is lost along the way through the `UnwrapFromMongoDbEnvelope`, we can only distinguish between deletions or not.

      The scenario

      I want to be able to tell whether the message is an update or insert thus easing logic within the Sink Connector.
      The rationale is about optimizing the Sink in a way it doesn't have to look for an item before knowing whether it should insert or update, this can potentially save a great number of queries in the Sink side.
      Basically avoid `saveOrUpdate` situations by knowing exactly which operation to run.

      Past proposed solution (rejected)

      Currently the value schema has a metadata name which informs the server, db and collection, I want to add on it the operation as in `io.debezium.data.Envelope.Operation`
      The current value looks like: `serverX.dbA.c1`
      Proposed value would be:

      • `serverX.dbA.c1.r` for initial read
      • `serverX.dbA.c1.c` for creation
      • `serverX.dbA.c1.u` for updates

      This is very similar to the initialSync key in the non cdc way.

      Proposed solution

      Add a new header into the kafka record, finally being available in the message itself, this header will be optional via a transformer flag (MONGO_OPERATION_HEADER) and will respect the `io.debezium.data.Envelope.Operation` enum.
      The proposed header name is `application/debezium-operation`

      Implementation details

      This can be an optional feature within the transformer thus not breaking BC

      Attachments

        Activity

          People

            renatomefi Renato Mefi (Inactive)
            renatomefi Renato Mefi (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: