Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-3698

PoC for adding transformations / SMT steps to the Debezium UI

XMLWordPrintable

      Create a PoC that adds another Debezium UI wizard page/step (or multiple steps/pages) to configure Kafka Connect transformations, aka SMTs - Single Message Transforms.
      SMTs provide capabilities to apply transformations like adding, dropping or aggregating field data, renaming fields, masking data, filter out messages that should not go to Kafka, etc.

      Here's a writeup by Gunnar about SMTs on his blog: https://www.morling.dev/blog/single-message-transforms-swiss-army-knife-of-kafka-connect/

      Confluent gives a good overview of the most common SMTs that come shipped with every Kafka distribution:
      https://docs.confluent.io/platform/current/connect/transforms/overview.html

      Some advanced SMTs by Debezium in the docs:
      https://debezium.io/documentation/reference/configuration/event-flattening.html
      https://debezium.io/documentation/reference/configuration/filtering.html
      https://debezium.io/documentation/reference/configuration/topic-routing.html
      https://debezium.io/documentation/reference/configuration/content-based-routing.html

      Please decide on 2-3 transformations (choose basic ones like filtering, replace field, etc) that should be implemented as a PoC, sketch a UI for it and let's discuss that with UX and finally implement a PoC to add transformations to the Debezium UI. It should be possible to use a single transformation or multiple and it should be possible to order them (which one gets called first, which next, which last).
      The UI does not necessarily have to return the exact Kafka Connect config format. An abstract format that can be converted to the Kafka Connect config is sufficient (as you see fit).

        1. image-2021-07-22-15-59-47-759.png
          15 kB
          Indra Shukla
        2. image-2021-07-22-16-00-33-348.png
          28 kB
          Indra Shukla
        3. image-2021-07-26-16-20-42-409.png
          100 kB
          Indra Shukla
        4. recording.webm
          8.68 MB
          Indra Shukla
        5. Screenshot from 2021-08-20 14-28-03.png
          69 kB
          Indra Shukla

              ishukla Indra Shukla
              rk3rn3r René Kerner
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated:
                Resolved: