Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-3843

Add a transformer to unwrap the records with struct-wrapped fields

    XMLWordPrintable

Details

    • Enhancement
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Won't Do
    • 1.7.0.Alpha1
    • None
    • cassandra-connector
    • None

    Description

      Some source connectors (Debezium Cassandra connector) generate CDC records with struct-wrapped values. For example, when a table has an integer "id" and string "name" fields, CDC records from Postgre or MySQL may contain something like

         "after" : {
            "id" : 2,
            "name" : "George Baily"
         }
      

      whereas CDC records from Cassandra connector may contain struct-wrapped values that look like

         "after" : {
            "id" : {
               "deletion_ts" : null,
               "set" : true,
               "value" : 2
            },
            "name" : {
               "deletion_ts" : null,
               "set" : true,
               "value" : "George Baily"
            }
         }
      

      As a result, applications that work for Debezium MySQL or Postgre connectors cannot be directly used with Cassandra connector.

      One approach to solve this problem would be to add an option in Cassandra connector to generate the plain records like the other connectors, however this approach was not favoured by the Cassandra connector maintainer. Therefore, I would propose to add an SMT to transform Cassandra's struct-wrapped CDC records to the plain CDC records. This approach is non-invasive and allows the sink connectors to transparently work with both types of records.

      The original posting google group https://groups.google.com/g/debezium/c/XKa8v0ovkis/m/9Nnxg8V1AgAJ

      Attachments

        Activity

          People

            Unassigned Unassigned
            elakito Aki Yoshida (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: