Uploaded image for project: 'Hybrid Cloud Console'
  1. Hybrid Cloud Console
  2. RHCLOUD-35697

Kafka connectors: Migrate to proto schema for messaging from outbox to relations-api

XMLWordPrintable

    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • False
    • None
    • Unset
    • None

      Currently, the debezium source connector that reads from the rbac outbox table writes messages to the topic according to a very simple schema, i.e. 

      "schema":{"type":"string","optional":true,"name":"io.debezium.data.Json","version":1} 

      which just specifies an optional string.

      The kafka-relations-sink needs to read with the corresponding schema.

      This default schema does very little to help us, because we need to stringify on the source connector side and parse strings on the sink side. However, if we specify the payload properly, we can marshall and unmarshall using boilerplate logic in the connectors.

      This task is to create and publish a proper spec as represented by the implied spec in https://github.com/RedHatInsights/insights-rbac/pull/1172. The main/only message (type) to specify is the "replication event". This spec can and should reference messages (types) in the the existing relations proto spec (i.e. Relationship).

      Definition of done:

      1. Create new proto spec.
      2. Publish the proto spec in an appropriate public registry.
      3. Refer to the new proto in the schema of both the debezium source connector and relations sink connector.
      4. Update any code that formerly did marshalling/unmarshalling of the replication event. e.g. rbac and kafka-relations-sink.
      5. Verify that relations updates continue to flow through kafka to spicedb, etc.

       

              Unassigned Unassigned
              mmclaugh@redhat.com Mark McLaughlin
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated: