Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-6617

Add shard field to events

XMLWordPrintable

      Feature request or enhancement

      Add the shard field to change events published by debezium.

      Which use case/requirement will be addressed by the proposed feature?

      There is no way to determine which shard or gtid the operation is for without including this shard field. It's present in the row events received in the VStream so there's no reason not to include it. We have both keyspace & table, in order to complete the source info we need the shard as well.

      Here's an example of what a row event looks like logged:

      2023-06-27 20:31:51,622 DEBUG  Vitess|dev.connector|streaming  VEvent: type: ROWtimestamp: 1687897910row_event {  
           table_name: "keyspace.table"  row_changes {[...]}  
      }  
      keyspace: "keyspace"
      shard: "-4000"
      } 

      We extract the shard field in addition to the keyspace/table name fields.

      One specific use case this is for is when we need to identify the specific GTID of an event within the VGTID set. Without the shard field, we don't know which shard's GTID set of the VGTID to examine. With the shard field, we will be able to know which shard's GTID set to use (and can thus deduce the exact GTID). This is necessary for building an auditor that will be checking that GTIDs in the source binary logs are present in the downstream kafka data.

      Implementation ideas (optional)

      Will create a PR.

            Unassigned Unassigned
            tthorn Thomas Thornton
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: