-
Feature Request
-
Resolution: Done
-
Major
-
None
-
False
-
None
-
False
Feature request or enhancement
Add the shard field to change events published by debezium.
Which use case/requirement will be addressed by the proposed feature?
There is no way to determine which shard or gtid the operation is for without including this shard field. It's present in the row events received in the VStream so there's no reason not to include it. We have both keyspace & table, in order to complete the source info we need the shard as well.
Here's an example of what a row event looks like logged:
2023-06-27 20:31:51,622 DEBUG Vitess|dev.connector|streaming VEvent: type: ROWtimestamp: 1687897910row_event { table_name: "keyspace.table" row_changes {[...]} } keyspace: "keyspace" shard: "-4000" }
We extract the shard field in addition to the keyspace/table name fields.
One specific use case this is for is when we need to identify the specific GTID of an event within the VGTID set. Without the shard field, we don't know which shard's GTID set of the VGTID to examine. With the shard field, we will be able to know which shard's GTID set to use (and can thus deduce the exact GTID). This is necessary for building an auditor that will be checking that GTIDs in the source binary logs are present in the downstream kafka data.
Implementation ideas (optional)
Will create a PR.
- links to
-
RHEA-2024:129636 Red Hat build of Debezium 2.5.4 release