Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-4409

Enforce consistent vgtid representation in vitess connector

XMLWordPrintable

    • Icon: Enhancement Enhancement
    • Resolution: Done
    • Icon: Major Major
    • 1.8.0.CR1
    • None
    • vitess-connector
    • None

      Currently vgtid is converted to JSON string (to represent transaction id in TransactionMonitor and TransactionContext) from a Set of shard gtids using Gson.

      An example string for vgtid:

      [{\"keyspace\":\"cdc_integration_test_keyspace\",\"shard\":\"-80\",\"gtid\":\"MySQL56/2582b27c-7e73-4cd2-8640-028069dc2704:1-945517:1000011-1086941:2000011-2152786\"},{\"keyspace\":\"cdc_integration_test_keyspace\",\"shard\":\"80-\",\"gtid\":\"MySQL56/f84895a8-4eed-42ba-b500-469fd71cc08b:1-798504:1000006-1086988:2000006-2226157\"}] 
      1. The order of the array element are inconsistent or non-deterministic. It depends on the raw VGTID from the VEvent received, but can change because a HashSet is used to store the list.
      2. The keys in the JSON object are not ordered.

      Since we pass vgtid as a string (instead of JSON) for transaction id in enrichment and transaction metadata, we want to make it deterministic for the convenience of downstream. This can be deemed as encoding format (raw JSON string) for transaction id.

      The proposed fix is to:

      1. Use a list instead of set for vgtid to store shardGTIDs, so the order is preserved with the original order of VEvent as source of truth.
      2. Ensure keys order during JSON serialization

              Unassigned Unassigned
              shichaoan Shichao An (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: