Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-4409

Enforce consistent vgtid representation in vitess connector

    XMLWordPrintable

Details

    • Enhancement
    • Resolution: Done
    • Major
    • 1.8.0.CR1
    • None
    • vitess-connector
    • None

    Description

      Currently vgtid is converted to JSON string (to represent transaction id in TransactionMonitor and TransactionContext) from a Set of shard gtids using Gson.

      An example string for vgtid:

      [{\"keyspace\":\"cdc_integration_test_keyspace\",\"shard\":\"-80\",\"gtid\":\"MySQL56/2582b27c-7e73-4cd2-8640-028069dc2704:1-945517:1000011-1086941:2000011-2152786\"},{\"keyspace\":\"cdc_integration_test_keyspace\",\"shard\":\"80-\",\"gtid\":\"MySQL56/f84895a8-4eed-42ba-b500-469fd71cc08b:1-798504:1000006-1086988:2000006-2226157\"}] 
      1. The order of the array element are inconsistent or non-deterministic. It depends on the raw VGTID from the VEvent received, but can change because a HashSet is used to store the list.
      2. The keys in the JSON object are not ordered.

      Since we pass vgtid as a string (instead of JSON) for transaction id in enrichment and transaction metadata, we want to make it deterministic for the convenience of downstream. This can be deemed as encoding format (raw JSON string) for transaction id.

      The proposed fix is to:

      1. Use a list instead of set for vgtid to store shardGTIDs, so the order is preserved with the original order of VEvent as source of truth.
      2. Ensure keys order during JSON serialization

      Attachments

        Activity

          People

            Unassigned Unassigned
            shichaoan Shichao An (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: