Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-6680

Reduce additional data created by provide.transaction.metadata setting

XMLWordPrintable

    • Icon: Feature Request Feature Request
    • Resolution: Done
    • Icon: Major Major
    • Backlog
    • None
    • vitess-connector
    • None

      Feature request or enhancement

      For feature requests or enhancements, provide this information, please:

      Which use case/requirement will be addressed by the proposed feature?

      In some cases, users just want to have the payload enriched with data collection order/total order fields. They do not want the topic.transaction. Additionally, for some cases, the repeated transaction ID can more than double the size of the payload. This is the case for Debezium Vitess Connector where the VGTID can be very large since it includes all shards.

      Implementation ideas (optional)

      There are three ways we can reduce the amount of extra data created by this setting

      1. Add setting for users to disable the topic.transaction so the connector can add the new fields to the DML payloads but will not output any data to the transaction.topic and doesn't require it to exist
      2. When provide.transaction.metadata is enabled, it stores the transaction ID twice in the message payload as a source field (e.g., source.vgtid) and in source.transaction (source.transaction.id). Provide a setting to only store ID once or make this the default behavior
      3. When provide.transaction.metadata is enabled, it stored the shards of every VGTID in every task. This is not the behavior when this is disabled. We should preserve the behavior of only storing those needed shards for each task

              Unassigned Unassigned
              tthorn Thomas Thornton
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: