-
Feature Request
-
Resolution: Done
-
Major
-
None
-
None
-
False
-
None
-
False
Feature request or enhancement
For feature requests or enhancements, provide this information, please:
Which use case/requirement will be addressed by the proposed feature?
In some cases, users just want to have the payload enriched with data collection order/total order fields. They do not want the topic.transaction. Additionally, for some cases, the repeated transaction ID can more than double the size of the payload. This is the case for Debezium Vitess Connector where the VGTID can be very large since it includes all shards.
Implementation ideas (optional)
There are three ways we can reduce the amount of extra data created by this setting
- Add setting for users to disable the topic.transaction so the connector can add the new fields to the DML payloads but will not output any data to the transaction.topic and doesn't require it to exist
- When provide.transaction.metadata is enabled, it stores the transaction ID twice in the message payload as a source field (e.g., source.vgtid) and in source.transaction (source.transaction.id). Provide a setting to only store ID once or make this the default behavior
- When provide.transaction.metadata is enabled, it stored the shards of every VGTID in every task. This is not the behavior when this is disabled. We should preserve the behavior of only storing those needed shards for each task