Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-8069

Vitess-connector should provide a topic naming strategy that supports separate connectors per-table

XMLWordPrintable

    • Icon: Feature Request Feature Request
    • Resolution: Done
    • Icon: Major Major
    • 3.0.0.Alpha2
    • None
    • vitess-connector
    • None
    • False
    • None
    • False

      Feature request or enhancement

      For feature requests or enhancements, provide this information, please:

      Which use case/requirement will be addressed by the proposed feature?

      There is a core limitation of the VStream COPY operation, it must be done on all tables that the VStream client subscribes to. Due to this, in order to avoid long COPY operations that block other tables in the keyspace, you need to have a separate connector for each table (so you can trigger COPY operations individually). This means that for several tables of the same keyspace topic.prefix would get redundant (include the table name in it, but then the result topic name has the table name twice) or lack clarity (add a suffix like -2 or -3, but then the topics don't have clear naming). Vitess connector should provide a topic naming strategy that allows users to easily set up multiple connectors for the same keyspace and can produce to consistently and clearly named topics.

      Additionally, make sure heartbeat topics and other metadata topics would have consistent and clear naming.

      I have reached out to the vitess community here but since it's a significant change to vstream copy we should provide something in the interim. 

      Zulip chat for more context

      Implementation ideas (optional)

      In order to workaround this, provide a topic naming strategy that reads the configs:

      • topic.prefix - used for everything standard still, like metrics/logs/identifying a connector. This should be unique (e.g., <keyspace>.<table>)
      • data.change.topic.prefix - used to override the topic.prefix used for data change topic naming, this does not have the same uniqueness requirement enforced, can be just <keyspace> and the resulting topic name for changes to tableA is <keyspace>.tableA

      Then for things like the metadata topics we will use topic.prefix since it will be unique. For the data change topic that uses the table name, we will use override.topic.prefix.

            Unassigned Unassigned
            tthorn Thomas Thornton
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: