Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-4478

Partition-scoped metrics for the SQL Server connector

    XMLWordPrintable

Details

    Description

      The connector task working with multiple partitions should expose metrics in a different format. The existing metrics logically can be divided into two categories:

      1. Task-scoped metrics. For instance, QueueTotalCapacity. While processing multiple source databases, the task will use a single queue for the messages sourced from all partitions.
      2. Partition-scoped metrics. For instance, MilliSecondsBehindSource. Such metrics should be reported individually for each source database which is currently impossible.

      The introduction of partition-scoped metrics is necessary for the following reasons:

      1. Being able to operate the connector. Without a partition identifier, the partition-scoped metrics from different partitions would clash and lose their meaning.
      2. Being able to write integration tests for multi-partition scenarios. The integration test suite synchronizes with the connector by means of metrics. For instance, there is a typical scenario where the connector waits for a snapshot to be completed. In a multi-partition configuration, the connector should either wait for a specific database snapshotted, or all databases snapshotted. It's impossible to implement on top of the currently existing metrics.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              sergeimorozov Sergei Morozov
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: