Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-6723

Add support for partitioning with Azure EventHubs

XMLWordPrintable

    • Icon: Feature Request Feature Request
    • Resolution: Done
    • Icon: Minor Minor
    • 2.5.0.Beta1
    • None
    • debezium-server
    • None
    • False
    • None
    • False
    • 0
    • 0% 0%

      The current Azure EventHubs sink implementation for debezium-server has limited partitioning options: users can either specify a fixed partitionID to stream all changes to a single partition, or provide a static partitionKey that will be set on all batch operations, which will ultimately also lead to all events being streamed to the same partition.

      In order to leverage increased throughput for downstream processing, it would be highly beneficial if the EventHubsChangeConsumer class would be able to route events to multiple partitions. The routing to the different partitions can be managed by the PartitionRouter, although this requires a change in the ChangeEvent interface to expose the PartitionId of a message.

      The EventHubsChangeConsumer can then read the PartitionId of each ChangeEvent, and by managing a separate EventDataBatch per partition ensure that the right event are sent to right partitions.

      This feature has previously been discussed/requested in this Zulip thread.

            Unassigned Unassigned
            sebastiaan@castoredc.com Sebastiaan Knijnenburg (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: