-
Feature Request
-
Resolution: Done
-
Minor
-
None
-
None
-
False
-
None
-
False
The current Azure EventHubs sink implementation for debezium-server has limited partitioning options: users can either specify a fixed partitionID to stream all changes to a single partition, or provide a static partitionKey that will be set on all batch operations, which will ultimately also lead to all events being streamed to the same partition.
In order to leverage increased throughput for downstream processing, it would be highly beneficial if the EventHubsChangeConsumer class would be able to route events to multiple partitions. The routing to the different partitions can be managed by the PartitionRouter, although this requires a change in the ChangeEvent interface to expose the PartitionId of a message.
The EventHubsChangeConsumer can then read the PartitionId of each ChangeEvent, and by managing a separate EventDataBatch per partition ensure that the right event are sent to right partitions.
This feature has previously been discussed/requested in this Zulip thread.