-
Feature Request
-
Resolution: Unresolved
-
Major
-
None
-
None
-
False
-
-
False
In order to make your issue reports as actionable as possible, please provide the following information, depending on the issue type.
Feature request or enhancement
For feature requests or enhancements, provide this information, please:
Which use case/requirement will be addressed by the proposed feature?
Current Behavior:
Debezium (DBZ) connectors use an in-memory queue (ArrayDeque) within the ChangeEventQueue class to buffer changes between the database reader and the Kafka Connect task. When the Connect task is slow to process events, the in-memory queue can become full, causing the database reader to slow down due to backpressure.
Proposed Improvement:
Replacing the in-memory queue (requires vertical scaling) with queue implementation (persisted queues) that can be infinitely scalable (Redis queue or filesystem-backed queue implementation). This would allow the database reading process to continue uninterrupted, with events being temporarily written to disk when the queue is full. When the Connect task is ready, it can read events back from the file system.
A few file system based queues :
- https://github.com/bulldog2011/bigqueue
- LMDB queue implementation
- Kafka or Redis etc
Add support for custom config to plug custom queue factory/impl into DBZ core process.
Benefits:
- Cost efficiency : Small instances can be used to host the connectors. Connectors are memory bound
- High throughput as local disc I/O (including serialization and deserialization) is faster than repeated reads from the remote database
- Increased overall connector resilience.
Implementation ideas (optional)
<Your answer>