Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-8968

Add support for pluggable ArrayDeque

XMLWordPrintable

    • Icon: Feature Request Feature Request
    • Resolution: Unresolved
    • Icon: Major Major
    • Backlog
    • None
    • debezium-core
    • None
    • False
    • Hide

      None

      Show
      None
    • False

      In order to make your issue reports as actionable as possible, please provide the following information, depending on the issue type.

      Feature request or enhancement

      For feature requests or enhancements, provide this information, please:

      Which use case/requirement will be addressed by the proposed feature?

      Current Behavior:
      Debezium (DBZ) connectors use an in-memory queue (ArrayDeque) within the ChangeEventQueue class to buffer changes between the database reader and the Kafka Connect task. When the Connect task is slow to process events, the in-memory queue can become full, causing the database reader to slow down due to backpressure.

      Proposed Improvement:
      Replacing the in-memory queue (requires vertical scaling) with queue implementation (persisted queues) that can be infinitely scalable (Redis queue or filesystem-backed queue implementation). This would allow the database reading process to continue uninterrupted, with events being temporarily written to disk when the queue is full. When the Connect task is ready, it can read events back from the file system. 

      A few file system based queues :

      1. https://github.com/bulldog2011/bigqueue
      2. LMDB queue implementation
      3. Kafka or Redis etc

      Add support for custom config to plug custom queue factory/impl into DBZ core process.

      Benefits:

      1. Cost efficiency : Small instances can be used to host the connectors. Connectors are memory bound
      2. High throughput as local disc I/O (including serialization and deserialization) is faster than repeated reads from the remote database
      3. Increased overall connector resilience.

        Implementation ideas (optional)

      <Your answer>

              Unassigned Unassigned
              dasarianil1505 Anil Dasari (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated: