Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-7994 [Epic] MongoDB sink connector enhancements
  3. DBZ-7122

Improve buffer flushing for JDBC sink connector

XMLWordPrintable

    • 8
    • False
    • None
    • False

      Current implementation will flush the buffer in any case every put call received by Kafka Connect.

      The consequence is that  batch.size will not be honored when consumer.max.poll.records < batch.size.

      We want to eliminate this limitation adding a heartbeat thread that will send messages to a dedicated topic so that Kafka is forced to call the connector put method. When the connector receive a message from heartbeat  the buffer will be flushed. 

      See DatabaseHeartbeatImpl for reference.

              rk3rn3r René Kerner
              rh-ee-mvitale Mario Fiore Vitale
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated: