Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-7994 [Epic] MongoDB sink connector enhancements
  3. DBZ-7122

Improve buffer flushing for JDBC sink connector

XMLWordPrintable

    • False
    • None
    • False

      Current implementation will flush the buffer in any case every put call received by Kafka Connect.

      The consequence is that  batch.size will not be honored when consumer.max.poll.records < batch.size.

      We want to eliminate this limitation adding a heartbeat thread that will send messages to a dedicated topic so that Kafka is forced to call the connector put method. When the connector receive a message from heartbeat  the buffer will be flushed. 

      See DatabaseHeartbeatImpl for reference.

            rk3rn3r René Kerner
            rh-ee-mvitale Mario Fiore Vitale
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: