Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-7122

Improve buffer flushing for JDBC sink connector

XMLWordPrintable

    • Icon: Enhancement Enhancement
    • Resolution: Unresolved
    • Icon: Major Major
    • Backlog
    • None
    • jdbc-connector
    • None
    • False
    • None
    • False

      Current implementation will flush the buffer in any case every put call received by Kafka Connect.

      The consequence is that  batch.size will not be honored when consumer.max.poll.records < batch.size.

      We want to eliminate this limitation adding a heartbeat thread that will send messages to a dedicated topic so that Kafka is forced to call the connector put method. When the connector receive a message from heartbeat  the buffer will be flushed. 

      See DatabaseHeartbeatImpl for reference.

            rh-ee-mvitale Mario Fiore Vitale
            rh-ee-mvitale Mario Fiore Vitale
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: