Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-4375

Allow additional config options for Debezium Server Pubsub Connector


    • Icon: Enhancement Enhancement
    • Resolution: Done
    • Icon: Minor Minor
    • 1.9.0.Alpha1
    • None
    • debezium-server
    • None
    • False
    • False

      Currently, the Google Cloud Pub/Sub connector included with Debezium Server does not make use of any of the additional configuration options available to the Pub/Sub publisher.  Without additional options enabled, we have only been able to stream at ~300 rows per minute

      Specifically, I would like to add config options for

      1) Batch settings:


      2) Retry settings:


      3) Flow Control:


      Ability to change these settings will greatly assist us in being able to tune the performance when pushing messages to the sink.

      There are many configurable items for pubsub, so there'd need to be thought given to which subset of them would be most important.  

      I've created a pull-request here that includes changes to allow config items to tuned:  https://github.com/debezium/debezium/pull/2984

      All the defaults and code are based on Google's Kafka --> Pubsub connector with the idea being that these are reasonable defaults that can then be configured further (  https://github.com/GoogleCloudPlatform/pubsub/tree/master/kafka-connector)

            Unassigned Unassigned
            nathan-smit-1 Nathan Smit
            1 Vote for this issue
            3 Start watching this issue