Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-4375

Allow additional config options for Debezium Server Pubsub Connector

XMLWordPrintable

    • Icon: Enhancement Enhancement
    • Resolution: Done
    • Icon: Minor Minor
    • 1.9.0.Alpha1
    • None
    • debezium-server
    • None
    • False
    • False

      Currently, the Google Cloud Pub/Sub connector included with Debezium Server does not make use of any of the additional configuration options available to the Pub/Sub publisher.  Without additional options enabled, we have only been able to stream at ~300 rows per minute

      Specifically, I would like to add config options for

      1) Batch settings:

      https://cloud.google.com/pubsub/docs/samples/pubsub-publisher-batch-settings

      2) Retry settings:

      https://cloud.google.com/pubsub/docs/samples/pubsub-publisher-retry-settings

      3) Flow Control:

      https://cloud.google.com/pubsub/docs/samples/pubsub-publisher-flow-control

      Ability to change these settings will greatly assist us in being able to tune the performance when pushing messages to the sink.

      There are many configurable items for pubsub, so there'd need to be thought given to which subset of them would be most important.  

      I've created a pull-request here that includes changes to allow config items to tuned:  https://github.com/debezium/debezium/pull/2984

      All the defaults and code are based on Google's Kafka --> Pubsub connector with the idea being that these are reasonable defaults that can then be configured further (  https://github.com/GoogleCloudPlatform/pubsub/tree/master/kafka-connector)

              Unassigned Unassigned
              nathan-smit-1 Nathan Smit
              Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: