Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-4375

Allow additional config options for Debezium Server Pubsub Connector

    XMLWordPrintable

Details

    • Enhancement
    • Resolution: Done
    • Minor
    • 1.9.0.Alpha1
    • None
    • debezium-server
    • None
    • False
    • False

    Description

      Currently, the Google Cloud Pub/Sub connector included with Debezium Server does not make use of any of the additional configuration options available to the Pub/Sub publisher.  Without additional options enabled, we have only been able to stream at ~300 rows per minute

      Specifically, I would like to add config options for

      1) Batch settings:

      https://cloud.google.com/pubsub/docs/samples/pubsub-publisher-batch-settings

      2) Retry settings:

      https://cloud.google.com/pubsub/docs/samples/pubsub-publisher-retry-settings

      3) Flow Control:

      https://cloud.google.com/pubsub/docs/samples/pubsub-publisher-flow-control

      Ability to change these settings will greatly assist us in being able to tune the performance when pushing messages to the sink.

      There are many configurable items for pubsub, so there'd need to be thought given to which subset of them would be most important.  

      I've created a pull-request here that includes changes to allow config items to tuned:  https://github.com/debezium/debezium/pull/2984

      All the defaults and code are based on Google's Kafka --> Pubsub connector with the idea being that these are reasonable defaults that can then be configured further (  https://github.com/GoogleCloudPlatform/pubsub/tree/master/kafka-connector)

      Attachments

        Activity

          People

            Unassigned Unassigned
            nathan-smit-1 Nathan Smit
            Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: