-
Enhancement
-
Resolution: Done
-
Minor
-
None
-
None
-
False
-
False
-
Currently, the Google Cloud Pub/Sub connector included with Debezium Server does not make use of any of the additional configuration options available to the Pub/Sub publisher. Without additional options enabled, we have only been able to stream at ~300 rows per minute
Specifically, I would like to add config options for
1) Batch settings:
https://cloud.google.com/pubsub/docs/samples/pubsub-publisher-batch-settings
2) Retry settings:
https://cloud.google.com/pubsub/docs/samples/pubsub-publisher-retry-settings
3) Flow Control:
https://cloud.google.com/pubsub/docs/samples/pubsub-publisher-flow-control
Ability to change these settings will greatly assist us in being able to tune the performance when pushing messages to the sink.
There are many configurable items for pubsub, so there'd need to be thought given to which subset of them would be most important.
I've created a pull-request here that includes changes to allow config items to tuned: https://github.com/debezium/debezium/pull/2984
All the defaults and code are based on Google's Kafka --> Pubsub connector with the idea being that these are reasonable defaults that can then be configured further ( https://github.com/GoogleCloudPlatform/pubsub/tree/master/kafka-connector)