Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-9144

PubSub Sink default maxBufferBytes allows the request to exceed the limit

XMLWordPrintable

    • Icon: Enhancement Enhancement
    • Resolution: Done
    • Icon: Minor Minor
    • 3.2.0.CR1
    • None
    • debezium-server
    • None

      Hi there,

      This issue was discussed here: https://debezium.zulipchat.com/#narrow/channel/350571-community-dbz-server/topic/pubsub.2Ebatch.2Erequest.2Ebyte.2Ethreshold.20default.20exceeds.20limit

      There is a narrow case where batches of large messages sent in pubsub will exceed the pubsub threshold of 10mb based on the default request threshold DBZ currently sets (debezium.sink.pubsub.batch.request.byte.threshold = 10000000)

      This apparently is due to the request overhead not being factored in. Similar issue raised here: https://github.com/GoogleCloudPlatform/pubsub/issues/267

      In response Google adjusted the default threshold to 9.5mb in their kafka connector for pubsub: https://github.com/GoogleCloudPlatform/pubsub/pull/275

      I would like to implement the same adjustment and set the default to 9.5mb which users can then change if necessary. I just would prefer not to have a scenario where DBZ crashes using the default configs.

              Unassigned Unassigned
              nathan-smit-1 Nathan Smit
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated:
                Resolved: