-
Enhancement
-
Resolution: Done
-
Minor
-
None
-
None
Hi there,
This issue was discussed here: https://debezium.zulipchat.com/#narrow/channel/350571-community-dbz-server/topic/pubsub.2Ebatch.2Erequest.2Ebyte.2Ethreshold.20default.20exceeds.20limit
There is a narrow case where batches of large messages sent in pubsub will exceed the pubsub threshold of 10mb based on the default request threshold DBZ currently sets (debezium.sink.pubsub.batch.request.byte.threshold = 10000000)
This apparently is due to the request overhead not being factored in. Similar issue raised here: https://github.com/GoogleCloudPlatform/pubsub/issues/267
In response Google adjusted the default threshold to 9.5mb in their kafka connector for pubsub: https://github.com/GoogleCloudPlatform/pubsub/pull/275
I would like to implement the same adjustment and set the default to 9.5mb which users can then change if necessary. I just would prefer not to have a scenario where DBZ crashes using the default configs.