See the discussion at https://coreos.slack.com/archives/CB3HXM2QK/p1617720645002700

Three issues around forwarding logs to Kafka have been identified:

• When forwarding logs to Kafka with OOTB configuration for message.max.bytes fluentd gets a MessageSizeTooLarge reply. It appears that the fluentd Kafka plugin forwards the entire chunks. See

https://github.com/openshift/cluster-logging-operator/pull/425#issue-391018432

https://github.com/openshift/cluster-logging-operator/pull/425#issuecomment-601216905

for background. Currently we suggest to customers to increase their message.max.bytes over the size of the fluent's chunk, which is 8MB atm. It may not be an option for some customers.
There's a chance what can help decouple the Kafka message size from the Fluent chunk size is the following parameter in the fluent's Kafka plugin
max_send_limit_bytes - default: nil - Max byte size to send message to avoid MessageSizeTooLarge.
https://github.com/fluent/fluent-plugin-kafka.

• We need to document what Kafka versions we support, as customers have run into incompatibilties. We can start from https://github.com/zendesk/ruby-kafka#compatibility and  https://github.com/edenhill/librdkafka/blob/master/INTRODUCTION.md#broker-version-compatibility.

• Document the end result at https://docs.openshift.com/container-platform/4.7/logging/cluster-logging-external.html#cluster-logging-collector-log-forward-kafka_cluster-logging-external