Normal 1. Proposed title of this feature request
Forward to multiple Kafka brokers per fluentd output
2. What is the nature and description of the request?
Allow an output to forward to multiple Kafka brokers. Currently only one broker could be specify
3. Why does the customer need this? (List the business requirements here)
To ensure high availability customer wants to set multiple Kafka brokers per output. According to customer it is a common configuration in Kafka landscape. Ultimately as an example this could specify this way
spec: outputs: - name: app-logs type: kafka url: - tls://kafka1.example.devlab.com:9093/app-topic - tls://kafka2.example.devlab.com:9093/app-topic
However fluentd plugin upstream code seems to be design to loop through a list of brokers already but I'm not sure if I totally understood the code
openshift/cluster-logging-operator/kafka_conf.go-L19]
if conf.Target.Kafka.Brokers != nil { brokers := []string{} for _, broker := range conf.Target.Kafka.Brokers { b := parseBroker(broker) if b != "" { brokers = append(brokers, b) } } if len(brokers) > 0 { return strings.Join(brokers, ",") } }
openshift/cluster-logging-operator/output_conf_kafka_test.go-L163
It("should produce well formed output label config", func() { outputs = []v1.OutputSpec{ { Type: v1.OutputTypeKafka, Name: "kafka-receiver", OutputTypeSpec: v1.OutputTypeSpec{ Kafka: &v1.Kafka{ Topic: "topic", Brokers: []string{ "tls://broker1-kafka.svc.messaging.cluster.local:9092", "tls://broker2-kafka.svc.messaging.cluster.local:9092", }, }, }, }, }
4. List any affected packages or components.
Unknown