Uploaded image for project: 'Observability and Data Analysis Program'
  1. Observability and Data Analysis Program
  2. OBSDA-1192

Allow to pass a partition key to a kafka output in OpenShift Logging

XMLWordPrintable

    • Icon: Feature Feature
    • Resolution: Unresolved
    • Icon: Undefined Undefined
    • None
    • Logging 6.0, Logging 6.1, Logging 6.2
    • PM Logging
    • None
    • False
    • Hide

      None

      Show
      None
    • False
    • Not Selected
    • 0

      Proposed title of this feature request

      Allow to pass a partition key to a kafka output in OpenShift Logging

      What is the nature and description of the request?

      Currently doesn't seem to be possible to configure a partition key to a kafka output in OpenShift Logging. The following API documentation provides the only available configuration fields for a Kafka output:

      $ oc explain clf.spec.outputs.kafka
      GROUP:      observability.openshift.io
      KIND:       ClusterLogForwarder
      VERSION:    v1
      
      FIELD: kafka <Object>
      
      
      DESCRIPTION:
          Kafka configures forwarding log events to Apache Kafka topics
          
      FIELDS:
        authentication	<Object>
          Authentication sets credentials for authenticating the requests.
      
        brokers	<[]string>
          Brokers specifies the list of broker endpoints of a Kafka cluster.
          
          The list represents only the initial set used by the collector's Kafka
          client for the
          first connection only. The collector's Kafka client fetches constantly an
          updated list
          from Kafka. These updates are not reconciled back to the collector
          configuration.
          
          If none provided the target URL from the OutputSpec is used as fallback.
      
        topic	<string>
          Topic specifies the target topic to send logs to. The value when not
          specified is 'topic'
          
          The Topic can be a combination of static and dynamic values consisting of
          field paths followed by `||` followed by another field path or a static
          value.
          
          A dynamic value is encased in single curly brackets `{}` and MUST end with a
          static fallback value separated with `||`.
          
          Static values can only contain alphanumeric characters along with dashes,
          underscores, dots and forward slashes.
          
          Example:
          
           1. foo-{.bar||"none"}
          
           2. {.foo||.bar||"missing"}
          
           3.
          foo.{.bar.baz||.qux.quux.corge||.grault||"nil"}-waldo.fred{.plugh||"none"}
      
        tuning	<Object>
          Tuning specs tuning for the output
      
        url	<string>
          URL to send log records to.
          
          The 'username@password' part of `url` is ignored.
      

      The above shows that it is possible to specify a topic, but not a partition key.

      Specifically this link describes what a partition key in Kafka means:
      https://www.confluent.io/learn/kafka-partition-key/

      Why does the customer need this? (List the business requirements)

      Customers are getting requests for this feature from their developers

      List any affected packages or components.

      OpenShift Logging 6.x
      ClusterLogForwarder CRD

              jamparke@redhat.com Jamie Parker
              fminafra-redhat Francesco Minafra
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated: