Uploaded image for project: 'Observability Documentation'
  1. Observability Documentation
  2. OBSDOCS-2959

Enhance Vector kafa sink tuning parameters

XMLWordPrintable

    • Icon: Feature Request Feature Request
    • Resolution: Unresolved
    • Icon: Normal Normal
    • None
    • None
    • Logging
    • False
    • Hide

      None

      Show
      None
    • False

      • The Red Hat Logging Operator Vector implementation does not support all configuration parameters that are available in the upstream Vector project.
      • Currently, the Kafka tuning specification in the Logging Operator supports only the following parameters: `deliveryMode`, `maxWrite`, and `compression`.
      • This request is a feature enhancement to extend the Kafka sink tuning parameters supported by Vector.
      • This feature is requested to gain better visibility and clarity on how retries are handled by the ClusterLogForwarder Vector Kafka sink.
      • When errors such as "Service call failed" or "No retries or retries exhausted" are encountered, it is not clear whether retry attempts were actually performed or whether the configured retries were exhausted.
      • There is an upstream issue [1] raised regarding the lack of retry support in Vector for Kafka. This issue also references retry-related configurations available in the librdkafka library used by Vector.
      • The Vector documentation does not provide details about retry behavior for the Kafka sink. However, relevant retry parameters are documented in the librdkafka documentation [2].
      • This request proposes adding the following Kafka sink tuning parameters:
        message.timeout.ms
        delivery.timeout.ms
        message.send.max.retries
        retries
        retry.backoff.ms
        retry.backoff.max.ms 
      • For additional context and reference, the upstream documentation link [3] is provided.

       [1] https://github.com/vectordotdev/vector/issues/8468 
       [2] https://docs.confluent.io/platform/current/clients/librdkafka/html/md_CONFIGURATION.html
       [3] https://vector.dev/docs/reference/configuration/sinks/kafka/#librdkafka_options 

              Unassigned Unassigned
              rhn-support-pripatil Prithviraj Patil
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated: