Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-5026

Log Collector Output Tuning

XMLWordPrintable

    • Icon: Epic Epic
    • Resolution: Done-Errata
    • Icon: Critical Critical
    • Logging 5.9.0
    • None
    • Log Collection
    • None
    • Log Collector tuning
    • False
    • None
    • False
    • Green
    • NEW
    • Administer, API, Deploy, Instructions, Migrate
    • To Do
    • OBSDA-549 - Reliability and performance tuning for log collection
    • OBSDA-549Reliability and performance tuning for log collection
    • NEW
    • 0% To Do, 0% In Progress, 100% Done
    • Hide
      This feature introduces the capability to tune some output settings (e.g. compression, retry duration, max payloads) to match characteristics of the receiver. Additionally, this features adds a delivery mode to allow administrators to choose between throughput and log durability. For example, AtLeastOnce configures minimal disk buffering of collected logs so those logs can be delivered after collector restarts.
      Show
      This feature introduces the capability to tune some output settings (e.g. compression, retry duration, max payloads) to match characteristics of the receiver. Additionally, this features adds a delivery mode to allow administrators to choose between throughput and log durability. For example, AtLeastOnce configures minimal disk buffering of collected logs so those logs can be delivered after collector restarts.
    • Feature

      Goals

      Enhance the ClusterLogforwarder API to

      • Allow tuning of individual output to support the unique characteristics of the receiver
      • Reduce the possibility of log loss when the collector restarts
      • Define a simple way to choose between throughput and durability of logs

      Non-Goals

      • Exposing the entirety of output configuration options for the underlying collector implementation

      Motivation

      • Some customers require collected log messages to survive a collector restart to support their regulatory mandates
      • Some customers use outputs (e.g. Cloudwatch) that have hard limitations to the size of batches they can receive

      Alternatives

      Acceptance Criteria

      • Verify their is no regression in log throughput and durability when the log forwarder does not spec any tuning
      • Verify collected log messages are not lost when the output for a log forwarder is optimized for log durability

      Risk and Assumptions

      Documentation Considerations

      • API documentation to support the added fields
      • Usage documentation to explain the feature

      Open Questions

      • How does e2e acks compare to disk buffering

      Additional Notes

          1.
          		 QE Tracker Sub-task Closed Undefined Unassigned

              jcantril@redhat.com Jeffrey Cantrill
              jcantril@redhat.com Jeffrey Cantrill
              Anping Li Anping Li
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Created:
                Updated:
                Resolved: