Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-8583

[release-6.4] Increase the timeout between Distributor and Ingester

XMLWordPrintable

    • Icon: Task Task
    • Resolution: Done
    • Icon: Normal Normal
    • Logging 6.4.2
    • Logging 6.4.1
    • Log Storage
    • None
    • Quality / Stability / Reliability
    • 1
    • False
    • Hide

      None

      Show
      None
    • False
    • Not Selected
    • NEW
    • VERIFIED
    • Before this update, a short timeout between the distributor and ingester component caused spurious HTTP 503 errors when ingesting logs into LokiStack, especially under high load. With this update, the timeout has been increased.
    • Bug Fix
    • Logging - Sprint 282, Logging - Sprint 283

      Context

      With the Loki Operator deployment of RHOBS we have learned that the current timeout that we configure between the Distributor and Ingesters is way too low which results in random 500 when the amount of Logs ingested increases momentarily. The goal of this issue is to configure this threshold to a higher value.

      Acceptance criteria

      • Increase the timeout between the Distributor and Ingester to a reasonable value

      Developer notes

      • Some testing has been done on a test cluster and the p99 value of this threshold seems to be at 2.5s

              rojacob@redhat.com Robert Jacob
              jmarcal@redhat.com Joao Marcal
              Kabir Bharti Kabir Bharti
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: