Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-3498

Loki returning timed out after 30000ms

XMLWordPrintable

    • False
    • None
    • False
    • auth
    • NEW
    • ASSIGNED
    • Before this update, query histogram from webconsole longer than 30 seconds to timeout. With this update, the timeout value can be configured in the configmap/logging-view-plugin
    • Moderate

      Description of problem:

      When using Loki as Logstore and accessing to {}OCP Console -> Observe -> Logs{} when it's taking more than 30 seconds in returning the data requested, it's observed an error as:

      LOG: Request: /api/proxy/plugin/logging-view-plugin/backend/api/logs/v1/application/loki/api/v1/query_range?query=sum+by+%28level%29+%28count_over_time%28%7B+log_type%3D%7E%22.%2B%22+%7D+%7C+json+%5B1h%5D%29%29&start=1669969260000000000&end=1670228460000000000&step=1h timed out after 30000ms.

      After discussion, it seems that the 30s of timeout is coming from the value defined as constant in the logging-view-plugin in:
      https://github.com/openshift/logging-view-plugin/blob/f19441a4b9c4ce6910f62b19703c553b1051bbf5/src/cancellable-fetch.ts#L33

      Usually, this problem happens when asking for logs for several days.

      The bug is opened because this timeout coming from the logging-view-plugin is not the same timeout that is configured in Loki, that it's bigger, they should both use the same timeout.

      Version-Release number of selected component (if applicable):

      $ oc get csv
      oc getNAME                    DISPLAY                     VERSION   REPLACES   PHASE
      cluster-logging.5.5.5   Red Hat OpenShift Logging   5.5.5                Succeeded
      loki-operator.5.5.5     Loki Operator               5.5.5                Succeeded
      

      How reproducible:

      Always

      Steps to Reproduce:

      1) Create a OCP cluster with pods logging massively.
      2) Select a time range from OCP -> Observe -> Logs to get this reproduced. For example, 2 days or 3 days.

      Actual results:

      Returned timeout error after 30 seconds

      Expected results:

      Returned the logs requested

      NOTE: take into consideration that by default, it's 30 days the days of retention logs, then, the timeout should be big enough to return the logs when asking for several days
       

       

        1. image-2023-02-28-16-48-43-060.png
          57 kB
          Giriyamma Karagere Ramaswamy
        2. image-2023-03-02-23-11-33-962.png
          201 kB
          Giriyamma Karagere Ramaswamy

              gbernal@redhat.com Gabriel Bernal
              rhn-support-ocasalsa Oscar Casal Sanchez
              Giriyamma Karagere Ramaswamy Giriyamma Karagere Ramaswamy (Inactive)
              Votes:
              2 Vote for this issue
              Watchers:
              8 Start watching this issue

                Created:
                Updated:
                Resolved: