-
Bug
-
Resolution: Not a Bug
-
Undefined
-
None
-
None
-
False
-
None
-
False
-
NEW
-
NEW
-
Bug Fix
-
-
- Loki queries reacting very slowly or timing out (30000ms)
- Set longer timeout on visualization but issue persists; including "Context deadline exceeded" (which is another form of timeout)
- total log data does not seem to be a factor here; we are quite below the tested size limits, and read actions that should be instantaneous (like the container dropdown box) are taking seconds or even timing out
- we tried querying directly with LogCLI on commandline and saw intermittent connection refused, which could possibly cause the issue if the Loki components themselves are also experiencing intermittent connection problems
- the index gateway fails when local files are present, and doesnt fail without local files. We are considering if this could be a local storage issue
- After testing we do not suspect storage is the bottleneck
$ date; logcli -o raw --tls-skip-verify --bearer-token="$(oc whoami -t)" --addr "https://logging-loki-openshift-logging.apps.dev.example.com/api/logs/v1/application" labels 'kubernetes_pod_name' | wc -l Mon Mar 25 08:41:02 UTC 2024 2024/03/25 08:41:02 https://logging-loki-openshift-logging.apps.example.com/api/logs/v1/application/loki/api/v1/label/kubernetes_pod_name/values?end=1711356062291259910&start=1711352462291259910 4960 $ date; oc get namespaces | wc -lMon Mar 25 08:41:31 UTC 2024 527 $ date; oc get pods --all-namespaces | wc -l Mon Mar 25 08:41:56 UTC 2024 3041
