Details
-
Bug
-
Resolution: Not a Bug
-
Minor
-
None
-
Logging 5.6.0
-
False
-
None
-
False
-
NEW
-
NEW
-
Release Note Not Required
-
Log Storage - Sprint 229, Log Storage - Sprint 230, Log Storage - Sprint 231, Log Storage - Sprint 232, Log Storage - Sprint 238
Description
Description:
Querying logs from loki using results in 'error processing request' and content cancelled errors under Loki querier and query-frontend
Logs
query-frontend:
level=error ts=2022-11-22T02:33:03.151833243Z caller=retry.go:73 org_id=audit msg="error processing request" try=0 err="context canceled" level=error ts=2022-11-22T02:32:15.205605655Z caller=retry.go:73 org_id=application msg="error processing request" try=0 err="context canceled"
querier:
level=error ts=2022-11-22T03:34:06.778980278Z caller=series_index_store.go:583 org_id=audit msg="error querying storage" err="rpc error: code = Canceled desc = context canceled"711level=info ts=2022-11-22T03:34:06.779072335Z caller=metrics.go:143 component=querier org_id=audit latency=fast query="{log_type=\"audit\"}" query_type=limited range_type=range length=25m53.555978234s start_delta=1h0m0.335046042s end_delta=34m6.779068008s step=14s duration=945.416µs status=500 limit=30 returned_lines=0 throughput=0B total_bytes=0B total_entries=0 queue_time=0s subqueries=1 cache_chunk_req=0 cache_chunk_hit=0 cache_chunk_bytes_stored=0 cache_chunk_bytes_fetched=0 cache_index_req=0 cache_index_hit=0 cache_result_req=0 cache_result_hit=0712level=warn ts=2022-11-22T03:34:06.779116196Z caller=logging.go:86 orgID=audit msg="GET /loki/api/v1/query_range?direction=BACKWARD&end=1669086000000000000&limit=30&query=%7Blog_type%3D%22audit%22%7D&shards=4_of_16&start=1669084446444021766&step=14.000000 (500) 1.146168ms Response: \"rpc error: code = Canceled desc = context canceled\\n\" ws: false; X-Scope-Orgid: audit; "713level=error ts=2022-11-22T03:34:06.779151123Z caller=frontend_processor.go:145 msg="error processing requests" err=EOF
CSV: loki-operator.v5.6.0
Steps to reproduce:
1) Deploy Loki and provision LokiStack CR.
2) Create CLF to send all tenant logs to Loki
3) Create CL and Configure loki as logStore.
4) Query logs using logcli for e.g
logcli -o raw --tls-skip-verify --bearer-token="$(oc whoami -t)" --addr "https://<LOKISTACK-ROUTE>/api/logs/v1/audit" query '{log_type="audit"}'
Actual Results:
logcli returns the result but errors are seen on querier and query frontend logs.
Expected result:
logcli should return response without errors under loki components.