-
Bug
-
Resolution: Done
-
Minor
-
Logging 5.6.0
-
False
-
None
-
False
-
NEW
-
OBSDA-115 - Create alerting rules based on logs
-
VERIFIED
-
Log Storage - Sprint 226, Log Storage - Sprint 227, Log Storage - Sprint 228
Loki ruler pod throws below error after alerting and recording rules are created for app and infra tenant.
Error:
level=info ts=2022-10-12T21:43:36.668205523Z caller=metrics.go:143 component=ruler org_id=application latency=fast query="(count_over_time({kubernetes_namespace_name=\"my-app-2\"}[2m]) > 10)" query_type=metric range_type=instant length=0s start_delta=9.782397ms end_delta=9.782566ms step=0s duration=9.273324ms status=200 limit=0 returned_lines=0 throughput=23MB total_bytes=215kB total_entries=2 queue_time=0s subqueries=1 cache_chunk_req=0 cache_chunk_hit=0 cache_chunk_bytes_stored=0 cache_chunk_bytes_fetched=0 cache_index_req=0 cache_index_hit=0 cache_result_req=0 cache_result_hit=0 level=error ts=2022-10-12T21:43:38.6501681Z caller=delete_requests_client.go:165 msg="error getting delete requests from the store" err="Get \"https://lokistack-dev-compactor-http.openshift-logging.svc.cluster.local:3100/loki/api/v1/delete\": x509: certificate signed by unknown authority" ts=2022-10-12T21:43:38.650205101Z caller=spanlogger.go:80 user=application level=error msg="failed loading deletes for user" err="Get \"https://lokistack-dev-compactor-http.openshift-logging.svc.cluster.local:3100/loki/api/v1/delete\": x509: certificate signed by unknown authority" level=info ts=2022-10-12T21:43:38.651954171Z caller=metrics.go:143 component=ruler org_id=application latency=fast query="(count_over_time({kubernetes_namespace_name=\"my-app\"}[2m]) > 10)" query_type=metric range_type=instant length=0s start_delta=8.477102ms end_delta=8.477272ms step=0s duration=7.50833ms status=200 limit=0 returned_lines=0 throughput=52MB total_bytes=391kB total_entries=2 queue_time=0s subqueries=1 cache_chunk_req=0 cache_chunk_hit=0 cache_chunk_bytes_stored=0 cache_chunk_bytes_fetched=0 cache_index_req=0 cache_index_hit=0 cache_result_req=0 cache_result_hit=0
Steps to reproduce:
1) Deploy Loki 5.6 Operator and provision LokiStack.
2) Create Alerting and recording rules.
3) Validate data in loki-ruler configmap.
4) Check logs under loki-ruler pod after restart.
OCP version
4.12, 4.11
Manifests:
Application Alerting rules: http://pastebin.test.redhat.com/1077334
Infra alerting rule: http://pastebin.test.redhat.com/1077335