Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-3168

Ruler pod throwing 'failed loading deletes for user' error after alerting/recording rules are created

XMLWordPrintable

    • False
    • None
    • False
    • NEW
    • OBSDA-115 - Create alerting rules based on logs
    • VERIFIED
    • Log Storage - Sprint 226, Log Storage - Sprint 227, Log Storage - Sprint 228

      Loki ruler pod throws below error after alerting and recording rules are created for app and infra tenant.

      Error:

      level=info ts=2022-10-12T21:43:36.668205523Z caller=metrics.go:143 component=ruler org_id=application latency=fast query="(count_over_time({kubernetes_namespace_name=\"my-app-2\"}[2m]) > 10)" query_type=metric range_type=instant length=0s start_delta=9.782397ms end_delta=9.782566ms step=0s duration=9.273324ms status=200 limit=0 returned_lines=0 throughput=23MB total_bytes=215kB total_entries=2 queue_time=0s subqueries=1 cache_chunk_req=0 cache_chunk_hit=0 cache_chunk_bytes_stored=0 cache_chunk_bytes_fetched=0 cache_index_req=0 cache_index_hit=0 cache_result_req=0 cache_result_hit=0
      level=error ts=2022-10-12T21:43:38.6501681Z caller=delete_requests_client.go:165 msg="error getting delete requests from the store" err="Get \"https://lokistack-dev-compactor-http.openshift-logging.svc.cluster.local:3100/loki/api/v1/delete\": x509: certificate signed by unknown authority"
      ts=2022-10-12T21:43:38.650205101Z caller=spanlogger.go:80 user=application level=error msg="failed loading deletes for user" err="Get \"https://lokistack-dev-compactor-http.openshift-logging.svc.cluster.local:3100/loki/api/v1/delete\": x509: certificate signed by unknown authority"
      level=info ts=2022-10-12T21:43:38.651954171Z caller=metrics.go:143 component=ruler org_id=application latency=fast query="(count_over_time({kubernetes_namespace_name=\"my-app\"}[2m]) > 10)" query_type=metric range_type=instant length=0s start_delta=8.477102ms end_delta=8.477272ms step=0s duration=7.50833ms status=200 limit=0 returned_lines=0 throughput=52MB total_bytes=391kB total_entries=2 queue_time=0s subqueries=1 cache_chunk_req=0 cache_chunk_hit=0 cache_chunk_bytes_stored=0 cache_chunk_bytes_fetched=0 cache_index_req=0 cache_index_hit=0 cache_result_req=0 cache_result_hit=0

      Steps to reproduce:

      1) Deploy Loki 5.6 Operator and provision LokiStack.

      2) Create Alerting and recording rules.

      3) Validate data in loki-ruler configmap.

      4) Check logs under loki-ruler pod after restart.

      OCP version

      4.12, 4.11

      Manifests:

      Application Alerting rules: http://pastebin.test.redhat.com/1077334

      Infra alerting rule: http://pastebin.test.redhat.com/1077335

       

              ptsiraki@redhat.com Periklis Tsirakidis
              rhn-support-kbharti Kabir Bharti
              Kabir Bharti Kabir Bharti
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: