Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-4836

[release-5.8] logging-loki-ruler pods are not injected with custom CA certificate defined in LokiStack CR

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done-Errata
    • Normal
    • Logging 5.8.1
    • Logging 5.7.7, Logging 5.8.0
    • Log Storage
    • False
    • None
    • False
    • NEW
    • NEW
    • Hide
      Before this update, the Loki Operator did not mount a custom CA Bundle to the ruler pods caused failing object storage access when evaluating alerting/recording rules. With this update, the Loki Operator mounting the custom CA bundle to all ruler pods resolves the issue and the ruler pods can download logs from object storage to evaluate alerting/recording rules.
      Show
      Before this update, the Loki Operator did not mount a custom CA Bundle to the ruler pods caused failing object storage access when evaluating alerting/recording rules. With this update, the Loki Operator mounting the custom CA bundle to all ruler pods resolves the issue and the ruler pods can download logs from object storage to evaluate alerting/recording rules.
    • Bug Fix
    • Log Storage - Sprint 245, Log Storage - Sprint 246
    • Moderate

    Description

      Description of problem:

      The CA certificate defined in LokiStack CR at "lokistack.spec.storage.tls.caName"  is not injected in loki-ruler pods but is injected in loki-ingester pods.

      Version-Release number of selected component (if applicable):

      Loki Operator 5.8

      How reproducible:

      100%

      Steps to Reproduce:

      1. Create LokiStack CR and inject a custom CA of the object storage at "lokistack.spec.storage.tls.caName"
      2. Enable lokistack.spec.rules and let the pods spin up
      3. Check the logs of logging-loki-ruler pods:

      level=error ts=2023-11-17T16:18:04.95523774Z caller=compat.go:78 user=infrastructure rule_name=k8spspallowedusers rule_type=alerting query="(sum(count_over_time({log_type=\"infrastructure\", kubernetes_namespace_name=\"openshift-gatekeeper-system\"} | openshift_labels_cluster_name=\"tenant-10\" | message=\".allowed-user-ranges.\" | message=\".K8sPSPAllowedUsers.\"[5m])) > 5)" query_hash=383797106 msg="rule evaluation failed" err="failed to load chunk 'infrastructure/1ce7e214bf1b9dfb/18bdda184da:18bde1043d2:d8ac40e2': failed to get s3 object: RequestError: send request failed\ncaused by: Get \"
      https://abc.dev.s3.example.int:443/observability-observability/infrastructure/1ce7e214bf1b9dfb/18bdda184da%3A18bde1043d2%3Ad8ac40e2
      \": tls: failed to verify certificate: x509: certificate signed by unknown authority"

      Actual results:

      The rule evaluation failed as ruler pods are unable to connect to object storage because of certificate verification failure.

      Expected results:

      The CA certificate should get injected in logging-loki-ruler pods and the rule evaluation should succeed.

      Additional info:

      As the CA certificate is injected in ingester pods at /etc/storage/ca, if the CA certificate is manually made available in /etc/storage/ca inside logging-loki-ruler pods, then it doesn't work.

      Attachments

        Activity

          People

            ptsiraki@redhat.com Periklis Tsirakidis
            rhn-support-dgautam Dhruv Gautam
            Anping Li Anping Li
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: