Uploaded image for project: 'OpenShift Service Mesh'
  1. OpenShift Service Mesh
  2. OSSM-6163

OSSM 2.5.0 Fix Prometheus and Grafana dashboard errors

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done-Errata
    • Icon: Critical Critical
    • OSSM 2.5.1
    • OSSM 2.5.0
    • Customer Impact
    • None
    • False
    • None
    • False
    • Release Notes
    • Release Notes
    • +
    • Hide
      Per Matej in comments:

      Before, Grafana UI displayed errors for all Service Mesh dashboards, incapable of presenting data. Now, all dashboards except Istio Service and Istio Workload dashboards function properly, displaying the relevant data.

      Per Yuanlin in comments:
      OSSM-6163 resolved two errors. Previously, in the Service Mesh control plane (SMCP) v2.5, an unstable Prometheus image was included in the Service Mesh operator manifest and users were not able to access the Prometheus Dashboard. Now, in the Service Mesh operator 2.5.1, the Prometheus image has been updated.
      Previously, in the Service Mesh control plane (SMCP), a Grafana data source was not able to set basic authentication password automatically and users were not able to view metrics from Prometheus in Grafana mesh dashboards.
      Now, a Grafana data source password is configured under the "secureJsonData" field. Metrics are displayed correctly in dashboards.

      Known Issue: OSSM-6267 After a data source is configured correctly in the Grafana, a data query returns authentication error.
      Users are not able to view data in the Istio service and Istio workload dashboards. This is a Grafana build issue and we will update it in a following release.
      Show
      Per Matej in comments: Before, Grafana UI displayed errors for all Service Mesh dashboards, incapable of presenting data. Now, all dashboards except Istio Service and Istio Workload dashboards function properly, displaying the relevant data. Per Yuanlin in comments: OSSM-6163 resolved two errors. Previously, in the Service Mesh control plane (SMCP) v2.5, an unstable Prometheus image was included in the Service Mesh operator manifest and users were not able to access the Prometheus Dashboard. Now, in the Service Mesh operator 2.5.1, the Prometheus image has been updated. Previously, in the Service Mesh control plane (SMCP), a Grafana data source was not able to set basic authentication password automatically and users were not able to view metrics from Prometheus in Grafana mesh dashboards. Now, a Grafana data source password is configured under the "secureJsonData" field. Metrics are displayed correctly in dashboards. Known Issue: OSSM-6267 After a data source is configured correctly in the Grafana, a data query returns authentication error. Users are not able to view data in the Istio service and Istio workload dashboards. This is a Grafana build issue and we will update it in a following release.
    • Bug Fix

      OSSM 2.5 Grafana not able to query metrics: client 403 error 

      How to reproduce:

      • Install OSSM operator 2.5.0
      • Create a 2.5 SMCP
      • Check Grafana UI from Route
      • Select Dashboard Istio Control Plane

      Resources Usage is showing error icon and client 403 error.

      We also need extend our integration test coverage. Add a new test case: send a request to Grafana Dashboard and then verify Grafana pod logs.

      Istio operator log error 

      {"level":"info","ts":1711312642.3412166,"logger":"webhookca-controller","msg":"could not get CA bundle","WebhookConfig":"mutating/istiod-basic-istio-system","caBundleSourceConfig":{"Namespace":"istio-system","SecretNameKeyPairs":[{"SecretName":"cacerts","Key":"ca-cert.pem"},{"SecretName":"cacerts","Key":"tls.crt"},{"SecretName":"istio-ca-secret","Key":"ca-cert.pem"},{"SecretName":"istiod-tls","Key":"tls.crt"}]},"error":"[Secret \"cacerts\" not found, Secret \"istio-ca-secret\" not found, Secret \"istiod-tls\" not found]","errorCauses":[{"error":"Secret \"cacerts\" not found"},{"error":"Secret \"cacerts\" not found"},{"error":"Secret \"istio-ca-secret\" not found"},{"error":"Secret \"istiod-tls\" not found"}]}

      grafana container error log

      logger=ngalert t=2024-04-03T22:33:39.445255391Z level=info msg="warming cache for startup"
logger=sqlstore.transactions t=2024-04-03T22:33:39.456503735Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0
logger=ticker t=2024-04-03T22:33:39.456680306Z level=info msg=starting first_tick=2024-04-03T22:33:40Z
logger=ngalert.multiorg.alertmanager t=2024-04-03T22:33:39.456709193Z level=info msg="starting MultiOrg Alertmanager"
logger=sqlstore.transactions t=2024-04-03T22:33:39.468742949Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1

      prometheus route error

      Error opening React index.html: open web/ui/static/react/index.html: no such file or directory

            yuaxu@redhat.com Yuanlin Xu
            yuaxu@redhat.com Yuanlin Xu
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: