Uploaded image for project: 'OpenShift Service Mesh'
  1. OpenShift Service Mesh
  2. OSSM-6267

Grafana Istio Service and Workload dashboards error

    • False
    • None
    • False
    • Release Notes
    • Release Notes
    • Hide
      Previously, after a data source was configured in the OSSM 2.5 Grafana, users would get a data query authentication error. Users were also not able to view data in the Istio service and Istio workload dashboards. Now, the Grafana error is fixed when an existing 2.5 SMCP is upgraded to version 2.5.2 or above.
      Show
      Previously, after a data source was configured in the OSSM 2.5 Grafana, users would get a data query authentication error. Users were also not able to view data in the Istio service and Istio workload dashboards. Now, the Grafana error is fixed when an existing 2.5 SMCP is upgraded to version 2.5.2 or above.
    • Bug Fix

      When you open Istio Service or Istio Workload dashboards in Grafana, they are not loaded correctly and this error shows up:

      The browser console contains the following error

      runRequest.catchError {"message":"\"Forbidden\"","status":403,"statusText":"Forbidden"}  

      and a request (in browser network monitor) on grafana api /api/datasources/1/resources/api/v1/query shows OCP login page as a response ( see image )

      Prometheus pod contains the following error: ( mkralik is a user in htpasswd idp )

      2024/04/08 13:28:38 oauthproxy.go:790: basicauth: 10.131.0.112:48962 mkralik not in HtpasswdFile
      

      I have also tried with Kubeadmin, the same error:

      2024/04/08 13:31:16 oauthproxy.go:790: basicauth: 10.131.0.112:56654 kube not in HtpasswdFile
      

        1. image.png
          image.png
          127 kB
        2. image(1).png
          image(1).png
          396 kB
        3. Screenshot from 2024-05-03 10-50-05.png
          Screenshot from 2024-05-03 10-50-05.png
          164 kB
        4. Screenshot from 2024-05-03 10-48-56.png
          Screenshot from 2024-05-03 10-48-56.png
          197 kB

            [OSSM-6267] Grafana Istio Service and Workload dashboards error

            Yuanlin Xu added a comment -

            Hi rhn-support-gmonahan , I have updated the "Release Notes Text". Sorry for missing your comment question earlier. This has been fixed. We can include this release note change in the next release. Thanks.

            Yuanlin Xu added a comment - Hi rhn-support-gmonahan , I have updated the "Release Notes Text". Sorry for missing your comment question earlier. This has been fixed. We can include this release note change in the next release. Thanks.

            Hey yuaxu@redhat.com ,

            The "Release Note Text" still says: "Known Issue: OSSM-6267 After a data source is configured correctly in the Grafana, a data query returns authentication error.
            Users are not able to view data in the Istio service and Istio workload dashboards."

            The "Release Note Type" is still "Known Issue" so it is still listed as such in 2.x release notes.

            I asked on 04/25/2024 https://issues.redhat.com/browse/OSSM-6267?focusedId=24607647&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-24607647 if this issue was fixed for 2.5.2 and to update the "Release Notes Text" field accordingly.

            Since the "Release Notes Text" field was not updated, no change has been made to this issue in 2.x release notes.

            If this has been fixed, please update the "Release Note Text" box accordingly: "Previously, <2-3 sentences describing the issue>. Now, <2-3 sentences describing the fix>" so it can be included in the next release.

            Gwynne Monahan added a comment - Hey yuaxu@redhat.com , The "Release Note Text" still says: "Known Issue:  OSSM-6267  After a data source is configured correctly in the Grafana, a data query returns authentication error. Users are not able to view data in the Istio service and Istio workload dashboards." The "Release Note Type" is still "Known Issue" so it is still listed as such in 2.x release notes. I asked on 04/25/2024 https://issues.redhat.com/browse/OSSM-6267?focusedId=24607647&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-24607647 if this issue was fixed for 2.5.2 and to update the "Release Notes Text" field accordingly. Since the "Release Notes Text" field was not updated, no change has been made to this issue in 2.x release notes. If this has been fixed, please update the "Release Note Text" box accordingly: "Previously, <2-3 sentences describing the issue>. Now, <2-3 sentences describing the fix>" so it can be included in the next release.

            Yuanlin Xu added a comment -

            rhn-support-gmonahan This issue has been fixed in OSSM 2.5.2. What do you mean "moved into fixed issues" ? I thought this has already been marked as fixed issue.

            Yuanlin Xu added a comment - rhn-support-gmonahan This issue has been fixed in OSSM 2.5.2. What do you mean "moved into fixed issues" ? I thought this has already been marked as fixed issue.

            Hey yuaxu@redhat.com ,

            This Jira was brought to my attention by a support engineer.

            The 2.x release notes list this as a "Known issue" with no work around.

            The "Release Notes Text" field has not been updated, but I see you have left a comment with the words "fix" and "work around." Does that mean this issue is fixed, and should be moved into "Fixed issues"? Or is there now a work around for it as a "Known issue"?

            Gwynne Monahan added a comment - Hey yuaxu@redhat.com , This Jira was brought to my attention by a support engineer. The 2.x release notes list this as a "Known issue" with no work around. The "Release Notes Text" field has not been updated, but I see you have left a comment with the words "fix" and "work around." Does that mean this issue is fixed, and should be moved into "Fixed issues"? Or is there now a work around for it as a "Known issue"?

            Yuanlin Xu added a comment - - edited

            The fix is in OSSM 2.5.2 . No manual operation needed after upgrade.
            If user need to configure custom Grafana Datasource, here is an example of workaround steps

            After creating SMCP 2.5, update the configmap `istio-grafana`
            For example, https://github.com/maistra/istio-operator/pull/1767/files#diff-8025b9e65c95d796ed881adf4443ecd158acb20d972b43625f4061df80464c74R70
            1. Add `httpHeaderName1: 'Authorization'` and remove `basicAuth: true`, `basicAuthUser: internal`

            2a. For OSSM 2.5.2, the {AuthToken} value is in the istio-system ns secret `htpasswd` field data.authToken
            Replace the secureJsonData fields with 

            secureJsonData:
                      httpHeaderValue1: '{AuthToken}'

            Save the configmap `istio-grafana`

            3. Restart the grafana pod in the SMCP

             

            Yuanlin Xu added a comment - - edited The fix is in OSSM 2.5.2 . No manual operation needed after upgrade. If user need to configure custom Grafana Datasource, here is an example of workaround steps After creating SMCP 2.5, update the configmap `istio-grafana` For example, https://github.com/maistra/istio-operator/pull/1767/files#diff-8025b9e65c95d796ed881adf4443ecd158acb20d972b43625f4061df80464c74R70 1. Add `httpHeaderName1: 'Authorization'` and remove `basicAuth: true`, `basicAuthUser: internal` 2a. For OSSM 2.5.2, the {AuthToken} value is in the istio-system ns secret `htpasswd` field data.authToken Replace the secureJsonData fields with  secureJsonData:           httpHeaderValue1: '{AuthToken}' Save the configmap `istio-grafana` 3. Restart the grafana pod in the SMCP  

            Yuanlin Xu added a comment - - edited

            Do I need to consider backward compatibility for my change working with existing Grafana 7 ?

            Yuanlin Xu added a comment - - edited Do I need to consider backward compatibility for my change working with existing Grafana 7 ?

            Yuanlin Xu added a comment - - edited

            related upstream issue: [-https://github.com/grafana/grafana/issues/16638-]

            Yuanlin Xu added a comment - - edited related upstream issue: [-https://github.com/grafana/grafana/issues/16638-]

            Yuanlin Xu added a comment -

            Investigation still in progress. It's probably caused by a feature added in Grafana 9.1:
            https://grafana.com/blog/2022/08/25/new-in-grafana-alerting-file-provisioning/

            In 2.5.0 SMCP is creating Grafana 9.2.10 and log shows :

            "can't read alerting provisioning files from directory" path=/etc/grafana/provisioning/alerting error="open /etc/grafana/provisioning/alerting: no such file or directory"
            

            First, we do need to pin a version of Grafana when we build grafana image in CPaaS:
            https://gitlab.cee.redhat.com/istio/cpaas/containers-midstream/-/blob/rhossm-2.5-rhel-8/distgit/containers/istio-grafana/Dockerfile.in?ref_type=heads#L18
            Next, I am trying to update the deployment helm chart and verify if that works.

            Yuanlin Xu added a comment - Investigation still in progress. It's probably caused by a feature added in Grafana 9.1: https://grafana.com/blog/2022/08/25/new-in-grafana-alerting-file-provisioning/ In 2.5.0 SMCP is creating Grafana 9.2.10 and log shows : "can't read alerting provisioning files from directory" path=/etc/grafana/provisioning/alerting error= "open /etc/grafana/provisioning/alerting: no such file or directory" First, we do need to pin a version of Grafana when we build grafana image in CPaaS: https://gitlab.cee.redhat.com/istio/cpaas/containers-midstream/-/blob/rhossm-2.5-rhel-8/distgit/containers/istio-grafana/Dockerfile.in?ref_type=heads#L18 Next, I am trying to update the deployment helm chart and verify if that works.

            Gwynne Monahan added a comment - - edited

            Release Note Text originally written here: https://issues.redhat.com/browse/OSSM-6163 but it applies to OSSM-6267 so adding here.

            Gwynne Monahan added a comment - - edited Release Note Text originally written here: https://issues.redhat.com/browse/OSSM-6163 but it applies to OSSM-6267 so adding here.

              yuaxu@redhat.com Yuanlin Xu
              mkralik@redhat.com Matej Kralik
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated:
                Resolved: