Uploaded image for project: 'Red Hat Advanced Cluster Management'
  1. Red Hat Advanced Cluster Management
  2. ACM-14988

2.12 Observability known issue: Dashboard failed to load

    • False
    • None
    • False
    • None

      Note: Doc team updates the current version of the documentation and the
      two previous versions (n-2), but we address *only high-priority, or
      customer-reported issues* for -2 releases in support.
      Describe the changes in the doc and link to your dev story:

      1. - [x] Mandatory: Add the required version to the Fix version/s field.

      2. - [ ] Mandatory: Choose the type of documentation change or review.

      • [x] We need to update to an existing topic
      • [ ] We need to add a new document to an existing section
      • [ ] We need a whole new section; this is a function not
        documented before and doesn't belong in any current section
      • [ ] We need an Operator Advisory review and approval
      • [ ] We need a z-Stream (Errata) Advisory and Release note
        for MCE and/or ACM

      3. - [ ] *Mandatory: *Use the following link to open the doc and find where the
      documentation update should go. Note: As the feature and doc is
      understood and developed, this placement decision may change:

      4. - [ ] Mandatory for GA content:

      • [x] Add steps, the diff, known issue, and/or other important
        conceptual information in the following space:

      Title: Grafana Dashboard missing
      In rare occasions a Grafana dashboard might fail to be loaded into the running Grafana instance, causing the dashboard to be missing in Grafana. To verify whether a dashboard has failed to load, the logs can be consulted, such as (note that you should check the logs for all running Grafana replicas):

      # oc logs observability-grafana-68f8489659-m79rv -c grafana-dashboard-loader -n open-cluster-management-observability
      ...
      E1017 12:55:24.532493 1 dashboard_controller.go:147] dashboard: sample-dashboard could not be created after retrying 40 times
      

      Note the above log line, which mentions that a specific dashboard failed to be created. In order to fix this issue, redeploy Grafana by scaling down the number of replicas to 0. The Multicluster Observability Operator will automatically scale up the number of replicas to the desired amount afterwards:

      oc scale deployment observability-grafana -n open-cluster-management-observability --replicas=0
      
      • [x] *Add Required access level *(example, *Cluster
        Administrator*) for the user to complete the task:

      Cluster admin

      • [x] Add verification at the end of the task, how does the user
        verify success (a command to run or a result to see?)

      To verify that the dashboard now loads correctly, check the logs of all Grafana pods and ensure no error message is present:

      oc logs observability-grafana-68f8489659-h6jd9 -c grafana-dashboard-loader -n open-cluster-management-observability | grep "could not be created"
      
      • [x] Add link to dev story here: ACM-14639 n(this dev story partially fixes the problem, but it may still occur on rare occasions. No decisions has currently be made whether we need to do more)

      5. - [ ] Mandatory for bugs: What is the diff? Clearly define what the
      problem is, what the change is, and link to the current documentation. Only
      use this for a documentation bug.

              mdockery@redhat.com Mikela Jackson (Inactive)
              rh-ee-jachanse Jacob Baungard Hansen
              Xiang Yin Xiang Yin
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: