-
Bug
-
Resolution: Done
-
Normal
-
rhos-17.1.4
-
None
-
2
-
False
-
-
False
-
?
-
None
-
-
-
Moderate
During the FFU upgrade of a disconnected OpenStack instance, we observed some unexpected attempts by Grafana to access the internet [2] after successfully completing the upgrading to CEPH 7. [1].
This issue was solved with the support of the CEPH SBR: the grafana.ini.j2 was replaced and grafana reconfigured as last remediation step [3]
In the FFU from 16.2 to 17.1, the sections related to the CEPH upgrade should be updated to support also the case where the FFU is applied to a disconnected OpenStack environment (no internet access granted).
[1] 14.1 Chapter 14. Upgrading Red Hat Ceph Storage 6 to 7
https://docs.redhat.com/en/documentation/red_hat_openstack_platform/17.1/html-single/framework_for_upgrades_16.2_to_17.1/index#assembly_ceph-6-to-7_upgrade_post-upgrade-external-ceph
[2]
Grafana is trying to navigate on internet to check version and available plugins:
logger=grafana.update.checker t=2025-01-31T16:40:13.526395413Z level=error msg="Update check failed" error="failed to get stable version from grafana.com: Get \"https://grafana.com/api/grafana/versions/stable\": dial tcp 34.120.177.193:443: i/o timeout" duration=10.001814247s logger=plugin.signature.key_retriever t=2025-01-31T16:40:13.550870805Z level=error msg="Error downloading plugin manifest keys" error="Get \"https://grafana.com/api/plugins/ci/keys\": dial tcp 34.120.177.193:443: i/o timeout (Client.Timeout exceeded while awaiting headers)" logger=infra.usagestats t=2025-01-31T16:41:21.527086398Z level=info msg="Usage stats are ready to report" logger=plugin.signature.key_retriever t=2025-01-31T16:41:23.553379671Z level=error msg="Error downloading plugin manifest keys" error="Get \"https://grafana.com/api/plugins/ci/keys\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
[3] # See only the solution below associated with the first problem (see Case 04033472 - Comment: #137)
From what I understand we have two issues:
- Grafana has timeout issues because we are in a disconnected environment and grafana wants to download updates
- Grafana certificate issues
For the first issue, let's create a new template file, in this example I create it in ~/grafana.ini.j2 with the following content: ``` # {{ cephadm_managed }} [users] default_theme = light {% if anonymous_access %} [auth.anonymous] enabled = true org_name = 'Main Org.' org_role = 'Viewer' {% endif %} [server] domain = 'bootstrap.storage.lab' protocol = {{ protocol }} cert_file = /etc/grafana/certs/cert_file cert_key = /etc/grafana/certs/cert_key http_port = {{ http_port }} http_addr = {{ http_addr }} [snapshots] external_enabled = false [security] {% if not initial_admin_password %} disable_initial_admin_creation = true {% else %} admin_user = admin admin_password = {{ initial_admin_password }} {% endif %} cookie_secure = true cookie_samesite = none allow_embedding = true [analytics] check_for_updates = false reporting_enabled = false [plugins] check_for_plugin_updates = false public_key_retrieval_disabled = true ``` Then let's set this new template for grafana: ``` # [root@mgmt-0 ~]# ls grafana* grafana.ini.j2 # ceph config-key set mgr/cephadm/services/grafana/grafana.ini -i $PWD/grafana.ini.j2 set mgr/cephadm/services/grafana/grafana.ini ``` Now reconfigure grafana: ``` [root@mgmt-0 ~]# ceph orch reconfig grafana Scheduled to reconfig grafana.mgmt-0 on host 'mgmt-0.rsachere7.lab.upshift.rdu2.redhat.com' ``` And a couple of minutes later we should see the new configuration with the analytics and plugin options: ``` [root@mgmt-0 ~]# cat /var/lib/ceph/e1c31fe0-d192-11ef-a7ef-fa163eef56d6/grafana.mgmt-0/etc/grafana/grafana.ini # This file is generated by cephadm. [users] default_theme = light [auth.anonymous] enabled = true org_name = 'Main Org.' org_role = 'Viewer' [server] domain = 'bootstrap.storage.lab' protocol = https cert_file = /etc/grafana/certs/cert_file cert_key = /etc/grafana/certs/cert_key http_port = 3000 http_addr = [snapshots] external_enabled = false [security] disable_initial_admin_creation = true cookie_secure = true cookie_samesite = none allow_embedding = true [analytics] check_for_updates = false reporting_enabled = false [plugins] check_for_plugin_updates = false public_key_retrieval_disabled = true
