Description of problem (please be detailed as possible and provide log
snippests):
Across several deployment types we are not able to get ceph_health_status Ok when the cluster workload is idle.
This happens with "Multi storage cluster", with "Compact mode and In Transit encryption"
Issue happens even though the Ceph health status is OK, when running ceph command
[2024-05-27T08:15:40.290Z] 04:15:39 - MainThread - tests.conftest - [32mINFO[0m - Checking for Ceph Health OK

Version of all relevant components (if applicable):
ODF v4.16.0-108
OCP 4.16.0-0.nightly-2024-05-23-173505

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
yes

Is there any workaround available to the best of your knowledge?
no

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
1

Can this issue reproducible?
4 failures observed

Can this issue reproduce from the UI?
yes

If this is a regression, please provide more details to justify this:
new types of deployment

Steps to Reproduce:
1.
2.
3.

Actual results:
no ceph health OK metrics in ranged queries

Expected results:
ceph health OK metrics are available and positive when the ceph health is Ok

Additional info:

must gather logs http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/j-023vii1cs30-t1/j-023vii1cs30-t1_20240526T234857/logs/testcases_1716787108/j-023vii1cs30-t1/

2024-05-27 04:32:00,399 - MainThread - INFO - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.query_range.549 - Performing prometheus range query 'ceph_health_status' over a time range (1716797812.3947108, 1716798719.0427072)
2024-05-27 04:32:00,399 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.get.428 - GET https://prometheus-k8s-openshift-monitoring.apps.j-023vii1cs30-t1.qe.rh-ocs.com/api/v1/query_range
2024-05-27 04:32:00,399 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.get.429 - headers=

{'Authorization': 'Bearer sha256~rBfBZ2WDHco6kELkjFgUImkeOH7C4dZ0WuTyT3yqr_U'}

2024-05-27 04:32:00,399 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.get.430 - verify=False
2024-05-27 04:32:00,400 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.get.431 - params=

{'query': 'ceph_health_status', 'start': 1716797812.3947108, 'end': 1716798719.0427072, 'step': 15}

2024-05-27 04:32:00,405 - MainThread - DEBUG - urllib3.connectionpool._new_conn.1019 - Starting new HTTPS connection (1): prometheus-k8s-openshift-monitoring.apps.j-023vii1cs30-t1.qe.rh-ocs.com:443
2024-05-27 04:32:00,457 - MainThread - DEBUG - urllib3.connectionpool._make_request.474 - https://prometheus-k8s-openshift-monitoring.apps.j-023vii1cs30-t1.qe.rh-ocs.com:443 "GET /api/v1/query_range?query=ceph_health_status&start=1716797812.3947108&end=1716798719.0427072&step=15 HTTP/1.1" 200 440
2024-05-27 04:32:00,495 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.validate_status.304 - content value: {'status': 'success', 'data': {'resultType': 'matrix', 'result': [{'metric':

{'__name__': 'ceph_health_status', 'container': 'mgr', 'endpoint': 'http-metrics', 'instance': '10.129.0.102:9283', 'job': 'rook-ceph-mgr', 'managedBy': 'ocs-storagecluster', 'namespace': 'openshift-storage', 'pod': 'rook-ceph-mgr-a-697476d944-wvtm7', 'service': 'rook-ceph-mgr'}

, 'values': [[1716797812.395, '0'], [1716797827.395, '0'], [1716797842.395, '0'], [1716797857.395, '0'], [1716797872.395, '1'], [1716797887.395, '1'], [1716797902.395, '0'], [1716797917.395, '0'], [1716797932.395, '0'], [1716797947.395, '0'], [1716797962.395, '1'], [1716797977.395, '1'], [1716797992.395, '0'], [1716798007.395, '0'], [1716798022.395, '0'], [1716798037.395, '0'], [1716798052.395, '0'], [1716798067.395, '0'], [1716798082.395, '0'], [1716798097.395, '0'], [1716798112.395, '0'], [1716798127.395, '0'], [1716798142.395, '0'], [1716798157.395, '0'], [1716798172.395, '0'], [1716798187.395, '0'], [1716798202.395, '0'], [1716798217.395, '0'], [1716798232.395, '0'], [1716798247.395, '0'], [1716798262.395, '0'], [1716798277.395, '0'], [1716798292.395, '0'], [1716798307.395, '0'], [1716798322.395, '0'], [1716798337.395, '0'], [1716798352.395, '0'], [1716798367.395, '0'], [1716798382.395, '0'], [1716798397.395, '0'], [1716798412.395, '0'], [1716798427.395, '0'], [1716798442.395, '0'], [1716798457.395, '0'], [1716798472.395, '0'], [1716798487.395, '0'], [1716798502.395, '0'], [1716798517.395, '0'], [1716798532.395, '0'], [1716798547.395, '0'], [1716798562.395, '0'], [1716798577.395, '0'], [1716798592.395, '0'], [1716798607.395, '0'], [1716798622.395, '0'], [1716798637.395, '0'], [1716798652.395, '0'], [1716798667.395, '0'], [1716798682.395, '0'], [1716798697.395, '0'], [1716798712.395, '0']]}]}}
2024-05-27 04:32:00,496 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.query_range.594 - there are no holes in the data
2024-05-27 04:32:00,499 - MainThread - INFO - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.117 - Validating a result of a range query
2024-05-27 04:32:00,500 - MainThread - INFO - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.136 - checking metric

{'__name__': 'ceph_health_status', 'container': 'mgr', 'endpoint': 'http-metrics', 'instance': '10.129.0.102:9283', 'job': 'rook-ceph-mgr', 'managedBy': 'ocs-storagecluster', 'namespace': 'openshift-storage', 'pod': 'rook-ceph-mgr-a-697476d944-wvtm7', 'service': 'rook-ceph-mgr'}

2024-05-27 04:32:00,500 - MainThread - INFO - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.140 - metrics for ceph_health_status starts at 2024-05-27 08:16:52.395000
2024-05-27 04:32:00,500 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:16:52.395000
2024-05-27 04:32:00,500 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:17:07.395000
2024-05-27 04:32:00,500 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:17:22.395000
2024-05-27 04:32:00,500 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:17:37.395000
2024-05-27 04:32:00,501 - MainThread - ERROR - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.158 - ceph_health_status has bad value 1 at 2024-05-27 08:17:52.395000
2024-05-27 04:32:00,501 - MainThread - ERROR - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.158 - ceph_health_status has bad value 1 at 2024-05-27 08:18:07.395000
2024-05-27 04:32:00,501 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:18:22.395000
2024-05-27 04:32:00,501 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:18:37.395000
2024-05-27 04:32:00,501 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:18:52.395000
2024-05-27 04:32:00,501 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:19:07.395000
2024-05-27 04:32:00,501 - MainThread - ERROR - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.158 - ceph_health_status has bad value 1 at 2024-05-27 08:19:22.395000
2024-05-27 04:32:00,502 - MainThread - ERROR - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.158 - ceph_health_status has bad value 1 at 2024-05-27 08:19:37.395000
2024-05-27 04:32:00,502 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:19:52.395000
2024-05-27 04:32:00,502 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:20:07.395000
2024-05-27 04:32:00,502 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:20:22.395000
2024-05-27 04:32:00,502 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:20:37.395000
2024-05-27 04:32:00,502 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:20:52.395000
2024-05-27 04:32:00,503 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:21:07.395000
2024-05-27 04:32:00,503 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:21:22.395000
2024-05-27 04:32:00,503 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:21:37.395000
2024-05-27 04:32:00,503 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:21:52.395000
2024-05-27 04:32:00,503 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:22:07.395000
2024-05-27 04:32:00,503 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:22:22.395000
2024-05-27 04:32:00,503 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:22:37.395000
2024-05-27 04:32:00,504 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:22:52.395000
2024-05-27 04:32:00,504 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:23:07.395000
2024-05-27 04:32:00,504 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:23:22.395000
2024-05-27 04:32:00,504 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:23:37.395000
2024-05-27 04:32:00,504 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:23:52.395000
2024-05-27 04:32:00,504 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:24:07.395000
2024-05-27 04:32:00,504 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:24:22.395000
2024-05-27 04:32:00,504 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:24:37.395000
2024-05-27 04:32:00,505 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:24:52.395000
2024-05-27 04:32:00,505 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:25:07.395000
2024-05-27 04:32:00,505 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:25:22.395000
2024-05-27 04:32:00,505 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:25:37.395000
2024-05-27 04:32:00,505 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:25:52.395000
2024-05-27 04:32:00,505 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:26:07.395000
2024-05-27 04:32:00,505 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:26:22.395000
2024-05-27 04:32:00,506 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:26:37.395000
2024-05-27 04:32:00,506 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:26:52.395000
2024-05-27 04:32:00,506 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:27:07.395000
2024-05-27 04:32:00,506 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:27:22.395000
2024-05-27 04:32:00,506 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:27:37.395000
2024-05-27 04:32:00,506 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:27:52.395000
2024-05-27 04:32:00,506 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:28:07.395000
2024-05-27 04:32:00,506 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:28:22.395000
2024-05-27 04:32:00,507 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:28:37.395000
2024-05-27 04:32:00,507 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:28:52.395000
2024-05-27 04:32:00,507 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:29:07.395000
2024-05-27 04:32:00,507 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:29:22.395000
2024-05-27 04:32:00,507 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:29:37.395000
2024-05-27 04:32:00,507 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:29:52.395000
2024-05-27 04:32:00,507 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:30:07.395000
2024-05-27 04:32:00,508 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:30:22.395000
2024-05-27 04:32:00,508 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:30:37.395000
2024-05-27 04:32:00,508 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:30:52.395000
2024-05-27 04:32:00,508 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:31:07.395000
2024-05-27 04:32:00,508 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:31:22.395000
2024-05-27 04:32:00,508 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:31:37.395000
2024-05-27 04:32:00,508 - MainThread - DEBUG - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.148 - ceph_health_status has good value 0 at 2024-05-27 08:31:52.395000
2024-05-27 04:32:00,509 - MainThread - INFO - /home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/prometheus.py.check_query_range_result_viafunction.172 - No invalid values detected
2024-05-27 04:32:00,509 - MainThread - ERROR - tests.functional.monitoring.prometheus.metrics.test_monitoring_defaults.test_monitoring_reporting_ok_when_idle.210 - ceph_health_status should report 0 (health ok) as expected

https://reportportal-ocs4.apps.ocp-c1.prod.psi.redhat.com/ui/#ocs/launches/678/21547/1024366/1024404/log?logParams=history%3D1024404%26page.page%3D1