Uploaded image for project: 'OpenShift Virtualization'
  1. OpenShift Virtualization
  2. CNV-31563

[2227785] KubevirtHyperconvergedClusterOperatorCRModification alert in firing state during cnv upgrade (4.12.4->4.12.5)

XMLWordPrintable

    • CNV I/U Operators Sprint 243
    • No

      Description of problem:
      During cnv upgrade v4.12.4 to 4.12.5-50 (OCP: 4.12.26), noticed KubevirtHyperconvergedClusterOperatorCRModification alert in firing state.

      Version-Release number of selected component (if applicable):
      cnv: 4.12.4 to 4.12.5-50 (OCP: 4.12.26)

      How reproducible:
      1 out of 1 attempt

      Steps to Reproduce:
      1. Check alerts fired during cnv upgrade
      2.
      3.

      Actual results:
      {'labels':

      {'alertname': 'KubevirtHyperconvergedClusterOperatorCRModification', 'component_name': 'role/hyperconverged-cluster-operator-metrics', 'kubernetes_operator_component': 'hyperconverged-cluster-operator', 'kubernetes_operator_part_of': 'kubevirt', 'severity': 'warning'}

      , 'annotations':

      {'description': 'Out-of-band modification for role/hyperconverged-cluster-operator-metrics.', 'runbook_url': 'https://kubevirt.io/monitoring/runbooks/KubevirtHyperconvergedClusterOperatorCRModification', 'summary': '1 out-of-band CR modifications were detected in the last 10 minutes.'}

      , 'state': 'firing', 'activeAt': '2023-07-26T20:47:55.920372176Z', 'value': '1e+00'}, {'labels':

      {'alertname': 'KubevirtHyperconvergedClusterOperatorCRModification', 'component_name': 'servicemonitor/kubevirt-hyperconverged-operator-metrics', 'kubernetes_operator_component': 'hyperconverged-cluster-operator', 'kubernetes_operator_part_of': 'kubevirt', 'severity': 'warning'}

      , 'annotations':

      {'description': 'Out-of-band modification for servicemonitor/kubevirt-hyperconverged-operator-metrics.', 'runbook_url': 'https://kubevirt.io/monitoring/runbooks/KubevirtHyperconvergedClusterOperatorCRModification', 'summary': '1 out-of-band CR modifications were detected in the last 10 minutes.'}

      , 'state': 'firing', 'activeAt': '2023-07-26T20:48:55.920372176Z', 'value': '1e+00'}, {'labels':

      {'alertname': 'KubevirtHyperconvergedClusterOperatorCRModification', 'component_name': 'rolebinding/hyperconverged-cluster-operator-metrics', 'kubernetes_operator_component': 'hyperconverged-cluster-operator', 'kubernetes_operator_part_of': 'kubevirt', 'severity': 'warning'}

      , 'annotations':

      {'description': 'Out-of-band modification for rolebinding/hyperconverged-cluster-operator-metrics.', 'runbook_url': 'https://kubevirt.io/monitoring/runbooks/KubevirtHyperconvergedClusterOperatorCRModification', 'summary': '1 out-of-band CR modifications were detected in the last 10 minutes.'}

      , 'state': 'firing', 'activeAt': '2023-07-26T20:48:55.920372176Z', 'value': '1e+00'}, {'labels':

      {'alertname': 'KubevirtHyperconvergedClusterOperatorCRModification', 'component_name': 'service/kubevirt-hyperconverged-operator-metrics', 'kubernetes_operator_component': 'hyperconverged-cluster-operator', 'kubernetes_operator_part_of': 'kubevirt', 'severity': 'warning'}

      , 'annotations':

      {'description': 'Out-of-band modification for service/kubevirt-hyperconverged-operator-metrics.', 'runbook_url': 'https://kubevirt.io/monitoring/runbooks/KubevirtHyperconvergedClusterOperatorCRModification', 'summary': '1 out-of-band CR modifications were detected in the last 10 minutes.'}

      , 'state': 'firing', 'activeAt': '2023-07-26T20:48:55.920372176Z', 'value': '1e+00'}]
      assert not [{'activeAt': '2023-07-26T20:49:22.832759694Z', 'annotations':

      {'runbook_url': 'https://kubevirt.io/monitoring/runbook...omponent': 'hyperconverged-cluster-operator', 'kubernetes_operator_part_of': 'kubevirt', ...}

      , 'state': 'firing', ...}]

      Expected results:
      we are trying to capture all the alerts that are fired during upgrades and reduce the noise generated.

      Additional info:

      http://pastebin.test.redhat.com/1106300

      must-gather.log
      http://pastebin.test.redhat.com/1106301

              stirabos Simone Tiraboschi
              rh-ee-ahafe Ahmad Hafi
              Ahmad Hafi Ahmad Hafi
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: