Description of problem:

during Upgrading CNV from v4.11.6-20 to v4.12.6-17
ocp version 4.12.30-x86_64
KubevirtHyperconvergedClusterOperatorCRModification fired

Version-Release number of selected component (if applicable):

How reproducible: 100%
run cnv upgrade from 4.11.z > 4.12.z

Steps to Reproduce:

Actual results:
AssertionError: Following alerts were fired during upgrade: [{'labels':

{'alertname': 'KubevirtHyperconvergedClusterOperatorCRModification', 'component_name': 'service/kubevirt-hyperconverged-operator-metrics', 'kubernetes_operator_component': 'hyperconverged-cluster-operator', 'kubernetes_operator_part_of': 'kubevirt', 'severity': 'warning'}

, 'annotations':

{'description': 'Out-of-band modification for service/kubevirt-hyperconverged-operator-metrics.', 'runbook_url': 'https://kubevirt.io/monitoring/runbooks/KubevirtHyperconvergedClusterOperatorCRModification', 'summary': '1 out-of-band CR modifications were detected in the last 10 minutes.'}

, 'state': 'firing', 'activeAt': '2023-08-28T19:36:00.184767418Z', 'value': '1e+00'}, {'labels':

{'alertname': 'KubevirtHyperconvergedClusterOperatorCRModification', 'component_name': 'servicemonitor/kubevirt-hyperconverged-operator-metrics', 'kubernetes_operator_component': 'hyperconverged-cluster-operator', 'kubernetes_operator_part_of': 'kubevirt', 'severity': 'warning'}

, 'annotations':

{'description': 'Out-of-band modification for servicemonitor/kubevirt-hyperconverged-operator-metrics.', 'runbook_url': 'https://kubevirt.io/monitoring/runbooks/KubevirtHyperconvergedClusterOperatorCRModification', 'summary': '1 out-of-band CR modifications were detected in the last 10 minutes.'}

, 'state': 'firing', 'activeAt': '2023-08-28T19:36:00.184767418Z', 'value': '1e+00'}, {'labels':

{'alertname': 'KubevirtHyperconvergedClusterOperatorCRModification', 'component_name': 'prometheusrule/kubevirt-hyperconverged-prometheus-rule', 'kubernetes_operator_component': 'hyperconverged-cluster-operator', 'kubernetes_operator_part_of': 'kubevirt', 'severity': 'warning'}

, 'annotations':

{'description': 'Out-of-band modification for prometheusrule/kubevirt-hyperconverged-prometheus-rule.', 'runbook_url': 'https://kubevirt.io/monitoring/runbooks/KubevirtHyperconvergedClusterOperatorCRModification', 'summary': '1 out-of-band CR modifications were detected in the last 10 minutes.'}

, 'state': 'firing', 'activeAt': '2023-08-28T19:36:00.184767418Z', 'value': '1e+00'}, {'labels':

{'alertname': 'KubevirtHyperconvergedClusterOperatorCRModification', 'component_name': 'role/hyperconverged-cluster-operator-metrics', 'kubernetes_operator_component': 'hyperconverged-cluster-operator', 'kubernetes_operator_part_of': 'kubevirt', 'severity': 'warning'}

, 'annotations':

{'description': 'Out-of-band modification for role/hyperconverged-cluster-operator-metrics.', 'runbook_url': 'https://kubevirt.io/monitoring/runbooks/KubevirtHyperconvergedClusterOperatorCRModification', 'summary': '1 out-of-band CR modifications were detected in the last 10 minutes.'}

, 'state': 'firing', 'activeAt': '2023-08-28T19:36:00.184767418Z', 'value': '1e+00'}, {'labels':

{'alertname': 'KubevirtHyperconvergedClusterOperatorCRModification', 'component_name': 'rolebinding/hyperconverged-cluster-operator-metrics', 'kubernetes_operator_component': 'hyperconverged-cluster-operator', 'kubernetes_operator_part_of': 'kubevirt', 'severity': 'warning'}

, 'annotations':

{'description': 'Out-of-band modification for rolebinding/hyperconverged-cluster-operator-metrics.', 'runbook_url': 'https://kubevirt.io/monitoring/runbooks/KubevirtHyperconvergedClusterOperatorCRModification', 'summary': '1 out-of-band CR modifications were detected in the last 10 minutes.'}

, 'state': 'firing', 'activeAt': '2023-08-28T19:36:00.184767418Z', 'value': '1e+00'}]
assert not [{'activeAt': '2023-08-28T19:36:00.184767418Z', 'annotations':

{'description': 'Out-of-band modification for service/k...omponent': 'hyperconverged-cluster-operator', 'kubernetes_operator_part_of': 'kubevirt', ...}

, 'state': 'firing', ...}]

Expected results:
no alerts should be fired during upgrade process

Additional info: