+++ This bug was initially created as a clone of Bug #2151693 +++

Description of problem:For EUS->EUS upgrade we are supposed to turn off hco.spec.workloadUpdateStrategy, so that workload updates can only happen after upgrading to 4.12. However, that is currently not happening. Post upgrade to 4.11.0, I see hco.spec.workloadUpdateStrategy set to be LiveMigrate and workloads are live migrating.

Version-Release number of selected component (if applicable):
4.11.0

How reproducible:
100%

Steps to Reproduce:
1. Before upgrade set hco.spec.workloadUpdateStrategy to []
2. After upgrade check that hco.spec.workloadUpdateStrategy is set to LiveMigrate
3.

Actual results:
Before upgrade:
==============
(cnv-tests) [cnv-qe-jenkins@cnv-qe-01 cnv-tests]$ kubectl get hco kubevirt-hyperconverged -n openshift-cnv -o json | jq ".status.versions"
[

{ "name": "operator", "version": "4.10.7" }

]
(cnv-tests) [cnv-qe-jenkins@cnv-qe-01 cnv-tests]$
(cnv-tests) [cnv-qe-jenkins@cnv-qe-01 cnv-tests]$ kubectl get vm -A
NAMESPACE NAME AGE STATUS READY
test-upgrade-namespace vm-for-product-upgrade-hos-1670437886-760028 15m Running True
test-upgrade-namespace vm-for-product-upgrade-hos-1670437887-3594792 14m Running True
test-upgrade-namespace vm-for-product-upgrade-nfs-1670437884-7562118 15m Running True
test-upgrade-namespace vm-for-product-upgrade-ocs-1670437885-5164044 15m Running True
(cnv-tests) [cnv-qe-jenkins@cnv-qe-01 cnv-tests]$
(cnv-tests) [cnv-qe-jenkins@cnv-qe-01 cnv-tests]$ kubectl get hco kubevirt-hyperconverged -n openshift-cnv -o json | jq ".spec.workloadUpdateStrategy"
{
"batchEvictionInterval": "1m0s",
"batchEvictionSize": 10,
"workloadUpdateMethods": []
}
(cnv-tests) [cnv-qe-jenkins@cnv-qe-01 cnv-tests]$
(cnv-tests) [cnv-qe-jenkins@cnv-qe-01 cnv-tests]$ kubectl get hco kubevirt-hyperconverged -n openshift-cnv -o json | jq ".status.conditions"
[

{ "lastTransitionTime": "2022-12-06T18:37:49Z", "message": "Reconcile completed successfully", "observedGeneration": 3, "reason": "ReconcileCompleted", "status": "True", "type": "ReconcileComplete" }

,

{ "lastTransitionTime": "2022-12-07T18:51:50Z", "message": "Reconcile completed successfully", "observedGeneration": 3, "reason": "ReconcileCompleted", "status": "True", "type": "Available" }

,

{ "lastTransitionTime": "2022-12-07T18:51:50Z", "message": "Reconcile completed successfully", "observedGeneration": 3, "reason": "ReconcileCompleted", "status": "False", "type": "Progressing" }

,

{ "lastTransitionTime": "2022-12-07T18:51:02Z", "message": "Reconcile completed successfully", "observedGeneration": 3, "reason": "ReconcileCompleted", "status": "False", "type": "Degraded" }

,

{ "lastTransitionTime": "2022-12-07T18:51:50Z", "message": "Reconcile completed successfully", "observedGeneration": 3, "reason": "ReconcileCompleted", "status": "True", "type": "Upgradeable" }

]
(cnv-tests) [cnv-qe-jenkins@cnv-qe-01 cnv-tests]$ kubectl get hco kubevirt-hyperconverged -n openshift-cnv -o json | jq ".spec.workloadUpdateStrategy"
{
"batchEvictionInterval": "1m0s",
"batchEvictionSize": 10,
"workloadUpdateMethods": []
}
(cnv-tests) [cnv-qe-jenkins@cnv-qe-01 cnv-tests]$
Post ocp upgrade:
(cnv-tests) [cnv-qe-jenkins@cnv-qe-01 cnv-tests]$ kubectl get catalogsource -n openshift-marketplace
NAME DISPLAY TYPE PUBLISHER AGE
certified-operators Certified Operators grpc Red Hat 26h
community-operators Community Operators grpc Red Hat 26h
hco-catalogsource OpenShift Virtualization Index Image grpc Red Hat 25h
ocs-catalogsource OpenShift Container Storage grpc Red Hat 25h
redhat-marketplace Red Hat Marketplace grpc Red Hat 26h
redhat-operators Red Hat Operators grpc Red Hat 26h
(cnv-tests) [cnv-qe-jenkins@cnv-qe-01 cnv-tests]$ kubectl patch operatorhub cluster --type merge -p '{"spec": {"disableAllDefaultSources": true}}'
operatorhub.config.openshift.io/cluster patched
(cnv-tests) [cnv-qe-jenkins@cnv-qe-01 cnv-tests]$ kubectl get catalogsource -n openshift-marketplace
NAME DISPLAY TYPE PUBLISHER AGE
hco-catalogsource OpenShift Virtualization Index Image grpc Red Hat 25h
ocs-catalogsource OpenShift Container Storage grpc Red Hat 25h
(cnv-tests) [cnv-qe-jenkins@cnv-qe-01 cnv-tests]$
(cnv-tests) [cnv-qe-jenkins@cnv-qe-01 cnv-tests]$ kubectl get ip -A
NAMESPACE NAME CSV APPROVAL APPROVED
openshift-cnv install-lqt5n kubevirt-hyperconverged-operator.v4.11.0 Manual false
openshift-cnv install-njk24 kubevirt-hyperconverged-operator.v4.10.7 Manual true
openshift-local-storage install-s8jm4 local-storage-operator.4.10.0-202211041323 Automatic true
openshift-local-storage install-v86lx local-storage-operator.4.11.0-202211072116 Automatic true
openshift-nfd install-t95wm nfd.4.10.0-202211041323 Automatic true
openshift-nfd install-xcqkn nfd.4.11.0-202211091549 Automatic true
openshift-operators install-4xbvx servicemeshoperator.v2.3.0 Automatic true
openshift-operators install-lkc4w jaeger-operator.v1.39.0-3 Automatic true
openshift-operators install-pj9kw jaeger-operator.v1.39.0-3 Automatic true
openshift-sriov-network-operator install-42c9t sriov-network-operator.4.10.0-202211180226 Automatic true
openshift-sriov-network-operator install-lw5p9 sriov-network-operator.4.11.0-202211211407 Automatic true
openshift-storage install-s9qfr ocs-operator.v4.10.9 Automatic true
(cnv-tests) [cnv-qe-jenkins@cnv-qe-01 cnv-tests]$ kubectl patch installplan install-lqt5n --namespace='openshift-cnv' --type='merge' --patch='{"spec":{"approved":true}}'
installplan.operators.coreos.com/install-lqt5n patched
(cnv-tests) [cnv-qe-jenkins@cnv-qe-01 cnv-tests]$ kubectl get csv -n openshift-cnv
NAME DISPLAY VERSION REPLACES PHASE
jaeger-operator.v1.39.0-3 Red Hat OpenShift distributed tracing platform 1.39.0-3 jaeger-operator.v1.34.1-5 Succeeded
kiali-operator.v1.57.3 Kiali Operator 1.57.3 kiali-operator.v1.48.3 Succeeded
kubevirt-hyperconverged-operator.v4.10.7 OpenShift Virtualization 4.10.7 kubevirt-hyperconverged-operator.v4.10.6 Replacing
kubevirt-hyperconverged-operator.v4.11.0 OpenShift Virtualization 4.11.0 kubevirt-hyperconverged-operator.v4.10.7 Installing
servicemeshoperator.v2.3.0 Red Hat OpenShift Service Mesh 2.3.0-0 servicemeshoperator.v2.2.3 Succeeded
(cnv-tests) [cnv-qe-jenkins@cnv-qe-01 cnv-tests]$ kubectl get csv -n openshift-cnv
NAME DISPLAY VERSION REPLACES PHASE
jaeger-operator.v1.39.0-3 Red Hat OpenShift distributed tracing platform 1.39.0-3 jaeger-operator.v1.34.1-5 Succeeded
kiali-operator.v1.57.3 Kiali Operator 1.57.3 kiali-operator.v1.48.3 Succeeded
kubevirt-hyperconverged-operator.v4.11.0 OpenShift Virtualization 4.11.0 kubevirt-hyperconverged-operator.v4.10.7 Succeeded
servicemeshoperator.v2.3.0 Red Hat OpenShift Service Mesh 2.3.0-0 servicemeshoperator.v2.2.3 Succeeded
(cnv-tests) [cnv-qe-jenkins@cnv-qe-01 cnv-tests]$ kubectl get hco kubevirt-hyperconverged -n openshift-cnv -o json | jq ".status.versions"
[

{ "name": "operator", "version": "4.11.0" }

]
(cnv-tests) [cnv-qe-jenkins@cnv-qe-01 cnv-tests]$ kubectl get hco kubevirt-hyperconverged -n openshift-cnv -o json | jq ".spec.workloadUpdateStrategy"
{
"batchEvictionInterval": "1m0s",
"batchEvictionSize": 10,
"workloadUpdateMethods": [
"LiveMigrate"
]
}
(cnv-tests) [cnv-qe-jenkins@cnv-qe-01 cnv-tests]$
Expected results:
hco.spec.workloadUpdateStrategy should stay unaltered.

Additional info:
This impacts EUS->EUS upgrade. Without ability to pause workload updates till CNV is updated to 4.12, we can't achive EUS->EUS upgradability

— Additional comment from Simone Tiraboschi on 2022-12-09 14:00:08 UTC —

This happens only with: workloadUpdateMethods: [],
setting something like workloadUpdateMethods: ["None"] sounds like a valid workaround.

— Additional comment from Red Hat Bugzilla on 2022-12-15 08:28:44 UTC —

Account disabled by LDAP Audit for extended failure

— Additional comment from Simone Tiraboschi on 2022-12-15 16:36:33 UTC —

Adding "olm.skipRange: '>=4.10.7 <4.11.0'" on 4.11.2 so that users coming from 4.10.7 and greater are not going to pass into 4.11.0 and 4.11.1 hitting this.

— Additional comment from Debarati Basu-Nag on 2022-12-20 17:23:09 UTC —

Validated with 4.11.2-19.
============================
cnv-qe-jenkins@cnv-qe-infra-01:~/dbasunag$ kubectl get hco kubevirt-hyperconverged -n openshift-cnv -o json | jq ".spec.workloadUpdateStrategy"
{
"batchEvictionInterval": "1m0s",
"batchEvictionSize": 10,
"workloadUpdateMethods": []
}
cnv-qe-jenkins@cnv-qe-infra-01:~/dbasunag$ kubectl get hco kubevirt-hyperconverged -n openshift-cnv -o json | jq ".status.versions"
[

{ "name": "operator", "version": "4.11.2" }

]
cnv-qe-jenkins@cnv-qe-infra-01:~/dbasunag$ kubectl get csv -n openshift-cnv
NAME DISPLAY VERSION REPLACES PHASE
jaeger-operator.v1.39.0-3 Red Hat OpenShift distributed tracing platform 1.39.0-3 jaeger-operator.v1.34.1-5 Succeeded
kiali-operator.v1.57.3 Kiali Operator 1.57.3 kiali-operator.v1.48.3 Succeeded
kubevirt-hyperconverged-operator.v4.11.2 OpenShift Virtualization 4.11.2 kubevirt-hyperconverged-operator.v4.10.7 Succeeded
servicemeshoperator.v2.3.0 Red Hat OpenShift Service Mesh 2.3.0-0 servicemeshoperator.v2.2.3 Succeeded
cnv-qe-jenkins@cnv-qe-infra-01:~/dbasunag$

— Additional comment from errata-xmlrpc on 2022-12-20 17:23:44 UTC —

This bug has been added to advisory RHSA-2022:95482 by Continuous Infra Team Autobot (contra/pipeline@REDHAT.COM)

— Additional comment from errata-xmlrpc on 2023-01-10 13:58:32 UTC —

This bug has been dropped from advisory RHSA-2022:95482 by Ying Cui (ycui@redhat.com)

— Additional comment from errata-xmlrpc on 2023-01-10 13:59:45 UTC —

This bug has been added to advisory RHEA-2022:106345 by Ying Cui (ycui@redhat.com)

— Additional comment from Ying Cui on 2023-01-10 14:04:28 UTC —

(In reply to errata-xmlrpc from comment #6)
> This bug has been dropped from advisory RHSA-2022:95482 by Ying Cui
> (ycui@redhat.com)

Note: This bug is targeted to 4.11.2 release, but it was attached to CNV 4.12.0 errata RHSA-2022:95482 automatically by CI.

You can see in the bugzilla bug Links table that there are several MRs all mentioning the same bug ID. Where 2 MRs are for 4.11, 1 for 4.12 and 1 for 4.13.

This situation is creating a race condition between the versions that the bug will be attached to by CI automatically.

The correct way to do this is to create a dedicated bug for each version for backporting, and that way each bug will have its own dedicated MRs on a specific branch/version and this issue won't happen again.

The only workaround is to manually remove the bug from the advisory.

— Additional comment from errata-xmlrpc on 2023-01-12 12:02:42 UTC —

Advisory RHEA-2022:106345 has moved into REL_PREP

— Additional comment from errata-xmlrpc on 2023-01-12 13:14:41 UTC —

Bug report changed to RELEASE_PENDING status by Errata System.
Advisory RHEA-2022:106345-04 has been changed to PUSH_READY status.
https://errata.devel.redhat.com/advisory/106345

— Additional comment from errata-xmlrpc on 2023-01-12 13:14:50 UTC —

Advisory RHEA-2022:106345 expected to be published on 2023-01-12

— Additional comment from errata-xmlrpc on 2023-01-12 13:14:56 UTC —

Advisory RHEA-2022:106345 has moved from REL_PREP to PUSH_READY

— Additional comment from errata-xmlrpc on 2023-01-12 14:08:54 UTC —

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Virtualization 4.11.2 Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2023:0155