-
Bug
-
Resolution: Done-Errata
-
Major
-
None
-
False
-
-
False
-
CLOSED
-
---
-
---
-
-
-
CNV I/U Operators Sprint 229
-
High
-
No
Description of problem:For EUS->EUS upgrade we are supposed to turn off hco.spec.workloadUpdateStrategy, so that workload updates can only happen after upgrading to 4.12. However, that is currently not happening. Post upgrade to 4.11.0, I see hco.spec.workloadUpdateStrategy set to be LiveMigrate and workloads are live migrating.
Version-Release number of selected component (if applicable):
4.11.0
How reproducible:
100%
Steps to Reproduce:
1. Before upgrade set hco.spec.workloadUpdateStrategy to []
2. After upgrade check that hco.spec.workloadUpdateStrategy is set to LiveMigrate
3.
Actual results:
Before upgrade:
==============
(cnv-tests) [cnv-qe-jenkins@cnv-qe-01 cnv-tests]$ kubectl get hco kubevirt-hyperconverged -n openshift-cnv -o json | jq ".status.versions"
[
]
(cnv-tests) [cnv-qe-jenkins@cnv-qe-01 cnv-tests]$
(cnv-tests) [cnv-qe-jenkins@cnv-qe-01 cnv-tests]$ kubectl get vm -A
NAMESPACE NAME AGE STATUS READY
test-upgrade-namespace vm-for-product-upgrade-hos-1670437886-760028 15m Running True
test-upgrade-namespace vm-for-product-upgrade-hos-1670437887-3594792 14m Running True
test-upgrade-namespace vm-for-product-upgrade-nfs-1670437884-7562118 15m Running True
test-upgrade-namespace vm-for-product-upgrade-ocs-1670437885-5164044 15m Running True
(cnv-tests) [cnv-qe-jenkins@cnv-qe-01 cnv-tests]$
(cnv-tests) [cnv-qe-jenkins@cnv-qe-01 cnv-tests]$ kubectl get hco kubevirt-hyperconverged -n openshift-cnv -o json | jq ".spec.workloadUpdateStrategy"
{
"batchEvictionInterval": "1m0s",
"batchEvictionSize": 10,
"workloadUpdateMethods": []
}
(cnv-tests) [cnv-qe-jenkins@cnv-qe-01 cnv-tests]$
(cnv-tests) [cnv-qe-jenkins@cnv-qe-01 cnv-tests]$ kubectl get hco kubevirt-hyperconverged -n openshift-cnv -o json | jq ".status.conditions"
[
,
,
,
,
{ "lastTransitionTime": "2022-12-07T18:51:50Z", "message": "Reconcile completed successfully", "observedGeneration": 3, "reason": "ReconcileCompleted", "status": "True", "type": "Upgradeable" }]
(cnv-tests) [cnv-qe-jenkins@cnv-qe-01 cnv-tests]$ kubectl get hco kubevirt-hyperconverged -n openshift-cnv -o json | jq ".spec.workloadUpdateStrategy"
{
"batchEvictionInterval": "1m0s",
"batchEvictionSize": 10,
"workloadUpdateMethods": []
}
(cnv-tests) [cnv-qe-jenkins@cnv-qe-01 cnv-tests]$
Post ocp upgrade:
(cnv-tests) [cnv-qe-jenkins@cnv-qe-01 cnv-tests]$ kubectl get catalogsource -n openshift-marketplace
NAME DISPLAY TYPE PUBLISHER AGE
certified-operators Certified Operators grpc Red Hat 26h
community-operators Community Operators grpc Red Hat 26h
hco-catalogsource OpenShift Virtualization Index Image grpc Red Hat 25h
ocs-catalogsource OpenShift Container Storage grpc Red Hat 25h
redhat-marketplace Red Hat Marketplace grpc Red Hat 26h
redhat-operators Red Hat Operators grpc Red Hat 26h
(cnv-tests) [cnv-qe-jenkins@cnv-qe-01 cnv-tests]$ kubectl patch operatorhub cluster --type merge -p '{"spec": {"disableAllDefaultSources": true}}'
operatorhub.config.openshift.io/cluster patched
(cnv-tests) [cnv-qe-jenkins@cnv-qe-01 cnv-tests]$ kubectl get catalogsource -n openshift-marketplace
NAME DISPLAY TYPE PUBLISHER AGE
hco-catalogsource OpenShift Virtualization Index Image grpc Red Hat 25h
ocs-catalogsource OpenShift Container Storage grpc Red Hat 25h
(cnv-tests) [cnv-qe-jenkins@cnv-qe-01 cnv-tests]$
(cnv-tests) [cnv-qe-jenkins@cnv-qe-01 cnv-tests]$ kubectl get ip -A
NAMESPACE NAME CSV APPROVAL APPROVED
openshift-cnv install-lqt5n kubevirt-hyperconverged-operator.v4.11.0 Manual false
openshift-cnv install-njk24 kubevirt-hyperconverged-operator.v4.10.7 Manual true
openshift-local-storage install-s8jm4 local-storage-operator.4.10.0-202211041323 Automatic true
openshift-local-storage install-v86lx local-storage-operator.4.11.0-202211072116 Automatic true
openshift-nfd install-t95wm nfd.4.10.0-202211041323 Automatic true
openshift-nfd install-xcqkn nfd.4.11.0-202211091549 Automatic true
openshift-operators install-4xbvx servicemeshoperator.v2.3.0 Automatic true
openshift-operators install-lkc4w jaeger-operator.v1.39.0-3 Automatic true
openshift-operators install-pj9kw jaeger-operator.v1.39.0-3 Automatic true
openshift-sriov-network-operator install-42c9t sriov-network-operator.4.10.0-202211180226 Automatic true
openshift-sriov-network-operator install-lw5p9 sriov-network-operator.4.11.0-202211211407 Automatic true
openshift-storage install-s9qfr ocs-operator.v4.10.9 Automatic true
(cnv-tests) [cnv-qe-jenkins@cnv-qe-01 cnv-tests]$ kubectl patch installplan install-lqt5n --namespace='openshift-cnv' --type='merge' --patch='{"spec":{"approved":true}}'
installplan.operators.coreos.com/install-lqt5n patched
(cnv-tests) [cnv-qe-jenkins@cnv-qe-01 cnv-tests]$ kubectl get csv -n openshift-cnv
NAME DISPLAY VERSION REPLACES PHASE
jaeger-operator.v1.39.0-3 Red Hat OpenShift distributed tracing platform 1.39.0-3 jaeger-operator.v1.34.1-5 Succeeded
kiali-operator.v1.57.3 Kiali Operator 1.57.3 kiali-operator.v1.48.3 Succeeded
kubevirt-hyperconverged-operator.v4.10.7 OpenShift Virtualization 4.10.7 kubevirt-hyperconverged-operator.v4.10.6 Replacing
kubevirt-hyperconverged-operator.v4.11.0 OpenShift Virtualization 4.11.0 kubevirt-hyperconverged-operator.v4.10.7 Installing
servicemeshoperator.v2.3.0 Red Hat OpenShift Service Mesh 2.3.0-0 servicemeshoperator.v2.2.3 Succeeded
(cnv-tests) [cnv-qe-jenkins@cnv-qe-01 cnv-tests]$ kubectl get csv -n openshift-cnv
NAME DISPLAY VERSION REPLACES PHASE
jaeger-operator.v1.39.0-3 Red Hat OpenShift distributed tracing platform 1.39.0-3 jaeger-operator.v1.34.1-5 Succeeded
kiali-operator.v1.57.3 Kiali Operator 1.57.3 kiali-operator.v1.48.3 Succeeded
kubevirt-hyperconverged-operator.v4.11.0 OpenShift Virtualization 4.11.0 kubevirt-hyperconverged-operator.v4.10.7 Succeeded
servicemeshoperator.v2.3.0 Red Hat OpenShift Service Mesh 2.3.0-0 servicemeshoperator.v2.2.3 Succeeded
(cnv-tests) [cnv-qe-jenkins@cnv-qe-01 cnv-tests]$ kubectl get hco kubevirt-hyperconverged -n openshift-cnv -o json | jq ".status.versions"
[
]
(cnv-tests) [cnv-qe-jenkins@cnv-qe-01 cnv-tests]$ kubectl get hco kubevirt-hyperconverged -n openshift-cnv -o json | jq ".spec.workloadUpdateStrategy"
{
"batchEvictionInterval": "1m0s",
"batchEvictionSize": 10,
"workloadUpdateMethods": [
"LiveMigrate"
]
}
(cnv-tests) [cnv-qe-jenkins@cnv-qe-01 cnv-tests]$
Expected results:
hco.spec.workloadUpdateStrategy should stay unaltered.
Additional info:
This impacts EUS->EUS upgrade. Without ability to pause workload updates till CNV is updated to 4.12, we can't achive EUS->EUS upgradability