Description of problem: Using nightly channel, on attempting to upgrade from 4.10.1->4.11.0, noticed, csv went to succeeded state, however, hco.status.versions did not get updated to 4.11.0.
Version-Release number of selected component (if applicable):
4.11
How reproducible:
1/1
Steps to Reproduce:
1. Deploy a cluster with 4.10.1 pre-release version
2. Upgrade ocp, and then attempt to upgrade cnv using nightly-4.11 channel.
3. CSV reports succeeded state, however, hco.status.version never goes to 4.11.0
Actual results:
from deployment log:
====================
{"level":"info","ts":1650159213.9953666,"logger":"controller_hyperconverged","msg":"could not complete the upgrade process. KubeVirt is not with the expected version. Check KubeVirt observed version in the status field of its CR","Request.Namespace":"openshift-cnv","Request.Name":"kubevirt-hyperconverged"}
...
{"level":"info","ts":1650159213.9955654,"logger":"controller_hyperconverged","msg":"could not complete the upgrade process. TektonTasks is not with the expected version. Check TektonTasks observed version in the status field of its CR","Request.Namespace":"openshift-cnv","Request.Name":"kubevirt-hyperconverged"}
Also seeing these errors flooding hco-operator deployment logs:
===================
{"level":"error","ts":1650159215.6852703,"logger":"controller_hyperconverged","msg":"Failed to update HCO Status","Request.Namespace":"openshift-cnv","Request.Name":"kubevirt-hyperconverged","error":"Operation cannot be fulfilled on hyperconvergeds.hco.kubevirt.io \"kubevirt-hyperconverged\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/kubevirt/hyperconverged-cluster-operator/controllers/hyperconverged.(*ReconcileHyperConverged).updateHyperConverged\n\t/remote-source/app/controllers/hyperconverged/hyperconverged_controller.go:548\ngithub.com/kubevirt/hyperconverged-cluster-operator/controllers/hyperconverged.(*ReconcileHyperConverged).Reconcile\n\t/remote-source/app/controllers/hyperconverged/hyperconverged_controller.go:303\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":1650159215.6853828,"logger":"controller.hyperconverged-controller","msg":"Reconciler error","name":"hco-controlled-cr-aab28ade-fc98-4a29-b8d0-6a719be6f065","namespace":"openshift-cnv","error":"Operation cannot be fulfilled on hyperconvergeds.hco.kubevirt.io \"kubevirt-hyperconverged\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":1650159215.6906817,"logger":"controller_hyperconverged","msg":"The reconciliation got triggered by a secondary CR object","Request.Namespace":"openshift-cnv","Request.Name":"hco-controlled-cr-aab28ade-fc98-4a29-b8d0-6a719be6f065"}
========================
hco.status.version:
========================
[cnv-qe-jenkins@c01-dbn-pr-qmhcw-executor ~]$ kubectl get hco kubevirt-hyperconverged -n openshift-cnv -o json | jq ".status.versions"
I0418 15:00:58.987087 110407 request.go:665] Waited for 1.127543644s due to client-side throttling, not priority and fairness, request: GET:https://api.c01-dbn-pr.cnv-qe.rhcloud.com:6443/apis/policy/v1?timeout=32s
[
{
"name": "operator",
"version": "4.10.1"
}
]
[cnv-qe-jenkins@c01-dbn-pr-qmhcw-executor ~]$
========================
[cnv-qe-jenkins@c01-dbn-pr-qmhcw-executor ~]$ kubectl get TektonTasks tto-kubevirt-hyperconverged -n openshift-cnv -o json | jq ".status"
{
"conditions": [
{
"lastHeartbeatTime": "2022-04-18T14:51:11Z",
"lastTransitionTime": "2022-04-17T01:33:30Z",
"message": "Tekton operator is available",
"reason": "Available",
"status": "True",
"type": "Available"
}
,
{
"lastHeartbeatTime": "2022-04-18T14:51:11Z",
"lastTransitionTime": "2022-04-17T01:33:30Z",
"message": "No Tekton tasks resources are progressing",
"reason": "Progressing",
"status": "False",
"type": "Progressing"
}
,
{
"lastHeartbeatTime": "2022-04-18T14:51:11Z",
"lastTransitionTime": "2022-04-17T01:33:30Z",
"message": "No Tekton tasks resources are degraded",
"reason": "Degraded",
"status": "False",
"type": "Degraded"
}
],
"observedGeneration": 1,
"observedVersion": "4.11.0",
"phase": "Deployed"
}
[cnv-qe-jenkins@c01-dbn-pr-qmhcw-executor ~]$
[cnv-qe-jenkins@c01-dbn-pr-qmhcw-executor ~]$ kubectl get csv -n openshift-cnv
NAME DISPLAY VERSION REPLACES PHASE
kubevirt-hyperconverged-operator.4.11.0-218 OpenShift Virtualization 4.11.0-218 kubevirt-hyperconverged-operator.4.10.1-70 Succeeded
[cnv-qe-jenkins@c01-dbn-pr-qmhcw-executor ~]$
Will attack kubevirt, operatorcondition and deployment log for hco-operator
Expected results:
upgrade completed successfully
Additional info: