Uploaded image for project: 'OpenShift Virtualization'
  1. OpenShift Virtualization
  2. CNV-17666

[2076292] Upgrade from 4.10.1->4.11 using nightly channel, is not completing with error "could not complete the upgrade process. KubeVirt is not with the expected version. Check KubeVirt observed version in the status field of its CR"

XMLWordPrintable

    • No

      Description of problem: Using nightly channel, on attempting to upgrade from 4.10.1->4.11.0, noticed, csv went to succeeded state, however, hco.status.versions did not get updated to 4.11.0.

      Version-Release number of selected component (if applicable):
      4.11

      How reproducible:
      1/1

      Steps to Reproduce:
      1. Deploy a cluster with 4.10.1 pre-release version
      2. Upgrade ocp, and then attempt to upgrade cnv using nightly-4.11 channel.
      3. CSV reports succeeded state, however, hco.status.version never goes to 4.11.0

      Actual results:
      from deployment log:
      ====================

      {"level":"info","ts":1650159213.9953666,"logger":"controller_hyperconverged","msg":"could not complete the upgrade process. KubeVirt is not with the expected version. Check KubeVirt observed version in the status field of its CR","Request.Namespace":"openshift-cnv","Request.Name":"kubevirt-hyperconverged"}

      ...

      {"level":"info","ts":1650159213.9955654,"logger":"controller_hyperconverged","msg":"could not complete the upgrade process. TektonTasks is not with the expected version. Check TektonTasks observed version in the status field of its CR","Request.Namespace":"openshift-cnv","Request.Name":"kubevirt-hyperconverged"}

      Also seeing these errors flooding hco-operator deployment logs:
      ===================

      {"level":"error","ts":1650159215.6852703,"logger":"controller_hyperconverged","msg":"Failed to update HCO Status","Request.Namespace":"openshift-cnv","Request.Name":"kubevirt-hyperconverged","error":"Operation cannot be fulfilled on hyperconvergeds.hco.kubevirt.io \"kubevirt-hyperconverged\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/kubevirt/hyperconverged-cluster-operator/controllers/hyperconverged.(*ReconcileHyperConverged).updateHyperConverged\n\t/remote-source/app/controllers/hyperconverged/hyperconverged_controller.go:548\ngithub.com/kubevirt/hyperconverged-cluster-operator/controllers/hyperconverged.(*ReconcileHyperConverged).Reconcile\n\t/remote-source/app/controllers/hyperconverged/hyperconverged_controller.go:303\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227"} {"level":"error","ts":1650159215.6853828,"logger":"controller.hyperconverged-controller","msg":"Reconciler error","name":"hco-controlled-cr-aab28ade-fc98-4a29-b8d0-6a719be6f065","namespace":"openshift-cnv","error":"Operation cannot be fulfilled on hyperconvergeds.hco.kubevirt.io \"kubevirt-hyperconverged\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227"} {"level":"info","ts":1650159215.6906817,"logger":"controller_hyperconverged","msg":"The reconciliation got triggered by a secondary CR object","Request.Namespace":"openshift-cnv","Request.Name":"hco-controlled-cr-aab28ade-fc98-4a29-b8d0-6a719be6f065"}

      ========================
      hco.status.version:
      ========================
      [cnv-qe-jenkins@c01-dbn-pr-qmhcw-executor ~]$ kubectl get hco kubevirt-hyperconverged -n openshift-cnv -o json | jq ".status.versions"
      I0418 15:00:58.987087 110407 request.go:665] Waited for 1.127543644s due to client-side throttling, not priority and fairness, request: GET:https://api.c01-dbn-pr.cnv-qe.rhcloud.com:6443/apis/policy/v1?timeout=32s
      [

      { "name": "operator", "version": "4.10.1" }

      ]
      [cnv-qe-jenkins@c01-dbn-pr-qmhcw-executor ~]$
      ========================
      [cnv-qe-jenkins@c01-dbn-pr-qmhcw-executor ~]$ kubectl get TektonTasks tto-kubevirt-hyperconverged -n openshift-cnv -o json | jq ".status"
      {
      "conditions": [

      { "lastHeartbeatTime": "2022-04-18T14:51:11Z", "lastTransitionTime": "2022-04-17T01:33:30Z", "message": "Tekton operator is available", "reason": "Available", "status": "True", "type": "Available" }

      ,

      { "lastHeartbeatTime": "2022-04-18T14:51:11Z", "lastTransitionTime": "2022-04-17T01:33:30Z", "message": "No Tekton tasks resources are progressing", "reason": "Progressing", "status": "False", "type": "Progressing" }

      ,

      { "lastHeartbeatTime": "2022-04-18T14:51:11Z", "lastTransitionTime": "2022-04-17T01:33:30Z", "message": "No Tekton tasks resources are degraded", "reason": "Degraded", "status": "False", "type": "Degraded" }

      ],
      "observedGeneration": 1,
      "observedVersion": "4.11.0",
      "phase": "Deployed"
      }
      [cnv-qe-jenkins@c01-dbn-pr-qmhcw-executor ~]$

      [cnv-qe-jenkins@c01-dbn-pr-qmhcw-executor ~]$ kubectl get csv -n openshift-cnv
      NAME DISPLAY VERSION REPLACES PHASE
      kubevirt-hyperconverged-operator.4.11.0-218 OpenShift Virtualization 4.11.0-218 kubevirt-hyperconverged-operator.4.10.1-70 Succeeded
      [cnv-qe-jenkins@c01-dbn-pr-qmhcw-executor ~]$

      Will attack kubevirt, operatorcondition and deployment log for hco-operator

      Expected results:
      upgrade completed successfully

      Additional info:

              stirabos Simone Tiraboschi
              rhn-support-dbasunag Debarati Basu-Nag
              Debarati Basu-Nag Debarati Basu-Nag
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated:
                Resolved: