Uploaded image for project: 'OpenShift Virtualization'
  1. OpenShift Virtualization
  2. CNV-27121

[2180146] upgrade cnv from 4.12.1 to v4.13.0.rhel9-1819 is stuck

XMLWordPrintable

    • CNV Virtualization Sprint 234, CNV Virtualization Sprint 235
    • Critical
    • None

      Description of problem: Attempted to upgrade cnv from 4.12.1->v4.13.0.rhel9-1819 and see that hco is not able to update to 4.13.0

      Version-Release number of selected component (if applicable):
      v4.13.0.rhel9-1819

      How reproducible:
      one out of one attempt

      Steps to Reproduce:
      1. Try to upgrade cnv to v4.13.0.rhel9-1819
      2.
      3.

      Actual results:
      CSV is in succeeded state:
      ==============
      cnv-qe-jenkins@cnv-qe-infra-01:~$ oc get csv -n openshift-cnv | grep kubevirt
      kubevirt-hyperconverged-operator.v4.13.0 OpenShift Virtualization 4.13.0 kubevirt-hyperconverged-operator.v4.12.1 Succeeded
      cnv-qe-jenkins@cnv-qe-infra-01:~$
      ===============
      HCO shows this:
      ===============
      cnv-qe-jenkins@cnv-qe-infra-01:~$ oc get hco kubevirt-hyperconverged -n openshift-cnv -o json | jq ".status.conditions"
      [

      { "lastTransitionTime": "2023-03-20T18:18:01Z", "message": "Reconcile completed successfully", "observedGeneration": 8, "reason": "ReconcileCompleted", "status": "True", "type": "ReconcileComplete" }

      ,

      { "lastTransitionTime": "2023-03-20T18:22:35Z", "message": "Reconcile completed successfully", "observedGeneration": 8, "reason": "ReconcileCompleted", "status": "True", "type": "Available" }

      ,

      { "lastTransitionTime": "2023-03-20T18:18:01Z", "message": "HCO is now upgrading to version 4.13.0", "observedGeneration": 8, "reason": "HCOUpgrading", "status": "True", "type": "Progressing" }

      ,

      { "lastTransitionTime": "2023-03-20T18:22:35Z", "message": "Reconcile completed successfully", "observedGeneration": 8, "reason": "ReconcileCompleted", "status": "False", "type": "Degraded" }

      ,

      { "lastTransitionTime": "2023-03-20T18:22:35Z", "message": "Reconcile completed successfully", "observedGeneration": 8, "reason": "ReconcileCompleted", "status": "True", "type": "Upgradeable" }

      ]
      cnv-qe-jenkins@cnv-qe-infra-01:~$ oc get hco kubevirt-hyperconverged -n openshift-cnv -o json | jq ".status.versions"
      [

      { "name": "operator", "version": "4.12.1" }

      ]
      cnv-qe-jenkins@cnv-qe-infra-01:~$

      From hco operator log I see the following error. Will attach the full log:
      =======================

      {"level":"info","ts":"2023-03-20T18:49:00Z","logger":"controller_hyperconverged","msg":"setting the Upgradeable operator condition","Request.Namespace":"openshift-cnv","Request.Name":"kubevirt-hyperconverged","requested status":false} {"level":"error","ts":"2023-03-20T18:49:00Z","logger":"controller_hyperconverged","msg":"Failed to update HCO CR","Request.Namespace":"openshift-cnv","Request.Name":"kubevirt-hyperconverged","error":"Operation cannot be fulfilled on hyperconvergeds.hco.kubevirt.io \"kubevirt-hyperconverged\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/kubevirt/hyperconverged-cluster-operator/controllers/hyperconverged.(*ReconcileHyperConverged).logHyperConvergedUpdateError\n\t/remote-source/app/controllers/hyperconverged/hyperconverged_controller.go:617\ngithub.com/kubevirt/hyperconverged-cluster-operator/controllers/hyperconverged.(*ReconcileHyperConverged).updateHyperConverged\n\t/remote-source/app/controllers/hyperconverged/hyperconverged_controller.go:576\ngithub.com/kubevirt/hyperconverged-cluster-operator/controllers/hyperconverged.(*ReconcileHyperConverged).Reconcile\n\t/remote-source/app/controllers/hyperconverged/hyperconverged_controller.go:332\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:235"}

      {"level":"error","ts":"2023-03-20T18:49:00Z","msg":"Reconciler error","controller":"hyperconverged-controller","object":

      {"name":"kubevirt-hyperconverged","namespace":"openshift-cnv"}

      ,"namespace":"openshift-cnv","name":"kubevirt-hyperconverged","reconcileID":"0be1f284-54ce-40f6-a9fc-998980a9f8b1","error":"Operation cannot be fulfilled on hyperconvergeds.hco.kubevirt.io \"kubevirt-hyperconverged\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:235"}

      {"level":"info","ts":"2023-03-20T18:49:00Z","logger":"controller_hyperconverged","msg":"Reconciling HyperConverged operator","Request.Namespace":"openshift-cnv","Request.Name":"kubevirt-hyperconverged"}

      {"level":"info","ts":"2023-03-20T18:49:00Z","logger":"controller_hyperconverged","msg":"Resource doesn't exist, there is nothing to remove","Request.Namespace":"openshift-cnv","Request.Name":"kubevirt-hyperconverged","Kind":"&TypeMeta

      {Kind:,APIVersion:,}

      "}
      {"level":"info","ts":"2023-03-20T18:49:00Z","logger":"controller_hyperconverged","msg":"Resource doesn't exist, there is nothing to remove","Request.Namespace":"openshift-cnv","Request.Name":"kubevirt-hyperconverged","Kind":"&TypeMeta

      {Kind:,APIVersion:,}

      "}
      {"level":"info","ts":"2023-03-20T18:49:00Z","logger":"controller_hyperconverged","msg":"Resource doesn't exist, there is nothing to remove","Request.Namespace":"openshift-cnv","Request.Name":"kubevirt-hyperconverged","Kind":"&TypeMeta

      {Kind:,APIVersion:,}

      "}

      {"level":"info","ts":"2023-03-20T18:49:00Z","logger":"controller_hyperconverged","msg":"reading quickstart guides","Request.Namespace":"openshift-cnv","Request.Name":"kubevirt-hyperconverged"} {"level":"info","ts":"2023-03-20T18:49:00Z","logger":"controller_hyperconverged","msg":"KubeVirtPriorityClass already exists","Request.Namespace":"openshift-cnv","Request.Name":"kubevirt-hyperconverged","KubeVirtPriorityClass.Namespace":"","KubeVirtPriorityClass.Name":"kubevirt-cluster-critical"} {"level":"info","ts":"2023-03-20T18:49:00Z","logger":"controller_hyperconverged","msg":"KubeVirt already exists","Request.Namespace":"openshift-cnv","Request.Name":"kubevirt-hyperconverged","KubeVirt.Namespace":"openshift-cnv","KubeVirt.Name":"kubevirt-kubevirt-hyperconverged"} {"level":"info","ts":"2023-03-20T18:49:00Z","logger":"controller_hyperconverged","msg":"could not complete the upgrade process. KubeVirt is not with the expected version. Check KubeVirt observed version in the status field of its CR","Request.Namespace":"openshift-cnv","Request.Name":"kubevirt-hyperconverged"}

      Expected results:
      HCO should upgrade successfully.

      Additional info:

              ibezukh Igor Bezukh
              rhn-support-dbasunag Debarati Basu-Nag
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated:
                Resolved: