Uploaded image for project: 'OpenShift Virtualization'
  1. OpenShift Virtualization
  2. CNV-31014

[2223049] Overwritten/ReconcileHCO events are not consistently getting generated on overwriting many managed objects of HCO

XMLWordPrintable

    • No

      Description of problem:Overwritten/ReconcileHCO events are not consistently getting generated on overwriting many managed objects of HCO

      Version-Release number of selected component (if applicable):
      CNV-v4.14.0.rhel9-1238

      How reproducible:
      Intermittently reproducible

      Steps to Reproduce:
      1. Clean all existing events
      2. Override a managed object, e.g. Role hyperconverged-cluster-operator-metrics
      3. Watch for events

      Actual results:
      No event was generated in HCO namespace
      [cloud-user@ocp-ipi-executor-xl ~]$ oc get hco -n openshift-cnv kubevirt-hyperconverged -o json | jq ".status.conditions"
      [

      { "lastTransitionTime": "2023-07-14T23:19:34Z", "message": "Reconcile completed successfully", "observedGeneration": 4, "reason": "ReconcileCompleted", "status": "True", "type": "ReconcileComplete" }

      ,

      { "lastTransitionTime": "2023-07-14T23:19:37Z", "message": "Reconcile completed successfully", "observedGeneration": 4, "reason": "ReconcileCompleted", "status": "True", "type": "Available" }

      ,

      { "lastTransitionTime": "2023-07-14T23:19:37Z", "message": "Reconcile completed successfully", "observedGeneration": 4, "reason": "ReconcileCompleted", "status": "False", "type": "Progressing" }

      ,

      { "lastTransitionTime": "2023-07-14T23:19:37Z", "message": "Reconcile completed successfully", "observedGeneration": 4, "reason": "ReconcileCompleted", "status": "False", "type": "Degraded" }

      ,

      { "lastTransitionTime": "2023-07-14T23:19:37Z", "message": "Reconcile completed successfully", "observedGeneration": 4, "reason": "ReconcileCompleted", "status": "True", "type": "Upgradeable" }

      ]
      [cloud-user@ocp-ipi-executor-xl ~]$
      [cloud-user@ocp-ipi-executor-xl ~]$ oc get events -n openshift-cnv
      No resources found in openshift-cnv namespace.
      [cloud-user@ocp-ipi-executor-xl ~]$
      I see the following entry in operator log:
      ========================

      {"level":"error","ts":"2023-07-15T01:03:23Z","logger":"controller_hyperconverged","msg":"Failed to update HCO Status","Request.Namespace":"openshift-cnv","Request.Name":"kubevirt-hyperconverged","error":"Operation cannot be fulfilled on hyperconvergeds.hco.kubevirt.io \"kubevirt-hyperconverged\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/kubevirt/hyperconverged-cluster-operator/controllers/hyperconverged.(*ReconcileHyperConverged).logHyperConvergedUpdateError\n\t/remote-source/app/controllers/hyperconverged/hyperconverged_controller.go:637\ngithub.com/kubevirt/hyperconverged-cluster-operator/controllers/hyperconverged.(*ReconcileHyperConverged).updateHyperConverged\n\t/remote-source/app/controllers/hyperconverged/hyperconverged_controller.go:586\ngithub.com/kubevirt/hyperconverged-cluster-operator/controllers/hyperconverged.(*ReconcileHyperConverged).Reconcile\n\t/remote-source/app/controllers/hyperconverged/hyperconverged_controller.go:337\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:118\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:314\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:226"}

      {"level":"error","ts":"2023-07-15T01:03:23Z","msg":"Reconciler error","controller":"hyperconverged-controller","object":

      {"name":"hco-controlled-cr-2a2c8e6e-4e58-4f0e-8eb3-04892f309d6c","namespace":"openshift-cnv"}

      ,"namespace":"openshift-cnv","name":"hco-controlled-cr-2a2c8e6e-4e58-4f0e-8eb3-04892f309d6c","reconcileID":"9f57b2de-6e12-40f8-93ef-2424129f5dcb","error":"Operation cannot be fulfilled on hyperconvergeds.hco.kubevirt.io \"kubevirt-hyperconverged\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:324\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:226"}

      {"level":"info","ts":"2023-07-15T01:03:23Z","logger":"controller_hyperconverged","msg":"The reconciliation got triggered by a secondary CR object","Request.Namespace":"openshift-cnv","Request.Name":"hco-controlled-cr-2a2c8e6e-4e58-4f0e-8eb3-04892f309d6c"}

      Attached is log collected.
      Expected results:
      Overwritten and ReconcileHCO events would be generated

      Additional info:
      Seen this for:
      1) Role hyperconverged-cluster-operator-metrics
      2) Service kubevirt-hyperconverged-operator-metrics
      3) PriorityClass kubevirt-cluster-critical -n openshift-cnv
      4) KubeVirt kubevirt-kubevirt-hyperconverged

      And more

              stirabos Simone Tiraboschi
              rhn-support-dbasunag Debarati Basu-Nag
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated:
                Resolved: