Uploaded image for project: 'OpenShift Virtualization'
  1. OpenShift Virtualization
  2. CNV-17900

[2080547] Metrics kubevirt_hco_out_of_band_modifications_count, does not reflect correct modification count when label is added to priorityclass/kubevirt-cluster-critical in a loop

XMLWordPrintable

    • No

      Description of problem: Metrics kubevirt_hco_out_of_band_modifications_count, does not reflect correct modification count when label is added to priorityclass/kubevirt-cluster-critical in a loop

      Version-Release number of selected component (if applicable):
      4.11.0

      How reproducible:
      Intermittently

      Steps to Reproduce:
      1. Update priorityclass/kubevirt-cluster-critical to add a random label
      2. Wait for priorityclass/kubevirt-cluster-critical resource version change
      3. Wait for reconciliation to take place and repeat the steps a few times

      Actual results:updated PriorityClass resource version and stable status condition. However, kubevirt_hco_out_of_band_modifications_count

      {component_name="priorityclass/kubevirt-cluster-critical"} metrics value does not get updated after first few times. I noticed the following error in hco-operator log
      =========== {"level":"info","ts":1651284657.6145072,"logger":"controller_hyperconverged","msg":"Reconciling for *v1.PriorityClass"} {"level":"info","ts":1651284657.6145704,"logger":"controller_hyperconverged","msg":"Reconciling for *v1.PriorityClass"} {"level":"info","ts":1651284657.6146228,"logger":"controller_hyperconverged","msg":"The reconciliation got triggered by a secondary CR object","Request.Namespace":"openshift-cnv","Request.Name":"hco-co ntrolled-cr-395acf97-848a-4940-bad9-5fd1334773c6"} {"level":"info","ts":1651284657.6147294,"logger":"controller_hyperconverged","msg":"KubeVirtPriorityClass already exists","Request.Namespace":"openshift-cnv","Request.Name":"kubevirt-hyperconverged","K ubeVirtPriorityClass.Namespace":"","KubeVirtPriorityClass.Name":"kubevirt-cluster-critical"} {"level":"info","ts":1651284657.6147447,"logger":"controller_hyperconverged","msg":"Reconciling an externally updated PriorityClass's Spec to its opinionated values","Request.Namespace":"openshift-cnv" ,"Request.Name":"kubevirt-hyperconverged"} {"level":"info","ts":1651284657.6408515,"logger":"controller_hyperconverged","msg":"Reconciling for *v1.PriorityClass"} {"level":"info","ts":1651284657.6568205,"logger":"controller_hyperconverged","msg":"Reconciling for *v1.PriorityClass"} {"level":"error","ts":1651284657.656815,"logger":"controller_hyperconverged","msg":"failed to ensure an operand","Request.Namespace":"openshift-cnv","Request.Name":"kubevirt-hyperconverged","error":"Pr iorityClass.scheduling.k8s.io \"kubevirt-cluster-critical\" not found","stacktrace":"github.com/kubevirt/hyperconverged-cluster-operator/controllers/hyperconverged.(*ReconcileHyperConverged).EnsureOper andAndComplete\n\t/remote-source/app/controllers/hyperconverged/hyperconverged_controller.go:480\ngithub.com/kubevirt/hyperconverged-cluster-operator/controllers/hyperconverged.(*ReconcileHyperConverge d).doReconcile\n\t/remote-source/app/controllers/hyperconverged/hyperconverged_controller.go:429\ngithub.com/kubevirt/hyperconverged-cluster-operator/controllers/hyperconverged.(*ReconcileHyperConverge d).Reconcile\n\t/remote-source/app/controllers/hyperconverged/hyperconverged_controller.go:293\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/remote-source/app/vend or/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/app/vendor/sigs.k8s. io/controller-runtime/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/app/vendor/sigs.k8s.io/contro ller-runtime/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pk g/internal/controller/controller.go:227"}
      ===================

      Expected results:
      kubevirt_hco_out_of_band_modifications_count{component_name="priorityclass/kubevirt-cluster-critical"}

      output will show correct out of band modification count. No error message should be present in hco-operator log

      Additional info:

              nunnatsa Nahshon Unna Tsameret
              rhn-support-dbasunag Debarati Basu-Nag
              Satyajit Bulage Satyajit Bulage (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated:
                Resolved: