Uploaded image for project: 'OpenShift Virtualization'
  1. OpenShift Virtualization
  2. CNV-16915

[2064034] On an upgraded cluster NetworkAddonsConfig seems to be reconciling in a loop

XMLWordPrintable

      Description of problem: This has been observed in two upgraded clusters. 4.9.2->4.10.0 and 4.9.3->4.10.0. In both the clusters hco-operator pod log is showing CNAO to be in reconcile loop.

      Version-Release number of selected component (if applicable):
      4.10.0

      How reproducible:
      Seen in two upgraded cluster.

      Steps to Reproduce:
      1. Check kubectl get hco kubevirt-hyperconverged -n openshift-cnv -o json | jq '.status.relatedObjects[] | select(.kind == "NetworkAddonsConfig")' and kubectl get NetworkAddonsConfig cluster -o json | jq ".metadata" every few seconds on an upgraded cluster
      2. Check kubectl logs -n openshift-cnv hco-operator-85fcb84fc5-bnxc4 | grep "Reconciling for *v1.NetworkAddonsConfig" | wc -l
      3.

      Actual results:
      ===================
      [cnv-qe-jenkins@upg493to410-dwlrj-executor ~]$ kubectl get hco kubevirt-hyperconverged -n openshift-cnv -o json | jq '.status.relatedObjects[] | select(.kind == "NetworkAddonsConfig")'
      {
      "apiVersion": "networkaddonsoperator.network.kubevirt.io/v1",
      "kind": "NetworkAddonsConfig",
      "name": "cluster",
      "resourceVersion": "5156364",
      "uid": "8abecbfc-c657-4068-aef3-2daf19eeb8ff"
      }
      [cnv-qe-jenkins@upg493to410-dwlrj-executor ~]$ kubectl get NetworkAddonsConfig cluster -o json | jq ".metadata"
      {
      "creationTimestamp": "2022-03-11T21:29:18Z",
      "generation": 75,
      "labels":

      { "app": "kubevirt-hyperconverged", "app.kubernetes.io/component": "network", "app.kubernetes.io/managed-by": "hco-operator", "app.kubernetes.io/part-of": "hyperconverged-cluster", "app.kubernetes.io/version": "4.10.0" }

      ,
      "name": "cluster",
      "resourceVersion": "5156364",
      "uid": "8abecbfc-c657-4068-aef3-2daf19eeb8ff"
      }
      [cnv-qe-jenkins@upg493to410-dwlrj-executor ~]$ kubectl get hco kubevirt-hyperconverged -n openshift-cnv -o json | jq '.status.relatedObjects[] | select(.kind == "NetworkAddonsConfig")'
      {
      "apiVersion": "networkaddonsoperator.network.kubevirt.io/v1",
      "kind": "NetworkAddonsConfig",
      "name": "cluster",
      "resourceVersion": "5156364",
      "uid": "8abecbfc-c657-4068-aef3-2daf19eeb8ff"
      }
      [cnv-qe-jenkins@upg493to410-dwlrj-executor ~]$ kubectl get hco kubevirt-hyperconverged -n openshift-cnv -o json | jq '.status.relatedObjects[] | select(.kind == "NetworkAddonsConfig")'
      {
      "apiVersion": "networkaddonsoperator.network.kubevirt.io/v1",
      "kind": "NetworkAddonsConfig",
      "name": "cluster",
      "resourceVersion": "5157488",
      "uid": "8abecbfc-c657-4068-aef3-2daf19eeb8ff"
      }
      [cnv-qe-jenkins@upg493to410-dwlrj-executor ~]$ kubectl get NetworkAddonsConfig cluster -o json | jq ".metadata"
      {
      "creationTimestamp": "2022-03-11T21:29:18Z",
      "generation": 75,
      "labels":

      { "app": "kubevirt-hyperconverged", "app.kubernetes.io/component": "network", "app.kubernetes.io/managed-by": "hco-operator", "app.kubernetes.io/part-of": "hyperconverged-cluster", "app.kubernetes.io/version": "4.10.0" }

      ,
      "name": "cluster",
      "resourceVersion": "5157488",
      "uid": "8abecbfc-c657-4068-aef3-2daf19eeb8ff"
      }
      [cnv-qe-jenkins@upg493to410-dwlrj-executor ~]$ kubectl logs -n openshift-cnv hco-operator-85fcb84fc5-bnxc4 | grep "Reconciling for *v1.NetworkAddonsConfig" | wc -l
      3869
      [cnv-qe-jenkins@upg493to410-dwlrj-executor ~]$
      ==========================

      [cnv-qe-jenkins@upg492to410-qp6zs-executor ~]$ oc logs -n openshift-cnv hco-operator-85fcb84fc5-vkh4z | grep "Reconciling for *v1.NetworkAddonsConfig" | wc -l
      272
      [cnv-qe-jenkins@upg492to410-qp6zs-executor ~]$ kubectl get NetworkAddonsConfig cluster -o json | jq ".metadata"
      {
      "creationTimestamp": "2022-03-11T08:09:20Z",
      "generation": 75,
      "labels":

      { "app": "kubevirt-hyperconverged", "app.kubernetes.io/component": "network", "app.kubernetes.io/managed-by": "hco-operator", "app.kubernetes.io/part-of": "hyperconverged-cluster", "app.kubernetes.io/version": "4.10.0" }

      ,
      "name": "cluster",
      "resourceVersion": "6105373",
      "uid": "191c4ea8-fc9c-495c-8881-b85af795c5ec"
      }
      [cnv-qe-jenkins@upg492to410-qp6zs-executor ~]$ kubectl get hco kubevirt-hyperconverged -n openshift-cnv -o json | jq '.status.relatedObjects[] | select(.kind == "NetworkAddonsConfig")'
      {
      "apiVersion": "networkaddonsoperator.network.kubevirt.io/v1",
      "kind": "NetworkAddonsConfig",
      "name": "cluster",
      "resourceVersion": "6105373",
      "uid": "191c4ea8-fc9c-495c-8881-b85af795c5ec"
      }
      [cnv-qe-jenkins@upg492to410-qp6zs-executor ~]$
      [cnv-qe-jenkins@upg492to410-qp6zs-executor ~]$ kubectl get hco kubevirt-hyperconverged -n openshift-cnv -o json | jq '.status.relatedObjects[] | select(.kind == "NetworkAddonsConfig")'
      I0314 22:18:30.871004 262375 request.go:665] Waited for 1.118742316s due to client-side throttling, not priority and fairness, request: GET:https://api.upg492to410.cnv-qe.rhcloud.com:6443/apis/migrations.kubevirt.io/v1alpha1?timeout=32s
      {
      "apiVersion": "networkaddonsoperator.network.kubevirt.io/v1",
      "kind": "NetworkAddonsConfig",
      "name": "cluster",
      "resourceVersion": "6106523",
      "uid": "191c4ea8-fc9c-495c-8881-b85af795c5ec"
      }
      [cnv-qe-jenkins@upg492to410-qp6zs-executor ~]$ kubectl get NetworkAddonsConfig cluster -o json | jq ".metadata"
      {
      "creationTimestamp": "2022-03-11T08:09:20Z",
      "generation": 75,
      "labels":

      { "app": "kubevirt-hyperconverged", "app.kubernetes.io/component": "network", "app.kubernetes.io/managed-by": "hco-operator", "app.kubernetes.io/part-of": "hyperconverged-cluster", "app.kubernetes.io/version": "4.10.0" }

      ,
      "name": "cluster",
      "resourceVersion": "6106523",
      "uid": "191c4ea8-fc9c-495c-8881-b85af795c5ec"
      }
      [cnv-qe-jenkins@upg492to410-qp6zs-executor ~]$

      Expected results:

      Additional info:

            ralavi@redhat.com Ram Lavi
            rhn-support-dbasunag Debarati Basu-Nag
            Yossi Segev Yossi Segev
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: