-
Bug
-
Resolution: Done-Errata
-
None
-
False
-
-
False
-
CLOSED
-
No
Description of problem: This has been observed in two upgraded clusters. 4.9.2->4.10.0 and 4.9.3->4.10.0. In both the clusters hco-operator pod log is showing CNAO to be in reconcile loop.
Version-Release number of selected component (if applicable):
4.10.0
How reproducible:
Seen in two upgraded cluster.
Steps to Reproduce:
1. Check kubectl get hco kubevirt-hyperconverged -n openshift-cnv -o json | jq '.status.relatedObjects[] | select(.kind == "NetworkAddonsConfig")' and kubectl get NetworkAddonsConfig cluster -o json | jq ".metadata" every few seconds on an upgraded cluster
2. Check kubectl logs -n openshift-cnv hco-operator-85fcb84fc5-bnxc4 | grep "Reconciling for *v1.NetworkAddonsConfig" | wc -l
3.
Actual results:
===================
[cnv-qe-jenkins@upg493to410-dwlrj-executor ~]$ kubectl get hco kubevirt-hyperconverged -n openshift-cnv -o json | jq '.status.relatedObjects[] | select(.kind == "NetworkAddonsConfig")'
{
"apiVersion": "networkaddonsoperator.network.kubevirt.io/v1",
"kind": "NetworkAddonsConfig",
"name": "cluster",
"resourceVersion": "5156364",
"uid": "8abecbfc-c657-4068-aef3-2daf19eeb8ff"
}
[cnv-qe-jenkins@upg493to410-dwlrj-executor ~]$ kubectl get NetworkAddonsConfig cluster -o json | jq ".metadata"
{
"creationTimestamp": "2022-03-11T21:29:18Z",
"generation": 75,
"labels":
,
"name": "cluster",
"resourceVersion": "5156364",
"uid": "8abecbfc-c657-4068-aef3-2daf19eeb8ff"
}
[cnv-qe-jenkins@upg493to410-dwlrj-executor ~]$ kubectl get hco kubevirt-hyperconverged -n openshift-cnv -o json | jq '.status.relatedObjects[] | select(.kind == "NetworkAddonsConfig")'
{
"apiVersion": "networkaddonsoperator.network.kubevirt.io/v1",
"kind": "NetworkAddonsConfig",
"name": "cluster",
"resourceVersion": "5156364",
"uid": "8abecbfc-c657-4068-aef3-2daf19eeb8ff"
}
[cnv-qe-jenkins@upg493to410-dwlrj-executor ~]$ kubectl get hco kubevirt-hyperconverged -n openshift-cnv -o json | jq '.status.relatedObjects[] | select(.kind == "NetworkAddonsConfig")'
{
"apiVersion": "networkaddonsoperator.network.kubevirt.io/v1",
"kind": "NetworkAddonsConfig",
"name": "cluster",
"resourceVersion": "5157488",
"uid": "8abecbfc-c657-4068-aef3-2daf19eeb8ff"
}
[cnv-qe-jenkins@upg493to410-dwlrj-executor ~]$ kubectl get NetworkAddonsConfig cluster -o json | jq ".metadata"
{
"creationTimestamp": "2022-03-11T21:29:18Z",
"generation": 75,
"labels":
,
"name": "cluster",
"resourceVersion": "5157488",
"uid": "8abecbfc-c657-4068-aef3-2daf19eeb8ff"
}
[cnv-qe-jenkins@upg493to410-dwlrj-executor ~]$ kubectl logs -n openshift-cnv hco-operator-85fcb84fc5-bnxc4 | grep "Reconciling for *v1.NetworkAddonsConfig" | wc -l
3869
[cnv-qe-jenkins@upg493to410-dwlrj-executor ~]$
==========================
[cnv-qe-jenkins@upg492to410-qp6zs-executor ~]$ oc logs -n openshift-cnv hco-operator-85fcb84fc5-vkh4z | grep "Reconciling for *v1.NetworkAddonsConfig" | wc -l
272
[cnv-qe-jenkins@upg492to410-qp6zs-executor ~]$ kubectl get NetworkAddonsConfig cluster -o json | jq ".metadata"
{
"creationTimestamp": "2022-03-11T08:09:20Z",
"generation": 75,
"labels":
,
"name": "cluster",
"resourceVersion": "6105373",
"uid": "191c4ea8-fc9c-495c-8881-b85af795c5ec"
}
[cnv-qe-jenkins@upg492to410-qp6zs-executor ~]$ kubectl get hco kubevirt-hyperconverged -n openshift-cnv -o json | jq '.status.relatedObjects[] | select(.kind == "NetworkAddonsConfig")'
{
"apiVersion": "networkaddonsoperator.network.kubevirt.io/v1",
"kind": "NetworkAddonsConfig",
"name": "cluster",
"resourceVersion": "6105373",
"uid": "191c4ea8-fc9c-495c-8881-b85af795c5ec"
}
[cnv-qe-jenkins@upg492to410-qp6zs-executor ~]$
[cnv-qe-jenkins@upg492to410-qp6zs-executor ~]$ kubectl get hco kubevirt-hyperconverged -n openshift-cnv -o json | jq '.status.relatedObjects[] | select(.kind == "NetworkAddonsConfig")'
I0314 22:18:30.871004 262375 request.go:665] Waited for 1.118742316s due to client-side throttling, not priority and fairness, request: GET:https://api.upg492to410.cnv-qe.rhcloud.com:6443/apis/migrations.kubevirt.io/v1alpha1?timeout=32s
{
"apiVersion": "networkaddonsoperator.network.kubevirt.io/v1",
"kind": "NetworkAddonsConfig",
"name": "cluster",
"resourceVersion": "6106523",
"uid": "191c4ea8-fc9c-495c-8881-b85af795c5ec"
}
[cnv-qe-jenkins@upg492to410-qp6zs-executor ~]$ kubectl get NetworkAddonsConfig cluster -o json | jq ".metadata"
{
"creationTimestamp": "2022-03-11T08:09:20Z",
"generation": 75,
"labels":
,
"name": "cluster",
"resourceVersion": "6106523",
"uid": "191c4ea8-fc9c-495c-8881-b85af795c5ec"
}
[cnv-qe-jenkins@upg492to410-qp6zs-executor ~]$
Expected results:
Additional info:
- external trackers