-
Bug
-
Resolution: Unresolved
-
Normal
-
None
-
4.17
-
None
-
Quality / Stability / Reliability
-
False
-
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Description of problem:
During the upgrade of an OCP cluster from version 4.17.37 to version 4.18.25, the network operator got stuck in version 4.17.37 in a progressing state, while the other operators were upgraded to version 4.18.25. The first suspicion was Bug https://issues.redhat.com/browse/OCPBUGS-57643, but this was ruled out because this cluster did not have daemonset whereabouts-token-watcher and whereabouts-controller deployment. It was noticed that all the pods in the cluster were in the Running state except for the ovnkube-upgrades-prepuller pods.
$ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.17.37 True True 7h23m Working towards 4.18.25: 740 of 904 done (81% complete), waiting on network over 30 minutes which is longer than expected
$ oc get pods -n openshift-ovn-kubernetes -o wide | grep -v Running NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES ovnkube-upgrades-prepuller-54nvf 0/1 ContainerCreating 0 7h 10.3.177.15 bcatxrvmst001.bocpcat01.cat.tenant <none> <none> ovnkube-upgrades-prepuller-m2ftn 0/1 ContainerCreating 0 7h 10.3.177.24 bcatxrvwrk007.bocpcat01.cat.tenant <none> <none>
# oc get co NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE authentication 4.18.25 True False False 8h baremetal 4.18.25 True False False 3y357d cloud-controller-manager 4.18.25 True False False 367d cloud-credential 4.18.25 True False False 3y357d cluster-autoscaler 4.18.25 True False False 3y357d config-operator 4.18.25 True False False 3y357d console 4.18.25 True False False 14h control-plane-machine-set 4.18.25 True False False 366d csi-snapshot-controller 4.18.25 True False False 366d dns 4.18.25 True False False 81d etcd 4.18.25 True False False 276d image-registry 4.18.25 True False False 8h ingress 4.18.25 True False False 13h insights 4.18.25 True False False 348d kube-apiserver 4.18.25 True False False 3y357d kube-controller-manager 4.18.25 True False False 3y357d kube-scheduler 4.18.25 True False False 3y357d kube-storage-version-migrator 4.18.25 True False False 13h machine-api 4.18.25 True False False 3y357d machine-approver 4.18.25 True False False 3y357d machine-config 4.17.37 True False False 22h marketplace 4.18.25 True False False 3y357d monitoring 4.18.25 True False False 53d network 4.17.37 True True False 3y357d DaemonSet "/openshift-ovn-kubernetes/ovnkube-upgrades-prepuller" is not available (awaiting 2 nodes) node-tuning 4.18.25 True False False 8h olm 4.18.25 True False False 8h openshift-apiserver 4.18.25 True False False 15h openshift-controller-manager 4.18.25 True False False 366d openshift-samples 4.18.25 True False False 18h operator-lifecycle-manager 4.18.25 True False False 366d operator-lifecycle-manager-catalog 4.18.25 True False False 3y357d operator-lifecycle-manager-packageserver 4.18.25 True False False 18h service-ca 4.18.25 True False False 3y357d storage 4.18.25 True False False 19h
