Description of problem:
SNO plus worker running 4.12.19 has failed to upgrade to 4.13.0

"Updating to "4.13.0" from "4.12.19" for 2 hours: Unable to apply 4.13.0: wait has exceeded 40 minutes for these operators: network"

Version-Release number of selected component

How reproducible:
First attempt

Steps to Reproduce:
1. Install SNO and worker node
2. Configure with DU profile for Nokia
3. Run upgrade to 4.13.0

Actual results

ClusterVersion: Updating to "4.13.0" from "4.12.19" for 2 hours: Unable to apply 4.13.0: wait has exceeded 40 minutes for these operators: network
ClusterOperators:
clusteroperator/image-registry is progressing: Progressing: All registry resources are removed
NodeCADaemonProgressing: The daemon set node-ca is deploying node pods
clusteroperator/network is degraded because DaemonSet "/openshift-multus/multus-additional-cni-plugins" rollout is not making progress - last change 2023-05-31T14:00:28Z
DaemonSet "/openshift-multus/multus" rollout is not making progress - last change 2023-05-31T14:00:18Z
DaemonSet "/openshift-ovn-kubernetes/ovnkube-node" rollout is not making progress - last change 2023-05-31T14:04:13Z

Expected results:
oc get co | grep -v "True False"
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE
image-registry 4.13.0 True True False 23h Progressing: All registry resources are removed...
network 4.12.19 True True True 24h DaemonSet "/openshift-multus/multus" rollout is not making progress - last change 2023-05-31T14:00:18Z...

Expected successful upgrade

Additional info:

oc get pods -n openshift-multus
NAME READY STATUS RESTARTS AGE
multus-additional-cni-plugins-ndwjn 1/1 Running 0 40m
multus-additional-cni-plugins-vpk7q 0/1 Pending 0 40m
multus-admission-controller-cf5fcd7cb-hsfd9 2/2 Running 0 40m
multus-nj6wl 0/1 Pending 0 40m
multus-z67kh 1/1 Running 0 40m
network-metrics-daemon-6wv86 0/2 Pending 0 40m
network-metrics-daemon-b78bj 2/2 Running 0 40m

8m Warning FailedScheduling pod/network-metrics-daemon-49htb 0/2 nodes are available: 1 Insufficient management.workload.openshift.io/cores. preemption: 0/2 nodes are available: 1 Insufficient management.workload.openshift.io/cores, 1 node(s) didn't match Pod's node affinity/selector..

"message": "DaemonSet \"/openshift-multus/network-metrics-daemon\" update is rolling out (1 out of 2 updated)\nDaemonSet \"/openshift-multus/multus-additional-cni-plugins\" is not available (awaiting 1 nodes)\nDaemonSet \"/openshift-multus/multus\" is not available (awaiting 1 nodes)\nDaemonSet \"/openshift-network-diagnostics/network-check-target\" update is rolling out (1 out of 2 updated)\nDaemonSet \"/openshift-ovn-kubernetes/ovnkube-node\" is not available (awaiting 1 nodes)",
"reason": "Deploying",

oc get pod -n openshift-multus -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
multus-additional-cni-plugins-ndwjn 1/1 Running 0 8m15s 10.1.101.17 sno-4 <none> <none>
multus-additional-cni-plugins-vpk7q 0/1 Pending 0 8m16s <none> <none> <none> <none>
multus-admission-controller-cf5fcd7cb-hsfd9 2/2 Running 0 8m16s 10.128.0.10 sno-4 <none> <none>
multus-nj6wl 0/1 Pending 0 8m16s <none> <none> <none> <none>
multus-z67kh 1/1 Running 0 8m14s 10.1.101.17 sno-4 <none> <none>
network-metrics-daemon-6wv86 0/2 Pending 0 8m16s <none> <none> <none> <none>
network-metrics-daemon-b78bj 2/2 Running 0 8m14s 10.128.0.11 sno-4 <none> <none>

oc get node
NAME STATUS ROLES AGE VERSION
sno-4 Ready control-plane,master,worker 23h v1.25.8+37a9a08
sno-5.kni-qe-32.lab.eng.rdu2.redhat.com Ready worker 21h v1.25.8+37a9a08

oc get events
LAST SEEN TYPE REASON OBJECT MESSAGE
73m Normal LeaderElection lease/ovn-kubernetes-master sno-4 became leader
73m Normal Killing pod/ovnkube-master-2nfp9 Stopping container northd
73m Normal Killing pod/ovnkube-master-2nfp9 Stopping container kube-rbac-proxy
73m Normal Killing pod/ovnkube-master-2nfp9 Stopping container sbdb
73m Normal Killing pod/ovnkube-master-2nfp9 Stopping container nbdb
73m Normal Killing pod/ovnkube-master-2nfp9 Stopping container ovn-dbchecker
73m Normal Killing pod/ovnkube-master-2nfp9 Stopping container ovnkube-master
73m Warning FailedPreStopHook pod/ovnkube-master-2nfp9 Exec lifecycle hook ([/bin/bash -c OVN_MANAGE_OVSDB=no /usr/share/ovn/scripts/ovn-ctl stop_northd]) for Container "northd" in Pod "ovnkube-master-2nfp9_openshift-ovn-kubernetes(062d9d29-50d5-44c9-9f7e-78753aaa3747)" failed - error: command '/bin/bash -c OVN_MANAGE_OVSDB=no /usr/share/ovn/scripts/ovn-ctl stop_northd' exited with 137: , message: "Exiting ovn-northd (8).\n"
73m Warning FailedPreStopHook pod/ovnkube-master-2nfp9 Exec lifecycle hook ([/bin/bash -c echo "$(date -Iseconds) - stopping nbdb"...
73m Warning FailedPreStopHook pod/ovnkube-master-2nfp9 Exec lifecycle hook ([/bin/bash -c echo "$(date -Iseconds) - stopping sbdb"...
73m Normal Scheduled pod/ovnkube-master-h7jd7 Successfully assigned openshift-ovn-kubernetes/ovnkube-master-h7jd7 to sno-4
73m Normal Pulled pod/ovnkube-master-h7jd7 Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:397285300c9886e779e553af5582fa9fdf9fb5bf4d99374e04721a5d5b25579e" already present on machine
73m Normal Created pod/ovnkube-master-h7jd7 Created container northd
73m Normal Started pod/ovnkube-master-h7jd7 Started container northd
73m Normal Pulled pod/ovnkube-master-h7jd7 Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:397285300c9886e779e553af5582fa9fdf9fb5bf4d99374e04721a5d5b25579e" already present on machine
73m Normal Created pod/ovnkube-master-h7jd7 Created container nbdb
73m Normal Started pod/ovnkube-master-h7jd7 Started container nbdb
73m Normal Pulled pod/ovnkube-master-h7jd7 Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2b2c89aedaa44964e4cf003ef94963da2e773ace08e601592078adefa482b52" already present on machine
73m Normal Created pod/ovnkube-master-h7jd7 Created container kube-rbac-proxy
73m Normal Started pod/ovnkube-master-h7jd7 Started container kube-rbac-proxy
73m Normal Pulled pod/ovnkube-master-h7jd7 Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:397285300c9886e779e553af5582fa9fdf9fb5bf4d99374e04721a5d5b25579e" already present on machine
73m Normal Created pod/ovnkube-master-h7jd7 Created container sbdb
73m Normal Started pod/ovnkube-master-h7jd7 Started container sbdb
73m Normal Pulled pod/ovnkube-master-h7jd7 Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:397285300c9886e779e553af5582fa9fdf9fb5bf4d99374e04721a5d5b25579e" already present on machine
73m Normal Created pod/ovnkube-master-h7jd7 Created container ovnkube-master
73m Normal Started pod/ovnkube-master-h7jd7 Started container ovnkube-master
73m Normal Pulled pod/ovnkube-master-h7jd7 Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:397285300c9886e779e553af5582fa9fdf9fb5bf4d99374e04721a5d5b25579e" already present on machine
73m Normal Created pod/ovnkube-master-h7jd7 Created container ovn-dbchecker
73m Normal Started pod/ovnkube-master-h7jd7 Started container ovn-dbchecker
73m Normal SuccessfulDelete daemonset/ovnkube-master Deleted pod: ovnkube-master-2nfp9
73m Normal SuccessfulCreate daemonset/ovnkube-master Created pod: ovnkube-master-h7jd7
131m Normal Killing pod/ovnkube-node-9f79k Stopping container ovn-controller
131m Normal Killing pod/ovnkube-node-9f79k Stopping container kube-rbac-proxy
131m Normal Killing pod/ovnkube-node-9f79k Stopping container ovn-acl-logging
131m Normal Killing pod/ovnkube-node-9f79k Stopping container ovnkube-node
131m Normal Killing pod/ovnkube-node-9f79k Stopping container kube-rbac-proxy-ovn-metrics
131m Warning FailedScheduling pod/ovnkube-node-gcj9h 0/2 nodes are available: 1 Insufficient management.workload.openshift.io/cores. preemption: 0/2 nodes are available: 1 Insufficient management.workload.openshift.io/cores, 1 node(s) didn't match Pod's node affinity/selector..
10m Warning FailedScheduling pod/ovnkube-node-gcj9h 0/2 nodes are available: 1 Insufficient management.workload.openshift.io/cores. preemption: 0/2 nodes are available: 1 Insufficient management.workload.openshift.io/cores, 1 node(s) didn't match Pod's node affinity/selector..
6m35s Warning FailedScheduling pod/ovnkube-node-gcj9h skip schedule deleting pod: openshift-ovn-kubernetes/ovnkube-node-gcj9h
6m35s Warning FailedScheduling pod/ovnkube-node-jns8x 0/2 nodes are available: 1 Insufficient management.workload.openshift.io/cores. preemption: 0/2 nodes are available: 1 Insufficient management.workload.openshift.io/cores, 1 node(s) didn't match Pod's node affinity/selector..
66s Warning FailedScheduling pod/ovnkube-node-jns8x 0/2 nodes are available: 1 Insufficient management.workload.openshift.io/cores. preemption: 0/2 nodes are available: 1 Insufficient management.workload.openshift.io/cores, 1 node(s) didn't match Pod's node affinity/selector..
131m Normal Scheduled pod/ovnkube-node-pc4bq Successfully assigned openshift-ovn-kubernetes/ovnkube-node-pc4bq to sno-4
131m Warning FailedMount pod/ovnkube-node-pc4bq MountVolume.SetUp failed for volume "ovn-node-metrics-cert" : failed to sync secret cache: timed out waiting for the condition
131m Normal Pulled pod/ovnkube-node-pc4bq Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:397285300c9886e779e553af5582fa9fdf9fb5bf4d99374e04721a5d5b25579e" already present on machine
131m Normal Created pod/ovnkube-node-pc4bq Created container ovn-controller
131m Normal Started pod/ovnkube-node-pc4bq Started container ovn-controller
131m Normal Pulled pod/ovnkube-node-pc4bq Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:397285300c9886e779e553af5582fa9fdf9fb5bf4d99374e04721a5d5b25579e" already present on machine
131m Normal Created pod/ovnkube-node-pc4bq Created container ovn-acl-logging
131m Normal Started pod/ovnkube-node-pc4bq Started container ovn-acl-logging
131m Normal Pulled pod/ovnkube-node-pc4bq Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2b2c89aedaa44964e4cf003ef94963da2e773ace08e601592078adefa482b52" already present on machine
131m Normal Created pod/ovnkube-node-pc4bq Created container kube-rbac-proxy
131m Normal Started pod/ovnkube-node-pc4bq Started container kube-rbac-proxy
131m Normal Pulled pod/ovnkube-node-pc4bq Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2b2c89aedaa44964e4cf003ef94963da2e773ace08e601592078adefa482b52" already present on machine
131m Normal Created pod/ovnkube-node-pc4bq Created container kube-rbac-proxy-ovn-metrics
131m Normal Started pod/ovnkube-node-pc4bq Started container kube-rbac-proxy-ovn-metrics
131m Normal Pulled pod/ovnkube-node-pc4bq Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:397285300c9886e779e553af5582fa9fdf9fb5bf4d99374e04721a5d5b25579e" already present on machine
131m Normal Created pod/ovnkube-node-pc4bq Created container ovnkube-node
131m Normal Started pod/ovnkube-node-pc4bq Started container ovnkube-node
132m Normal Killing pod/ovnkube-node-r8gzz Stopping container ovn-controller
132m Normal Killing pod/ovnkube-node-r8gzz Stopping container kube-rbac-proxy-ovn-metrics
132m Normal Killing pod/ovnkube-node-r8gzz Stopping container kube-rbac-proxy
132m Normal Killing pod/ovnkube-node-r8gzz Stopping container ovnkube-node
132m Normal Killing pod/ovnkube-node-r8gzz Stopping container ovn-acl-logging
132m Warning Unhealthy pod/ovnkube-node-r8gzz Readiness probe errored: rpc error: code = NotFound desc = container is not created or running: checking if PID of 4ecc7e2ca65de8fa05a52c3e50239c5a582340ff329c1cfbc29ecb34231b359c is running failed: container process not found
132m Normal SuccessfulDelete daemonset/ovnkube-node Deleted pod: ovnkube-node-r8gzz
132m Normal SuccessfulCreate daemonset/ovnkube-node Created pod: ovnkube-node-pc4bq
131m Normal SuccessfulDelete daemonset/ovnkube-node Deleted pod: ovnkube-node-9f79k
131m Normal SuccessfulCreate daemonset/ovnkube-node Created pod: ovnkube-node-gcj9h
6m36s Normal SuccessfulCreate daemonset/ovnkube-node Created pod: ovnkube-node-jns8x
135m Normal Scheduled pod/ovnkube-upgrades-prepuller-57x4k Successfully assigned openshift-ovn-kubernetes/ovnkube-upgrades-prepuller-57x4k to sno-4
135m Normal Pulling pod/ovnkube-upgrades-prepuller-57x4k Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:397285300c9886e779e553af5582fa9fdf9fb5bf4d99374e04721a5d5b25579e"
135m Normal Pulled pod/ovnkube-upgrades-prepuller-57x4k Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:397285300c9886e779e553af5582fa9fdf9fb5bf4d99374e04721a5d5b25579e" in 5.5774436s (5.577449896s including waiting)
135m Normal Created pod/ovnkube-upgrades-prepuller-57x4k Created container ovnkube-upgrades-prepuller
135m Normal Started pod/ovnkube-upgrades-prepuller-57x4k Started container ovnkube-upgrades-prepuller
132m Normal Killing pod/ovnkube-upgrades-prepuller-57x4k Stopping container ovnkube-upgrades-prepuller
135m Normal Scheduled pod/ovnkube-upgrades-prepuller-t49qx Successfully assigned openshift-ovn-kubernetes/ovnkube-upgrades-prepuller-t49qx to sno-5.kni-qe-32.lab.eng.rdu2.redhat.com
135m Normal Pulling pod/ovnkube-upgrades-prepuller-t49qx Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:397285300c9886e779e553af5582fa9fdf9fb5bf4d99374e04721a5d5b25579e"
135m Normal Pulled pod/ovnkube-upgrades-prepuller-t49qx Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:397285300c9886e779e553af5582fa9fdf9fb5bf4d99374e04721a5d5b25579e" in 5.285854175s (5.28585818s including waiting)
135m Normal Created pod/ovnkube-upgrades-prepuller-t49qx Created container ovnkube-upgrades-prepuller
135m Normal Started pod/ovnkube-upgrades-prepuller-t49qx Started container ovnkube-upgrades-prepuller
132m Normal Killing pod/ovnkube-upgrades-prepuller-t49qx Stopping container ovnkube-upgrades-prepuller
135m Normal SuccessfulCreate daemonset/ovnkube-upgrades-prepuller Created pod: ovnkube-upgrades-prepuller-t49qx
135m Normal SuccessfulCreate daemonset/ovnkube-upgrades-prepuller Created pod: ovnkube-upgrades-prepuller-57x4k
[weliang@weliang Debug]$ oc get events
LAST SEEN TYPE REASON OBJECT MESSAGE
79m Normal LeaderElection lease/ovn-kubernetes-master sno-4 became leader
79m Normal Killing pod/ovnkube-master-2nfp9 Stopping container northd
79m Normal Killing pod/ovnkube-master-2nfp9 Stopping container kube-rbac-proxy
79m Normal Killing pod/ovnkube-master-2nfp9 Stopping container sbdb
79m Normal Killing pod/ovnkube-master-2nfp9 Stopping container nbdb
79m Normal Killing pod/ovnkube-master-2nfp9 Stopping container ovn-dbchecker
79m Normal Killing pod/ovnkube-master-2nfp9 Stopping container ovnkube-master
79m Warning FailedPreStopHook pod/ovnkube-master-2nfp9 Exec lifecycle hook ([/bin/bash -c OVN_MANAGE_OVSDB=no /usr/share/ovn/scripts/ovn-ctl stop_northd]) for Container "northd" in Pod "ovnkube-master-2nfp9_openshift-ovn-kubernetes(062d9d29-50d5-44c9-9f7e-78753aaa3747)" failed - error: command '/bin/bash -c OVN_MANAGE_OVSDB=no /usr/share/ovn/scripts/ovn-ctl stop_northd' exited with 137: , message: "Exiting ovn-northd (8).\n"
79m Warning FailedPreStopHook pod/ovnkube-master-2nfp9 Exec lifecycle hook ([/bin/bash -c echo "$(date -Iseconds) - stopping nbdb"...
79m Warning FailedPreStopHook pod/ovnkube-master-2nfp9 Exec lifecycle hook ([/bin/bash -c echo "$(date -Iseconds) - stopping sbdb"...
79m Normal Scheduled pod/ovnkube-master-h7jd7 Successfully assigned openshift-ovn-kubernetes/ovnkube-master-h7jd7 to sno-4
79m Normal Pulled pod/ovnkube-master-h7jd7 Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:397285300c9886e779e553af5582fa9fdf9fb5bf4d99374e04721a5d5b25579e" already present on machine
79m Normal Created pod/ovnkube-master-h7jd7 Created container northd
79m Normal Started pod/ovnkube-master-h7jd7 Started container northd
79m Normal Pulled pod/ovnkube-master-h7jd7 Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:397285300c9886e779e553af5582fa9fdf9fb5bf4d99374e04721a5d5b25579e" already present on machine
79m Normal Created pod/ovnkube-master-h7jd7 Created container nbdb
79m Normal Started pod/ovnkube-master-h7jd7 Started container nbdb
79m Normal Pulled pod/ovnkube-master-h7jd7 Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2b2c89aedaa44964e4cf003ef94963da2e773ace08e601592078adefa482b52" already present on machine
79m Normal Created pod/ovnkube-master-h7jd7 Created container kube-rbac-proxy
79m Normal Started pod/ovnkube-master-h7jd7 Started container kube-rbac-proxy
79m Normal Pulled pod/ovnkube-master-h7jd7 Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:397285300c9886e779e553af5582fa9fdf9fb5bf4d99374e04721a5d5b25579e" already present on machine
79m Normal Created pod/ovnkube-master-h7jd7 Created container sbdb
79m Normal Started pod/ovnkube-master-h7jd7 Started container sbdb
79m Normal Pulled pod/ovnkube-master-h7jd7 Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:397285300c9886e779e553af5582fa9fdf9fb5bf4d99374e04721a5d5b25579e" already present on machine
79m Normal Created pod/ovnkube-master-h7jd7 Created container ovnkube-master
79m Normal Started pod/ovnkube-master-h7jd7 Started container ovnkube-master
79m Normal Pulled pod/ovnkube-master-h7jd7 Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:397285300c9886e779e553af5582fa9fdf9fb5bf4d99374e04721a5d5b25579e" already present on machine
79m Normal Created pod/ovnkube-master-h7jd7 Created container ovn-dbchecker
79m Normal Started pod/ovnkube-master-h7jd7 Started container ovn-dbchecker
79m Normal SuccessfulDelete daemonset/ovnkube-master Deleted pod: ovnkube-master-2nfp9
79m Normal SuccessfulCreate daemonset/ovnkube-master Created pod: ovnkube-master-h7jd7
137m Normal Killing pod/ovnkube-node-9f79k Stopping container ovn-controller
137m Normal Killing pod/ovnkube-node-9f79k Stopping container kube-rbac-proxy
137m Normal Killing pod/ovnkube-node-9f79k Stopping container ovn-acl-logging
137m Normal Killing pod/ovnkube-node-9f79k Stopping container ovnkube-node
137m Normal Killing pod/ovnkube-node-9f79k Stopping container kube-rbac-proxy-ovn-metrics
137m Warning FailedScheduling pod/ovnkube-node-gcj9h 0/2 nodes are available: 1 Insufficient management.workload.openshift.io/cores. preemption: 0/2 nodes are available: 1 Insufficient management.workload.openshift.io/cores, 1 node(s) didn't match Pod's node affinity/selector..
16m Warning FailedScheduling pod/ovnkube-node-gcj9h 0/2 nodes are available: 1 Insufficient management.workload.openshift.io/cores. preemption: 0/2 nodes are available: 1 Insufficient management.workload.openshift.io/cores, 1 node(s) didn't match Pod's node affinity/selector..
12m Warning FailedScheduling pod/ovnkube-node-gcj9h skip schedule deleting pod: openshift-ovn-kubernetes/ovnkube-node-gcj9h
12m Warning FailedScheduling pod/ovnkube-node-jns8x 0/2 nodes are available: 1 Insufficient management.workload.openshift.io/cores. preemption: 0/2 nodes are available: 1 Insufficient management.workload.openshift.io/cores, 1 node(s) didn't match Pod's node affinity/selector..
7m2s Warning FailedScheduling pod/ovnkube-node-jns8x 0/2 nodes are available: 1 Insufficient management.workload.openshift.io/cores. preemption: 0/2 nodes are available: 1 Insufficient management.workload.openshift.io/cores, 1 node(s) didn't match Pod's node affinity/selector..
137m Normal Scheduled pod/ovnkube-node-pc4bq Successfully assigned openshift-ovn-kubernetes/ovnkube-node-pc4bq to sno-4
137m Warning FailedMount pod/ovnkube-node-pc4bq MountVolume.SetUp failed for volume "ovn-node-metrics-cert" : failed to sync secret cache: timed out waiting for the condition
137m Normal Pulled pod/ovnkube-node-pc4bq Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:397285300c9886e779e553af5582fa9fdf9fb5bf4d99374e04721a5d5b25579e" already present on machine
137m Normal Created pod/ovnkube-node-pc4bq Created container ovn-controller
137m Normal Started pod/ovnkube-node-pc4bq Started container ovn-controller
137m Normal Pulled pod/ovnkube-node-pc4bq Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:397285300c9886e779e553af5582fa9fdf9fb5bf4d99374e04721a5d5b25579e" already present on machine
137m Normal Created pod/ovnkube-node-pc4bq Created container ovn-acl-logging
137m Normal Started pod/ovnkube-node-pc4bq Started container ovn-acl-logging
137m Normal Pulled pod/ovnkube-node-pc4bq Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2b2c89aedaa44964e4cf003ef94963da2e773ace08e601592078adefa482b52" already present on machine
137m Normal Created pod/ovnkube-node-pc4bq Created container kube-rbac-proxy
137m Normal Started pod/ovnkube-node-pc4bq Started container kube-rbac-proxy
137m Normal Pulled pod/ovnkube-node-pc4bq Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2b2c89aedaa44964e4cf003ef94963da2e773ace08e601592078adefa482b52" already present on machine
137m Normal Created pod/ovnkube-node-pc4bq Created container kube-rbac-proxy-ovn-metrics
137m Normal Started pod/ovnkube-node-pc4bq Started container kube-rbac-proxy-ovn-metrics
137m Normal Pulled pod/ovnkube-node-pc4bq Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:397285300c9886e779e553af5582fa9fdf9fb5bf4d99374e04721a5d5b25579e" already present on machine
137m Normal Created pod/ovnkube-node-pc4bq Created container ovnkube-node
137m Normal Started pod/ovnkube-node-pc4bq Started container ovnkube-node
137m Normal Killing pod/ovnkube-node-r8gzz Stopping container ovn-controller
137m Normal Killing pod/ovnkube-node-r8gzz Stopping container kube-rbac-proxy-ovn-metrics
137m Normal Killing pod/ovnkube-node-r8gzz Stopping container kube-rbac-proxy
137m Normal Killing pod/ovnkube-node-r8gzz Stopping container ovnkube-node
137m Normal Killing pod/ovnkube-node-r8gzz Stopping container ovn-acl-logging
137m Warning Unhealthy pod/ovnkube-node-r8gzz Readiness probe errored: rpc error: code = NotFound desc = container is not created or running: checking if PID of 4ecc7e2ca65de8fa05a52c3e50239c5a582340ff329c1cfbc29ecb34231b359c is running failed: container process not found
137m Normal SuccessfulDelete daemonset/ovnkube-node Deleted pod: ovnkube-node-r8gzz
137m Normal SuccessfulCreate daemonset/ovnkube-node Created pod: ovnkube-node-pc4bq
137m Normal SuccessfulDelete daemonset/ovnkube-node Deleted pod: ovnkube-node-9f79k
137m Normal SuccessfulCreate daemonset/ovnkube-node Created pod: ovnkube-node-gcj9h
12m Normal SuccessfulCreate daemonset/ovnkube-node Created pod: ovnkube-node-jns8x
141m Normal Scheduled pod/ovnkube-upgrades-prepuller-57x4k Successfully assigned openshift-ovn-kubernetes/ovnkube-upgrades-prepuller-57x4k to sno-4
141m Normal Pulling pod/ovnkube-upgrades-prepuller-57x4k Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:397285300c9886e779e553af5582fa9fdf9fb5bf4d99374e04721a5d5b25579e"
141m Normal Pulled pod/ovnkube-upgrades-prepuller-57x4k Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:397285300c9886e779e553af5582fa9fdf9fb5bf4d99374e04721a5d5b25579e" in 5.5774436s (5.577449896s including waiting)
141m Normal Created pod/ovnkube-upgrades-prepuller-57x4k Created container ovnkube-upgrades-prepuller
141m Normal Started pod/ovnkube-upgrades-prepuller-57x4k Started container ovnkube-upgrades-prepuller
137m Normal Killing pod/ovnkube-upgrades-prepuller-57x4k Stopping container ovnkube-upgrades-prepuller
141m Normal Scheduled pod/ovnkube-upgrades-prepuller-t49qx Successfully assigned openshift-ovn-kubernetes/ovnkube-upgrades-prepuller-t49qx to sno-5.kni-qe-32.lab.eng.rdu2.redhat.com
141m Normal Pulling pod/ovnkube-upgrades-prepuller-t49qx Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:397285300c9886e779e553af5582fa9fdf9fb5bf4d99374e04721a5d5b25579e"
141m Normal Pulled pod/ovnkube-upgrades-prepuller-t49qx Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:397285300c9886e779e553af5582fa9fdf9fb5bf4d99374e04721a5d5b25579e" in 5.285854175s (5.28585818s including waiting)
141m Normal Created pod/ovnkube-upgrades-prepuller-t49qx Created container ovnkube-upgrades-prepuller
141m Normal Started pod/ovnkube-upgrades-prepuller-t49qx Started container ovnkube-upgrades-prepuller
137m Normal Killing pod/ovnkube-upgrades-prepuller-t49qx Stopping container ovnkube-upgrades-prepuller
141m Normal SuccessfulCreate daemonset/ovnkube-upgrades-prepuller Created pod: ovnkube-upgrades-prepuller-t49qx
141m Normal SuccessfulCreate daemonset/ovnkube-upgrades-prepuller Created pod: ovnkube-upgrades-prepuller-57x4k

oc get pod -n openshift-multus -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
multus-additional-cni-plugins-ndwjn 1/1 Running 0 8m15s 10.1.101.17 sno-4 <none> <none>
multus-additional-cni-plugins-vpk7q 0/1 Pending 0 8m16s <none> <none> <none> <none>
multus-admission-controller-cf5fcd7cb-hsfd9 2/2 Running 0 8m16s 10.128.0.10 sno-4 <none> <none>
multus-nj6wl 0/1 Pending 0 8m16s <none> <none> <none> <none>
multus-z67kh 1/1 Running 0 8m14s 10.1.101.17 sno-4 <none> <none>
network-metrics-daemon-6wv86 0/2 Pending 0 8m16s <none> <none> <none> <none>
network-metrics-daemon-b78bj 2/2 Running 0 8m14s 10.128.0.11 sno-4 <none> <none>

apiVersion: v1
kind: Node
metadata:
annotations:
k8s.ovn.org/host-addresses: '["2620:52:0:165::11","10.1.101.17"]'
k8s.ovn.org/l3-gateway-config: '{"default":{"mode":"shared","interface-id":"br-ex_sno-4","mac-address":"50:7c:6f:3b:74:bc","ip-addresses":["10.1.101.17/28","2620:52:0:165::11/124"],"next-hops":["10.1.101.30","2620:52:0:165::1e"],"node-port-enable":"true","vlan-id":"0"}}'
k8s.ovn.org/node-chassis-id: 19eabb55-8734-4a0c-93cf-d4f71839d067
k8s.ovn.org/node-gateway-router-lrp-ifaddr: '

'
k8s.ovn.org/node-mgmt-port-mac-address: 02:ec:c7:ce:07:b7
k8s.ovn.org/node-primary-ifaddr: '

'
k8s.ovn.org/node-subnets: '

'
machineconfiguration.openshift.io/controlPlaneTopology: SingleReplica
machineconfiguration.openshift.io/currentConfig: rendered-master-14d9b60624be75c9a0a0d5533e82a62e
machineconfiguration.openshift.io/desiredConfig: rendered-master-14d9b60624be75c9a0a0d5533e82a62e
machineconfiguration.openshift.io/desiredDrain: uncordon-rendered-master-14d9b60624be75c9a0a0d5533e82a62e
machineconfiguration.openshift.io/lastAppliedDrain: uncordon-rendered-master-14d9b60624be75c9a0a0d5533e82a62e
machineconfiguration.openshift.io/reason: ""
machineconfiguration.openshift.io/state: Done
sriovnetwork.openshift.io/state: Idle
volumes.kubernetes.io/controller-managed-attach-detach: "true"

sno-5--

apiVersion: v1
kind: Node
metadata:
annotations:
k8s.ovn.org/host-addresses: '["10.1.101.18","2620:52:0:165::12"]'
k8s.ovn.org/l3-gateway-config: '{"default":{"mode":"shared","interface-id":"br-ex_sno-5.kni-qe-32.lab.eng.rdu2.redhat.com","mac-address":"50:7c:6f:3b:0e:e8","ip-addresses":["10.1.101.18/28","2620:52:0:165::12/124"],"next-hops":["10.1.101.30","2620:52:0:165::1e"],"node-port-enable":"true","vlan-id":"0"}}'
k8s.ovn.org/node-chassis-id: 4d566f72-a3c0-42e5-9444-1a1808862228
k8s.ovn.org/node-gateway-router-lrp-ifaddr: '

'
k8s.ovn.org/node-mgmt-port-mac-address: 4e:c3:07:98:4d:9d
k8s.ovn.org/node-primary-ifaddr: '

'
k8s.ovn.org/node-subnets: '

'
machineconfiguration.openshift.io/controlPlaneTopology: SingleReplica
machineconfiguration.openshift.io/currentConfig: rendered-worker-d4bd09f4009d6c51007704399141a473
machineconfiguration.openshift.io/desiredConfig: rendered-worker-d4bd09f4009d6c51007704399141a473
machineconfiguration.openshift.io/desiredDrain: uncordon-rendered-worker-d4bd09f4009d6c51007704399141a473
machineconfiguration.openshift.io/lastAppliedDrain: uncordon-rendered-worker-d4bd09f4009d6c51007704399141a473
machineconfiguration.openshift.io/reason: ""
machineconfiguration.openshift.io/state: Done
sriovnetwork.openshift.io/state: Idle
volumes.kubernetes.io/controller-managed-attach-detach: "true"