-
Bug
-
Resolution: Duplicate
-
Undefined
-
None
-
4.14
-
Quality / Stability / Reliability
-
False
-
-
None
-
Critical
-
No
-
8/30: telco priority pending triage, now w/ OVN-K
-
None
-
None
-
Proposed
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Description of problem:
Bunch of containers in different namespaces are in ContainerCreating state
oc get po -A | grep -v -e Run -e Com NAMESPACE NAME READY STATUS RESTARTS AGE openshift-dns dns-default-2gf6f 0/2 ContainerCreating 2 4h55m openshift-dns dns-default-956d7 0/2 ContainerCreating 2 3h4m openshift-dns dns-default-9gg4f 0/2 ContainerCreating 2 4h55m openshift-dns dns-default-cpvxq 0/2 ContainerCreating 2 4h openshift-dns dns-default-l7cb5 0/2 ContainerCreating 2 4h55m openshift-dns dns-default-nd9s4 0/2 ContainerCreating 2 4h openshift-dns dns-default-vx98w 0/2 ContainerCreating 2 3h6m openshift-ingress-canary ingress-canary-99zkv 0/1 ContainerCreating 1 3h6m openshift-ingress-canary ingress-canary-qj7kd 0/1 ContainerCreating 1 4h openshift-ingress-canary ingress-canary-scszm 0/1 ContainerCreating 1 3h4m openshift-ingress-canary ingress-canary-v72p5 0/1 ContainerCreating 1 4h openshift-multus network-metrics-daemon-75g5s 0/2 ContainerCreating 2 3h6m openshift-multus network-metrics-daemon-czr52 0/2 ContainerCreating 2 3h5m openshift-multus network-metrics-daemon-kbpt4 0/2 ContainerCreating 2 4h1m openshift-multus network-metrics-daemon-m9qb5 0/2 ContainerCreating 2 4h58m openshift-multus network-metrics-daemon-npsf9 0/2 ContainerCreating 2 4h58m openshift-multus network-metrics-daemon-vqxgt 0/2 ContainerCreating 2 4h openshift-multus network-metrics-daemon-zc4cx 0/2 ContainerCreating 2 4h58m openshift-network-diagnostics network-check-target-2xdh4 0/1 ContainerCreating 1 3h5m openshift-network-diagnostics network-check-target-6cgf5 0/1 ContainerCreating 1 3h6m openshift-network-diagnostics network-check-target-fk7qs 0/1 ContainerCreating 1 4h58m openshift-network-diagnostics network-check-target-n4fzf 0/1 ContainerCreating 1 4h58m openshift-network-diagnostics network-check-target-qm5z7 0/1 ContainerCreating 1 4h58m openshift-network-diagnostics network-check-target-smhxh 0/1 ContainerCreating 1 4h openshift-network-diagnostics network-check-target-v5pgj 0/1 ContainerCreating 1 4h1m
all of them report same error: Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 3h8m default-scheduler Successfully assigned openshift-multus/network-metrics-daemon-75g5s to openshift-worker-2.ocp-edge2.lab.eng.tlv2.redhat.com Warning FailedMount 3h7m (x7 over 3h8m) kubelet MountVolume.SetUp failed for volume "metrics-certs" : object "openshift-multus"/"metrics-daemon-secret" not registered Warning NetworkNotReady 3h7m (x18 over 3h8m) kubelet network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: N o CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started? Normal AddedInterface 3h6m multus Add eth0 [10.129.2.3/23 fd01:0:0:6::3/64] from ovn-kubernetes Warning NodeNotReady 102m node-controller Node is not ready Warning FailedMount 91m (x5 over 91m) kubelet MountVolume.SetUp failed for volume "metrics-certs" : object "openshift-multus"/"metrics-daemon-secret" not registered Warning NetworkNotReady 91m (x7 over 91m) kubelet network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: N o CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started? Warning FailedCreatePodSandBox 91m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_network-metrics-daemon-75g5s_openshift-mul tus_d6552aa7-4f5c-45ed-a3e7-5966ef4a9ba0_0(342ab0d1e6108d6744a133804e2ac7266b638eb80e1510b04272fd50e33c38f3): error adding pod openshift-multus_network-metrics-daemon-75g5s to CNI network "multus-cni-network": plugin type= "multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: '&{ContainerID:342ab0d1e6108d6744a133804e2ac7266b638eb80e1510b04272fd50e33c38f3 Netns:/var/run/netns/e7974bdd-8a75-42 cd-9b4f-6dcdcc61053e IfName:eth0 Args:IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-multus;K8S_POD_NAME=network-metrics-daemon-75g5s;K8S_POD_INFRA_CONTAINER_ID=342ab0d1e6108d6744a133804e2ac7266b638eb80e1510b04272fd50e33c38f3 ;K8S_POD_UID=d6552aa7-4f5c-45ed-a3e7-5966ef4a9ba0 Path: StdinData:[123 34 98 105 110 68 105 114 34 58 34 47 118 97 114 47 108 105 98 47 99 110 105 47 98 105 110 34 44 34 99 108 117 115 116 101 114 78 101 116 119 111 114 10 7 34 58 34 47 104 111 115 116 47 114 117 110 47 109 117 108 116 117 115 47 99 110 105 47 110 101 116 46 100 47 49 48 45 111 118 110 45 107 117 98 101 114 110 101 116 101 115 46 99 111 110 102 34 44 34 99 110 105 86 101 114 115 105 111 110 34 58 34 48 46 51 46 49 34 44 34 100 97 101 109 111 110 83 111 99 107 101 116 68 105 114 34 58 34 47 114 117 110 47 109 117 108 116 117 115 47 115 111 99 107 101 116 34 44 34 103 108 111 98 97 108 78 97 10 9 101 115 112 97 99 101 115 34 58 34 100 101 102 97 117 108 116 44 111 112 101 110 115 104 105 102 116 45 109 117 108 116 117 115 44 111 112 101 110 115 104 105 102 116 45 115 114 105 111 118 45 110 101 116 119 111 114 107 45 111 112 101 114 97 116 111 114 34 44 34 108 111 103 76 101 118 101 108 34 58 34 118 101 114 98 111 115 101 34 44 34 108 111 103 84 111 83 116 100 101 114 114 34 58 116 114 117 101 44 34 110 97 109 101 34 58 34 109 117 108 116 117 115 45 99 110 105 45 110 101 116 119 111 114 107 34 44 34 110 97 109 101 115 112 97 99 101 73 115 111 108 97 116 105 111 110 34 58 116 114 117 101 44 34 114 101 97 100 105 110 101 115 115 105 110 100 105 99 97 116 111 114 102 105 108 101 34 58 34 47 104 111 115 116 47 114 117 110 47 109 117 108 116 117 115 47 99 110 105 47 110 101 116 46 100 47 49 48 45 111 118 110 45 107 117 98 101 114 110 101 116 101 115 46 99 111 110 102 34 4 4 34 116 121 112 101 34 58 34 109 117 108 116 117 115 45 115 104 105 109 34 44 10 32 32 32 32 34 99 110 105 86 101 114 115 105 111 110 34 58 32 34 48 46 51 46 49 34 44 10 32 32 32 32 34 99 104 114 111 111 116 68 105 114 34 58 32 34 47 104 111 115 116 114 111 111 116 34 44 10 32 32 32 32 34 108 111 103 84 111 83 116 100 101 114 114 34 58 32 116 114 117 101 44 10 32 32 32 32 34 108 111 103 76 101 118 101 108 34 58 32 34 118 101 114 98 111 115 101 34 44 10 32 32 32 32 34 98 105 110 68 105 114 34 58 32 34 47 118 97 114 47 108 105 98 47 99 110 105 47 98 105 110 34 44 10 32 32 32 32 34 99 110 105 67 111 110 102 105 103 68 105 114 34 58 32 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 34 44 10 32 32 32 32 34 109 117 108 116 117 115 67 111 110 102 105 103 70 105 108 101 34 58 32 34 97 117 116 111 34 44 10 32 32 32 32 34 109 117 108 116 117 115 65 117 116 111 99 111 110 102 105 103 68 105 114 34 58 32 34 47 104 111 115 116 47 114 117 110 47 109 117 108 116 117 115 47 99 110 105 47 110 101 116 46 100 34 44 10 32 32 32 32 34 110 97 109 101 115 112 97 99 101 73 115 111 108 97 116 105 111 110 34 58 32 116 114 117 101 44 10 32 32 32 32 34 103 108 111 98 97 108 78 97 109 101 115 112 97 99 101 115 34 58 32 34 100 101 102 97 117 108 116 44 111 112 101 110 115 104 105 102 116 45 109 117 108 116 117 115 44 111 112 101 110 115 104 105 102 116 45 115 114 105 111 118 45 110 101 116 119 111 114 107 45 111 112 101 114 97 116 111 114 34 44 10 32 32 32 32 34 114 101 97 100 105 110 101 115 115 105 110 100 105 99 97 116 111 114 10 2 105 108 101 34 58 32 34 47 104 111 115 116 47 114 117 110 47 109 117 108 116 117 115 47 99 110 105 47 110 101 116 46 100 47 49 48 45 111 118 110 45 107 117 98 101 114 110 101 116 101 115 46 99 111 110 102 34 44 10 32 32 32 32 34 100 97 101 109 111 110 83 111 99 107 101 116 68 105 114 34 58 32 34 47 114 117 110 47 109 117 108 116 117 115 47 115 111 99 107 101 116 34 44 10 32 32 32 32 34 115 111 99 107 101 116 68 105 114 34 58 32 34 47 104 111 115 116 47 114 117 110 47 109 117 108 116 117 115 47 115 111 99 107 101 116 34 10 125 10]} ContainerID:"342ab0d1e6108d6744a133804e2ac7266b638eb80e1510b04272fd50e33c38f3" Netns:"/var/run/netns/e7974bdd-8a75-42cd-9b4f-6d cdcc61053e" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-multus;K8S_POD_NAME=network-metrics-daemon-75g5s;K8S_POD_INFRA_CONTAINER_ID=342ab0d1e6108d6744a133804e2ac7266b638eb80e1510b04272fd50e33c38f3;K8S_P OD_UID=d6552aa7-4f5c-45ed-a3e7-5966ef4a9ba0" Path:"" ERRORED: error configuring pod [openshift-multus/network-metrics-daemon-75g5s] networking: [openshift-multus/network-metrics-daemon-75g5s/d6552aa7-4f5c-45ed-a3e7-5966ef4 a9ba0:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[openshift-multus/network-metrics-daemon-75g5s 342ab0d1e6108d6744a133804e2ac7266b638eb80e1510b04272fd50e33c38f 3 network default NAD default] [openshift-multus/network-metrics-daemon-75g5s 342ab0d1e6108d6744a133804e2ac7266b638eb80e1510b04272fd50e33c38f3 network default NAD default] failed to configure pod interface: failed to add p od route 10.128.0.0/14 via 10.129.2.1: file exists
Version-Release number of selected component (if applicable):
4.14.0-0.nightly-2023-08-11-055332
How reproducible:
So far 100%
Steps to Reproduce:
1. Deploy disconnected dualstack baremetal cluster with day1 networking 2. Run OVN migration to 2nd interface 3. Check pods
Actual results:
Pods in multiple namespace stuck in ContainerCreating which affect day2 operations(e.g. installation of day2 operators fails)
Expected results:
All pods are successfully restarted
Additional info:
