-
Bug
-
Resolution: Unresolved
-
Critical
-
None
-
4.17
-
None
-
Quality / Stability / Reliability
-
False
-
-
3
-
Important
-
Yes
-
None
-
None
-
None
-
CNF Network Sprint 258
-
1
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Description of problem:
During testing regression for UserDefinedNetworks based on build created via PR https://github.com/openshift/api/pull/1988, SRIOV cases failed.
The pods with SRIOV VF as secondary interfaces stuck in ContainerCreating with error as below.
% oc get pods -n e2e-25961-e810xxv
NAME READY STATUS RESTARTS AGE
testpod0 0/1 ContainerCreating 0 89m
# oc describe pods
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 13m default-scheduler Successfully assigned e2e-25961-e810xxv/testpod0 to openshift-qe-027.lab.eng.rdu2.redhat.com
Warning FailedCreatePodSandBox 13m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_testpod0_e2e-25961-e810xxv_64ac1bfb-4d33-4503-a5eb-a2c04bdd46b7_0(cbc9d5cb2d556f3b1e7db3c708553fde783dc3d0c8b217a10c8b3c394794acaa): error adding pod e2e-25961-e810xxv_testpod0 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"cbc9d5cb2d556f3b1e7db3c708553fde783dc3d0c8b217a10c8b3c394794acaa" Netns:"/var/run/netns/b7f4f587-76dd-41d0-ba3d-c8692d2cdc42" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=e2e-25961-e810xxv;K8S_POD_NAME=testpod0;K8S_POD_INFRA_CONTAINER_ID=cbc9d5cb2d556f3b1e7db3c708553fde783dc3d0c8b217a10c8b3c394794acaa;K8S_POD_UID=64ac1bfb-4d33-4503-a5eb-a2c04bdd46b7" Path:"" ERRORED: error configuring pod [e2e-25961-e810xxv/testpod0] networking: [e2e-25961-e810xxv/testpod0/64ac1bfb-4d33-4503-a5eb-a2c04bdd46b7:e810xxv]: error adding container to network "e810xxv": plugin type="tuning" failed (add): open /proc/sys/net/ipv4/conf/net1/arp_accept: no such file or directory
': StdinData: {"binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Warning FailedCreatePodSandBox 13m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_testpod0_e2e-25961-e810xxv_64ac1bfb-4d33-4503-a5eb-a2c04bdd46b7_0(7052307b84e91fba3622f475be190eef0e7e82668ce54c21a4afd6c331e7dd18): error adding pod e2e-25961-e810xxv_testpod0 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"7052307b84e91fba3622f475be190eef0e7e82668ce54c21a4afd6c331e7dd18" Netns:"/var/run/netns/a3a10633-6c9c-4f75-badb-e1d21ed9ac79" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=e2e-25961-e810xxv;K8S_POD_NAME=testpod0;K8S_POD_INFRA_CONTAINER_ID=7052307b84e91fba3622f475be190eef0e7e82668ce54c21a4afd6c331e7dd18;K8S_POD_UID=64ac1bfb-4d33-4503-a5eb-a2c04bdd46b7" Path:"" ERRORED: error configuring pod [e2e-25961-e810xxv/testpod0] networking: [e2e-25961-e810xxv/testpod0/64ac1bfb-4d33-4503-a5eb-a2c04bdd46b7:e810xxv]: error adding container to network "e810xxv": plugin type="tuning" failed (add): open /proc/sys/net/ipv4/conf/net1/arp_accept: no such file or directory
': StdinData: {"binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Warning FailedCreatePodSandBox 13m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_testpod0_e2e-25961-e810xxv_64ac1bfb-4d33-4503-a5eb-a2c04bdd46b7_0(e7fe650f820e74b068c1d2c992dc4161c30da1dd595611a84a12f4fd2e702b12): error adding pod e2e-25961-e810xxv_testpod0 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"e7fe650f820e74b068c1d2c992dc4161c30da1dd595611a84a12f4fd2e702b12" Netns:"/var/run/netns/cb74cf25-b51b-49c5-9f8f-28cda4b04b96" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=e2e-25961-e810xxv;K8S_POD_NAME=testpod0;K8S_POD_INFRA_CONTAINER_ID=e7fe650f820e74b068c1d2c992dc4161c30da1dd595611a84a12f4fd2e702b12;K8S_POD_UID=64ac1bfb-4d33-4503-a5eb-a2c04bdd46b7" Path:"" ERRORED: error configuring pod [e2e-25961-e810xxv/testpod0] networking: [e2e-25961-e810xxv/testpod0/64ac1bfb-4d33-4503-a5eb-a2c04bdd46b7:e810xxv]: error adding container to network "e810xxv": plugin type="tuning" failed (add): open /proc/sys/net/ipv4/conf/net1/arp_accept: no such file or directory
Version-Release number of selected component (if applicable):
# oc version Client Version: 4.17.0-0.ci.test-2024-08-05-014157-ci-ln-d8tr5z2-latest Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: 4.17.0-0.ci.test-2024-08-05-014157-ci-ln-d8tr5z2-latest Kubernetes Version: v1.30.2-dirty
How reproducible:
some times
Steps to Reproduce:
1. create sriov network node policy
{ "kind": "List", "apiVersion": "v1", "metadata": {}, "items": [ { "apiVersion": "sriovnetwork.openshift.io/v1", "kind": "SriovNetworkNodePolicy", "metadata": { "name": "e810xxv", "namespace": "openshift-sriov-network-operator" }, "spec": { "deviceType": "netdevice", "nicSelector": { "deviceID": "159b", "pfNames": [ "ens2f0" ], "vendor": "8086" }, "nodeSelector": { "feature.node.kubernetes.io/sriov-capable": "true" }, "numVfs": 2, "resourceName": "e810xxv" } } ] }
2. create sriov network
{ "kind": "List", "apiVersion": "v1", "metadata": {}, "items": [ { "apiVersion": "sriovnetwork.openshift.io/v1", "kind": "SriovNetwork", "metadata": { "name": "e810xxv", "namespace": "openshift-sriov-network-operator" }, "spec": { "ipam": "{\n \"type\": \"whereabouts\",\n \"ipRanges\": [{\n \"range\": \"10.30.0.0/30\"\n }, {\n \"range\": \"2001::0/126\"\n }]\n}\n", "linkState": "auto", "maxTxRate": 0, "metaPlugins": "{\n \"type\": \"tuning\",\n \"sysctl\": {\n \"net.ipv4.conf.IFNAME.arp_accept\": \"0\"\n }\n}\n", "minTxRate": 0, "networkNamespace": "e2e-25961-e810xxv", "resourceName": "e810xxv", "spoofChk": "off", "trust": "on", "vlan": 0, "vlanQoS": 0 } } ] }
3. create pod
{ "kind": "List", "apiVersion": "v1", "metadata": {}, "items": [ { "apiVersion": "v1", "kind": "Pod", "metadata": { "annotations": { "k8s.v1.cni.cncf.io/networks": "e810xxv" }, "labels": { "app": "testpod0", "name": "sriov-netdevice" }, "name": "testpod0", "namespace": "e2e-25961-e810xxv" }, "spec": { "containers": [ { "image": "quay.io/openshifttest/hello-sdn@sha256:c89445416459e7adea9a5a416b3365ed3d74f2491beb904d61dc8d1eb89a72a4", "imagePullPolicy": "IfNotPresent", "name": "sriovnet" } ] } } ] }
Actual results:
pod fails to run
Expected results:
pod should be able to run
Additional info:
build was created from PR https://github.com/openshift/api/pull/1988 (SDN-4919,SDN-5135: Graduate UserDefinedNetworks to GA)
# oc version
Client Version: 4.17.0-0.ci.test-2024-08-05-014157-ci-ln-d8tr5z2-latest
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: 4.17.0-0.ci.test-2024-08-05-014157-ci-ln-d8tr5z2-latest
Kubernetes Version: v1.30.2-dirty
cheked logs on ovnkube pod of worker node:
2024-08-06T03:12:07.194Z|01309|binding|INFO|Claiming lport e2e-25961-e810xxv_testpod0 for this chassis.
2024-08-06T03:12:07.194Z|01310|binding|INFO|e2e-25961-e810xxv_testpod0: Claiming 0a:58:0a:82:02:80 10.130.2.128
2024-08-06T03:12:07.211Z|01311|binding|INFO|Setting lport e2e-25961-e810xxv_testpod0 ovn-installed in OVS
2024-08-06T03:12:07.211Z|01312|binding|INFO|Setting lport e2e-25961-e810xxv_testpod0 up in Southbound
2024-08-06T03:12:07.756Z|01313|binding|INFO|Releasing lport e2e-25961-e810xxv_testpod0 from this chassis (sb_readonly=0)
must-gather: https://drive.google.com/file/d/106ygS971IK02-xyedwxKj8hu5VCw8L1c/view?usp=sharing