Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-38005

pod with SRIOV as secondary interface stuck in ContainerCreating

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Critical Critical
    • None
    • 4.17
    • Networking / SR-IOV
    • None
    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • 3
    • Important
    • Yes
    • None
    • None
    • None
    • CNF Network Sprint 258
    • 1
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Description of problem:

          During testing regression for UserDefinedNetworks based on build created via PR https://github.com/openshift/api/pull/1988, SRIOV cases failed.
      The pods with SRIOV VF as secondary interfaces stuck in ContainerCreating with error as below.
      % oc get pods -n e2e-25961-e810xxv
      NAME       READY   STATUS              RESTARTS   AGE
      testpod0   0/1     ContainerCreating   0          89m
      # oc describe pods
      ...
       Events:
        Type     Reason                  Age   From               Message
        ----     ------                  ----  ----               -------
        Normal   Scheduled               13m   default-scheduler  Successfully assigned e2e-25961-e810xxv/testpod0 to openshift-qe-027.lab.eng.rdu2.redhat.com
        Warning  FailedCreatePodSandBox  13m   kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_testpod0_e2e-25961-e810xxv_64ac1bfb-4d33-4503-a5eb-a2c04bdd46b7_0(cbc9d5cb2d556f3b1e7db3c708553fde783dc3d0c8b217a10c8b3c394794acaa): error adding pod e2e-25961-e810xxv_testpod0 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"cbc9d5cb2d556f3b1e7db3c708553fde783dc3d0c8b217a10c8b3c394794acaa" Netns:"/var/run/netns/b7f4f587-76dd-41d0-ba3d-c8692d2cdc42" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=e2e-25961-e810xxv;K8S_POD_NAME=testpod0;K8S_POD_INFRA_CONTAINER_ID=cbc9d5cb2d556f3b1e7db3c708553fde783dc3d0c8b217a10c8b3c394794acaa;K8S_POD_UID=64ac1bfb-4d33-4503-a5eb-a2c04bdd46b7" Path:"" ERRORED: error configuring pod [e2e-25961-e810xxv/testpod0] networking: [e2e-25961-e810xxv/testpod0/64ac1bfb-4d33-4503-a5eb-a2c04bdd46b7:e810xxv]: error adding container to network "e810xxv": plugin type="tuning" failed (add): open /proc/sys/net/ipv4/conf/net1/arp_accept: no such file or directory
      ': StdinData: {"binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
        Warning  FailedCreatePodSandBox  13m  kubelet  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_testpod0_e2e-25961-e810xxv_64ac1bfb-4d33-4503-a5eb-a2c04bdd46b7_0(7052307b84e91fba3622f475be190eef0e7e82668ce54c21a4afd6c331e7dd18): error adding pod e2e-25961-e810xxv_testpod0 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"7052307b84e91fba3622f475be190eef0e7e82668ce54c21a4afd6c331e7dd18" Netns:"/var/run/netns/a3a10633-6c9c-4f75-badb-e1d21ed9ac79" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=e2e-25961-e810xxv;K8S_POD_NAME=testpod0;K8S_POD_INFRA_CONTAINER_ID=7052307b84e91fba3622f475be190eef0e7e82668ce54c21a4afd6c331e7dd18;K8S_POD_UID=64ac1bfb-4d33-4503-a5eb-a2c04bdd46b7" Path:"" ERRORED: error configuring pod [e2e-25961-e810xxv/testpod0] networking: [e2e-25961-e810xxv/testpod0/64ac1bfb-4d33-4503-a5eb-a2c04bdd46b7:e810xxv]: error adding container to network "e810xxv": plugin type="tuning" failed (add): open /proc/sys/net/ipv4/conf/net1/arp_accept: no such file or directory
      ': StdinData: {"binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
        Warning  FailedCreatePodSandBox  13m  kubelet  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_testpod0_e2e-25961-e810xxv_64ac1bfb-4d33-4503-a5eb-a2c04bdd46b7_0(e7fe650f820e74b068c1d2c992dc4161c30da1dd595611a84a12f4fd2e702b12): error adding pod e2e-25961-e810xxv_testpod0 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"e7fe650f820e74b068c1d2c992dc4161c30da1dd595611a84a12f4fd2e702b12" Netns:"/var/run/netns/cb74cf25-b51b-49c5-9f8f-28cda4b04b96" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=e2e-25961-e810xxv;K8S_POD_NAME=testpod0;K8S_POD_INFRA_CONTAINER_ID=e7fe650f820e74b068c1d2c992dc4161c30da1dd595611a84a12f4fd2e702b12;K8S_POD_UID=64ac1bfb-4d33-4503-a5eb-a2c04bdd46b7" Path:"" ERRORED: error configuring pod [e2e-25961-e810xxv/testpod0] networking: [e2e-25961-e810xxv/testpod0/64ac1bfb-4d33-4503-a5eb-a2c04bdd46b7:e810xxv]: error adding container to network "e810xxv": plugin type="tuning" failed (add): open /proc/sys/net/ipv4/conf/net1/arp_accept: no such file or directory

      Version-Release number of selected component (if applicable):

      # oc version
      Client Version: 4.17.0-0.ci.test-2024-08-05-014157-ci-ln-d8tr5z2-latest
      Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
      Server Version: 4.17.0-0.ci.test-2024-08-05-014157-ci-ln-d8tr5z2-latest
      Kubernetes Version: v1.30.2-dirty    

      How reproducible:

      some times

       

      Steps to Reproduce:

       1. create sriov network node policy
      {     "kind": "List",     "apiVersion": "v1",     "metadata": {},     "items": [         {             "apiVersion": "sriovnetwork.openshift.io/v1",             "kind": "SriovNetworkNodePolicy",             "metadata": {                 "name": "e810xxv",                 "namespace": "openshift-sriov-network-operator"             },             "spec": {                 "deviceType": "netdevice",                 "nicSelector": {                     "deviceID": "159b",                     "pfNames": [                         "ens2f0"                     ],                     "vendor": "8086"                 },                 "nodeSelector": {                     "feature.node.kubernetes.io/sriov-capable": "true"                 },                 "numVfs": 2,                 "resourceName": "e810xxv"             }         }     ] }
      
      2. create sriov network
      {     "kind": "List",     "apiVersion": "v1",     "metadata": {},     "items": [         {             "apiVersion": "sriovnetwork.openshift.io/v1",             "kind": "SriovNetwork",             "metadata": {                 "name": "e810xxv",                 "namespace": "openshift-sriov-network-operator"             },             "spec": {                 "ipam": "{\n  \"type\": \"whereabouts\",\n  \"ipRanges\": [{\n      \"range\": \"10.30.0.0/30\"\n    }, {\n      \"range\": \"2001::0/126\"\n  }]\n}\n",                 "linkState": "auto",                 "maxTxRate": 0,                 "metaPlugins": "{\n  \"type\": \"tuning\",\n  \"sysctl\": {\n    \"net.ipv4.conf.IFNAME.arp_accept\": \"0\"\n  }\n}\n",                 "minTxRate": 0,                 "networkNamespace": "e2e-25961-e810xxv",                 "resourceName": "e810xxv",                 "spoofChk": "off",                 "trust": "on",                 "vlan": 0,                 "vlanQoS": 0             }         }     ] }
      
      3. create pod
      {     "kind": "List",     "apiVersion": "v1",     "metadata": {},     "items": [         {             "apiVersion": "v1",             "kind": "Pod",             "metadata": {                 "annotations": {                     "k8s.v1.cni.cncf.io/networks": "e810xxv"                 },                 "labels": {                     "app": "testpod0",                     "name": "sriov-netdevice"                 },                 "name": "testpod0",                 "namespace": "e2e-25961-e810xxv"             },             "spec": {                 "containers": [                     {                         "image": "quay.io/openshifttest/hello-sdn@sha256:c89445416459e7adea9a5a416b3365ed3d74f2491beb904d61dc8d1eb89a72a4",                         "imagePullPolicy": "IfNotPresent",                         "name": "sriovnet"                     }                 ]             }         }     ] }

      Actual results:

          pod fails to run

      Expected results:

          pod should be able to run

      Additional info:

          build was created from PR https://github.com/openshift/api/pull/1988 (SDN-4919,SDN-5135: Graduate UserDefinedNetworks to GA)
      
      # oc version
      Client Version: 4.17.0-0.ci.test-2024-08-05-014157-ci-ln-d8tr5z2-latest
      Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
      Server Version: 4.17.0-0.ci.test-2024-08-05-014157-ci-ln-d8tr5z2-latest
      Kubernetes Version: v1.30.2-dirty
      
      cheked logs on ovnkube pod of worker node:
      2024-08-06T03:12:07.194Z|01309|binding|INFO|Claiming lport e2e-25961-e810xxv_testpod0 for this chassis.
      2024-08-06T03:12:07.194Z|01310|binding|INFO|e2e-25961-e810xxv_testpod0: Claiming 0a:58:0a:82:02:80 10.130.2.128
      2024-08-06T03:12:07.211Z|01311|binding|INFO|Setting lport e2e-25961-e810xxv_testpod0 ovn-installed in OVS
      2024-08-06T03:12:07.211Z|01312|binding|INFO|Setting lport e2e-25961-e810xxv_testpod0 up in Southbound
      2024-08-06T03:12:07.756Z|01313|binding|INFO|Releasing lport e2e-25961-e810xxv_testpod0 from this chassis (sb_readonly=0)
      
      must-gather: https://drive.google.com/file/d/106ygS971IK02-xyedwxKj8hu5VCw8L1c/view?usp=sharing

              sscheink@redhat.com Sebastian Scheinkman
              rhn-support-yingwang Ying Wang
              None
              None
              Ying Wang Ying Wang
              None
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

                Created:
                Updated: