pod cannot be ready due to "incompatible CNI versions". It doesn't always happen, it only happens on 2 out of 6 statefulset pods.

(combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_upf-packetprocessing-0-1_cupf_0858c9e3-d9de-498f-95aa-8019a7070fdb_0(47c9115abad6b9ee8a0269ce0785f59176916ac4fa73b4ee70d47a46f107d2d1): error adding pod cupf_upf-packetprocessing-0-1 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"47c9115abad6b9ee8a0269ce0785f59176916ac4fa73b4ee70d47a46f107d2d1" Netns:"/var/run/netns/c858a600-6e8b-44ce-a362-39a7d0902f84" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=cupf;K8S_POD_NAME=upf-packetprocessing-0-1;K8S_POD_INFRA_CONTAINER_ID=47c9115abad6b9ee8a0269ce0785f59176916ac4fa73b4ee70d47a46f107d2d1;K8S_POD_UID=0858c9e3-d9de-498f-95aa-8019a7070fdb" Path:"" ERRORED: error configuring pod [cupf/upf-packetprocessing-0-1] networking: [cupf/upf-packetprocessing-0-1/0858c9e3-d9de-498f-95aa-8019a7070fdb:perf-common-sriov-dpdk-ext0]: error adding container to network "perf-common-sriov-dpdk-ext0": incompatible CNI versions; config is "1.0.0", plugin supports ["0.1.0" "0.2.0" "0.3.0" "0.3.1" "0.4.0"] ': StdinData: {"binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} 

1. Create cluster with sriov operator
# cluster version
NAME     VERSION  AVAILABLE  PROGRESSING  SINCE  STATUS
version  4.16.27  True       False        2d     Cluster version is 4.16.27

# sriov operator version 
sriov-network-operator.v4.16.0-202501081506

2. Create VF by snnp
~~~
spec:
  deviceType: vfio-pci
  mtu: 1500
  needVhostNet: true
  nicSelector:
    deviceID: "1583"
    pfNames:
    - ens2f0
    - ens5f0
    vendor: "8086"
  nodeSelector:
    node-role.kubernetes.io/commonupfp: ""
  numVfs: 2
  priority: 99
  resourceName: perf_common_sriov_dpdk_ext0
~~~

3. Create NAD by sriovnetwork as below
# sriovnetwork
~~~
spec:
  ipam: '{}'
  logLevel: info
  networkNamespace: cupf
  resourceName: perf_common_sriov_dpdk_ext0
  trust: "on"
~~~

# NAD
~~~
Name:         perf-common-sriov-dpdk-ext0
Namespace:    cupf
Labels:       <none>
Annotations:  k8s.v1.cni.cncf.io/resourceName: openshift.io/perf_common_sriov_dpdk_ext0
API Version:  k8s.cni.cncf.io/v1
Kind:         NetworkAttachmentDefinition
Metadata:
  Creation Timestamp:  2025-02-19T02:28:46Z
  Generation:          3
  Resource Version:    1846985014
  UID:                 2a4a707f-0f9a-4705-9e8e-94c1854d9229
Spec:
  Config:  {
    "cniVersion": "1.0.0",
    "name": "perf-common-sriov-dpdk-ext0",
    "type": "sriov",
    "vlan": 0,
    "trust": "on",
    "vlanQoS": 0,
    "logLevel": "info",
    "ipam": {}
}
~~~

4. Deploy 6 statefulset pods which is using "perf-common-sriov-dpdk-ext0". 

2 out of 6 pods are not getting ready. 

$ omg get pods -n cupf -o wide | grep -i packetprocessing
upf-packetprocessing-0-0           15/15  Running  0         1d   10.130.14.54   d112u14-r750.perfcluster2.samsung.local
upf-packetprocessing-0-1           0/15   Pending  0         1d                  d112u16-r750.perfcluster2.samsung.local
upf-packetprocessing-0-2           15/15  Running  0         23h  10.128.20.108  d112u20-r750.perfcluster2.samsung.local
upf-packetprocessing-0-3           0/15   Pending  0         1d                  d112u12-r750.perfcluster2.samsung.local
upf-packetprocessing-0-4           15/15  Running  0         1d   10.128.20.107  d112u20-r750.perfcluster2.samsung.local
upf-packetprocessing-0-5           15/15  Running  0         1d   10.130.14.55   d112u14-r750.perfcluster2.samsung.local

All pods should be up and running.

There are "incompatible CNI versions" errors in multus-jn9gb pod. 

2025-02-27T11:53:28.326153934+09:00 2025-02-27T02:53:28Z [error] [cupf/upf-packetprocessing-0-1/0858c9e3-d9de-498f-95aa-8019a7070fdb:perf-common-sriov-dpdk-ext0]: error adding container to network "perf-common-sriov-dpdk-ext0": incompatible CNI versions; config is "1.0.0", plugin supports ["0.1.0" "0.2.0" "0.3.0" "0.3.1" "0.4.0"]
2025-02-27T11:53:28.326171873+09:00 2025-02-27T02:53:28Z [verbose] ADD finished CNI request ContainerID:"f8619d89e6495c1ed97a5ed0e94ad2b7f40118c5e107b0fe37a1b8a372457459" Netns:"/var/run/netns/b005cd7f-4aea-4e6c-9811-8556d48f4770" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=cupf;K8S_POD_NAME=upf-packetprocessing-0-1;K8S_POD_INFRA_CONTAINER_ID=f8619d89e6495c1ed97a5ed0e94ad2b7f40118c5e107b0fe37a1b8a372457459;K8S_POD_UID=0858c9e3-d9de-498f-95aa-8019a7070fdb" Path:"", result: "", err: error configuring pod [cupf/upf-packetprocessing-0-1] networking: [cupf/upf-packetprocessing-0-1/0858c9e3-d9de-498f-95aa-8019a7070fdb:perf-common-sriov-dpdk-ext0]: error adding container to network "perf-common-sriov-dpdk-ext0": incompatible CNI versions; config is "1.0.0", plugin supports ["0.1.0" "0.2.0" "0.3.0" "0.3.1" "0.4.0"]