-
Bug
-
Resolution: Done-Errata
-
Blocker
-
None
-
None
From 4.14, due to thick plugin, multus need to consolidate the CNI plugin into one directory '/var/lib/cni/bin'. Hence istio CNI should be moved to '/var/lib/cni/bin', as other CNI plugins located.
—
In OCP 4.13 and earlier, Multus looks for CNI plugin binaries in the following directories:
- /opt/multus/bin
- /var/lib/cni/bin
- /usr/libexec/cni
In OCP 4.14, this is no longer the case. It seems that it only looks for the binaries in /var/lib/cni/bin, as evident from the following failure message:
...error adding container to network "v2-4-istio-cni": failed to find plugin "v2-4-istio-cni" in path [/var/lib/cni/bin]
In OpenShift Service Mesh, we install the binary in /opt/multus/bin, hence the failure. Note: the directory doesn't seem to be mounted in the multus Pods at all).
Version-Release number of selected component (if applicable):
$ oc version
Client Version: 4.13.0-202301261135.p0.g2b6f970.assembly.stream-2b6f970
Kustomize Version: v4.5.7
Server Version: 4.14.0-0.ci-2023-09-13-233607
Kubernetes Version: v1.27.1-3233+2c287eb3d35003-dirty
$ oc -n openshift-multus get ds multus -oyaml | yq '.spec.template.spec.containers[].image'
registry.ci.openshift.org/ocp/4.14-2023-09-13-233607@sha256:0adcfc902237f8e730f8b85553786512e998bbf3bebe97b6a963743a8f776d83
How reproducible:
Always
Steps to Reproduce:
1. Use a 4.14 OpenShift cluster
2. Create the CNI plugin installer DaemonSet in namespace test:
oc apply -f https://gist.githubusercontent.com/luksa/c4d444e918124604839c424339c29a62/raw/1454bd389138980ea3f93bcfaf6026d4821e3543/noop-cni-plugin-installer.yaml
3. Create the test Deployment:
oc apply -f https://gist.githubusercontent.com/luksa/4c7c144ef88b1b0d8f772d6eacdeec14/raw/06b161fdb8c71406f4531d35550bd507a6a25200/test-deployment.yaml
4. Describe the test pod:
oc -n test describe po test
Actual results:
Warning FailedCreatePodSandBox 8s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_test-6cf67dcfb6-bd7bt_test_f7bd5101-6ea9-4c3c-885c-f2a705679e2b_0(9c7928c18663722767f4c439e8e609b607827712ceb37ff76a8d10af0e003ce3): error adding pod test_test-6cf67dcfb6-bd7bt to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: '&{ContainerID:9c7928c18663722767f4c439e8e609b607827712ceb37ff76a8d10af0e003ce3 Netns:/var/run/netns/4bb60c65-f830-43b7-8f4d-1178003678a4 IfName:eth0 Args:IgnoreUnknown=1;K8S_POD_NAMESPACE=test;K8S_POD_NAME=test-6cf67dcfb6-bd7bt;K8S_POD_INFRA_CONTAINER_ID=9c7928c18663722767f4c439e8e609b607827712ceb37ff76a8d10af0e003ce3;K8S_POD_UID=f7bd5101-6ea9-4c3c-885c-f2a705679e2b Path: StdinData:[...]} ContainerID:"9c7928c18663722767f4c439e8e609b607827712ceb37ff76a8d10af0e003ce3" Netns:"/var/run/netns/4bb60c65-f830-43b7-8f4d-1178003678a4" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=test;K8S_POD_NAME=test-6cf67dcfb6-bd7bt;K8S_POD_INFRA_CONTAINER_ID=9c7928c18663722767f4c439e8e609b607827712ceb37ff76a8d10af0e003ce3;K8S_POD_UID=f7bd5101-6ea9-4c3c-885c-f2a705679e2b" Path:"" ERRORED: error configuring pod [test/test-6cf67dcfb6-bd7bt] networking: [test/test-6cf67dcfb6-bd7bt/f7bd5101-6ea9-4c3c-885c-f2a705679e2b:noop-cni-plugin]: error adding container to network "noop-cni-plugin": failed to find plugin "noop-cni-plugin" in path [/var/lib/cni/bin]
Expected results:
The Pod should start with no failures. Multus should find the binary in /opt/multus/bin.
Additional notes:
The same reproducer runs fine on OCP 4.13 and earlier.
We recently reported a related issue where multus-thick didn't look for network conf files in /etc/cni/multus/net.d (see OCPBUGS-18363). That issue was fixed.
- causes
-
-
- Closed
-
-
-
- Closed
-
- is related to
-
-
- Closed
-
- links to
-
RHBA-2023:121906
Red Hat OpenShift Service Mesh Containers for 2.4.4
- mentioned on
1.
|
Cherry-pick OSSM-4957 to 2.4.x |
|
Closed | 
|
Marko Luksa |