Uploaded image for project: 'OpenShift Service Mesh'
  1. OpenShift Service Mesh
  2. OSSM-4957

istio CNI should be in /var/lib/cni/bin (old title: Regression: multus-thick doesn't look for CNI binaries in /opt/multus/bin)


    • Icon: Bug Bug
    • Resolution: Done-Errata
    • Icon: Blocker Blocker
    • OSSM 2.4.4
    • None
    • Maistra
    • None
    • Proposed

      From 4.14, due to thick plugin, multus need to consolidate the CNI plugin into one directory '/var/lib/cni/bin'. Hence istio CNI should be moved to '/var/lib/cni/bin', as other CNI plugins located.

      In OCP 4.13 and earlier, Multus looks for CNI plugin binaries in the following directories:

      • /opt/multus/bin
      • /var/lib/cni/bin
      • /usr/libexec/cni

      In OCP 4.14, this is no longer the case. It seems that it only looks for the binaries in /var/lib/cni/bin, as evident from the following failure message:

      ...error adding container to network "v2-4-istio-cni": failed to find plugin "v2-4-istio-cni" in path [/var/lib/cni/bin]

      In OpenShift Service Mesh, we install the binary in /opt/multus/bin, hence the failure. Note: the directory doesn't seem to be mounted in the multus Pods at all).

      Version-Release number of selected component (if applicable):

      $ oc version
      Client Version: 4.13.0-202301261135.p0.g2b6f970.assembly.stream-2b6f970
      Kustomize Version: v4.5.7
      Server Version: 4.14.0-0.ci-2023-09-13-233607
      Kubernetes Version: v1.27.1-3233+2c287eb3d35003-dirty
      $ oc -n openshift-multus get ds multus -oyaml | yq '.spec.template.spec.containers[].image'


      How reproducible:



      Steps to Reproduce:

      1. Use a 4.14 OpenShift cluster
      2. Create the CNI plugin installer DaemonSet in namespace test:

      oc apply -f https://gist.githubusercontent.com/luksa/c4d444e918124604839c424339c29a62/raw/1454bd389138980ea3f93bcfaf6026d4821e3543/noop-cni-plugin-installer.yaml

      3. Create the test Deployment:

      oc apply -f https://gist.githubusercontent.com/luksa/4c7c144ef88b1b0d8f772d6eacdeec14/raw/06b161fdb8c71406f4531d35550bd507a6a25200/test-deployment.yaml

      4. Describe the test pod:

      oc -n test describe po test


      Actual results:

        Warning  FailedCreatePodSandBox  8s               kubelet  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_test-6cf67dcfb6-bd7bt_test_f7bd5101-6ea9-4c3c-885c-f2a705679e2b_0(9c7928c18663722767f4c439e8e609b607827712ceb37ff76a8d10af0e003ce3): error adding pod test_test-6cf67dcfb6-bd7bt to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: '&{ContainerID:9c7928c18663722767f4c439e8e609b607827712ceb37ff76a8d10af0e003ce3 Netns:/var/run/netns/4bb60c65-f830-43b7-8f4d-1178003678a4 IfName:eth0 Args:IgnoreUnknown=1;K8S_POD_NAMESPACE=test;K8S_POD_NAME=test-6cf67dcfb6-bd7bt;K8S_POD_INFRA_CONTAINER_ID=9c7928c18663722767f4c439e8e609b607827712ceb37ff76a8d10af0e003ce3;K8S_POD_UID=f7bd5101-6ea9-4c3c-885c-f2a705679e2b Path: StdinData:[...]} ContainerID:"9c7928c18663722767f4c439e8e609b607827712ceb37ff76a8d10af0e003ce3" Netns:"/var/run/netns/4bb60c65-f830-43b7-8f4d-1178003678a4" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=test;K8S_POD_NAME=test-6cf67dcfb6-bd7bt;K8S_POD_INFRA_CONTAINER_ID=9c7928c18663722767f4c439e8e609b607827712ceb37ff76a8d10af0e003ce3;K8S_POD_UID=f7bd5101-6ea9-4c3c-885c-f2a705679e2b" Path:"" ERRORED: error configuring pod [test/test-6cf67dcfb6-bd7bt] networking: [test/test-6cf67dcfb6-bd7bt/f7bd5101-6ea9-4c3c-885c-f2a705679e2b:noop-cni-plugin]: error adding container to network "noop-cni-plugin": failed to find plugin "noop-cni-plugin" in path [/var/lib/cni/bin]


      Expected results:
      The Pod should start with no failures. Multus should find the binary in /opt/multus/bin.

      Additional notes:
      The same reproducer runs fine on OCP 4.13 and earlier.

      We recently reported a related issue where multus-thick didn't look for network conf files in /etc/cni/multus/net.d (see OCPBUGS-18363). That issue was fixed.

            mluksa@redhat.com Marko Luksa
            mluksa@redhat.com Marko Luksa
            0 Vote for this issue
            10 Start watching this issue