Uploaded image for project: 'OpenShift Service Mesh'
  1. OpenShift Service Mesh
  2. OSSM-12436

CNI fails on OCP clusters with RHEL 8 worker nodes

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Critical Critical
    • None
    • OSSM 3.0.0
    • Istio, Sail Operator
    • None
    • False
    • Hide

      None

      Show
      None
    • False

      RHEL8 worker node is still supported on OCP 4.16.

       

      RHEL8 contains glibc-2.28 which is causing failures in istio-cni:

      Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_cards-6b8bd54c76-fpfjg_cbo-cards-dev_f3a37e91-4212-4ca5-8b13-4306ff1b7af0_0(763d748a3cb3b0a86e0b488624efeaf33dc3ed684dcb98a586b55552ff0d5527): error adding pod cbo-cards-dev_cards-6b8bd54c76-fpfjg to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"763d748a3cb3b0a86e0b488624efeaf33dc3ed684dcb98a586b55552ff0d5527" Netns:"/var/run/netns/f64da979-d3b6-49f4-85ae-a8f46870a265" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=cbo-cards-dev;K8S_POD_NAME=cards-6b8bd54c76-fpfjg;K8S_POD_INFRA_CONTAINER_ID=763d748a3cb3b0a86e0b488624efeaf33dc3ed684dcb98a586b55552ff0d5527;K8S_POD_UID=f3a37e91-4212-4ca5-8b13-4306ff1b7af0" Path:"" ERRORED: error configuring pod [cbo-cards-dev/cards-6b8bd54c76-fpfjg] networking: [cbo-cards-dev/cards-6b8bd54c76-fpfjg/f3a37e91-4212-4ca5-8b13-4306ff1b7af0:istio-cni]: error adding container to network "istio-cni": netplugin failed: "/var/lib/cni/bin/istio-cni: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by /var/lib/cni/bin/istio-cni)\n/var/lib/cni/bin/istio-cni: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by /var/lib/cni/bin/istio-cni)\n" ': StdinData: {"binDir":"/var/lib/cni/bin","capabilities":{"bandwidth":true,"portMappings":true},"clusterNetwork":"/host/run/multus/cni/net.d/10-calico.conflist","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
      

      Istio CNI version: v1.24.3

      By default OCP 4.16 uses RHCOS worker nodes which are based on RHEL 9 so the issue is not visible. It's also not visible on OSSM 2.6.

      cni bin is dynamically linked because of FIPS compliance:

      sh-5.1# file /var/lib/cni/bin/istio-cni /var/lib/cni/bin/istio-cni: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=68ef7feaa2afe01c86792098eef13134106ede05, for GNU/Linux 3.2.0, stripped

      It's not clear which OCP versions support RHEL worker nodes: https://issues.redhat.com/browse/OCPBUGS-75891

              Unassigned Unassigned
              fbrychta@redhat.com Filip Brychta
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated: