Uploaded image for project: 'RHEL'
  1. RHEL
  2. RHEL-84950

HOME has disappeared after switching the container runtime from runc to crun - [rhel 9.6] 0day

    • crun-1.21-1.el9_6
    • Yes
    • Important
    • 0day
    • 2
    • rhel-sst-container-tools
    • 2
    • False
    • Hide

      None

      Show
      None
    • None
    • RUN 268, RUN 269
    • Unspecified
    • Unspecified
    • Unspecified
    • All
    • None

      A new crun is needed for RHEL 9.6/10.0 Zero Day.  This issue is based on the one found in OCP with this card: https://issues.redhat.com/browse/OCPBUGS-45016

      Description from that card, many details in the comments there.

      Description of problem:

      There is the following incompatibility between crun and runc.

      • crun: HOME environment variable is missing when uid is not in /etc/passwd
      • runc: HOME environment variable is set to "/" when uid is not in /etc/passwd

      It can cause unexpected different behaviors on containers.
      In our case, curl became unable to run on our elasticsearch pod after switched to crun.

      This incompatibility should be fixed before OpenShift 4.18 release, which plans to use crun as default container runtime.

      Version-Release number of selected component (if applicable):

      OpenShift v4.17.5

      How reproducible:

      Always

      Steps to Reproduce:

      Step1. Configure the cluster to use crun as Container Runtime.

      $ cat defaultruntime-crun-worker.yaml
      apiVersion: machineconfiguration.openshift.io/v1
      kind: ContainerRuntimeConfig
      metadata:
        name: defaultruntime-crun-worker
      spec:
        containerRuntimeConfig:
          defaultRuntime: crun
        machineConfigPoolSelector:
          matchLabels:
            pools.operator.machineconfiguration.openshift.io/worker: "" 
      $ oc create -f defaultruntime-crun-worker.yaml
      

      Step2. Create namespace "external-es"

      $ oc create namespace external-es
      

      Step3. Deploy the test pod whose name is "external-es".

      $ cat external-es.yaml
      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: external-es
        namespace: external-es
        labels:
          app: external-es
      spec:
        replicas: 1
        selector:
          matchLabels:
            app: external-es
        template:
          metadata:
            labels:
              app: external-es
          spec:
            containers:
            - name: external-es-container
              image: docker.io/elasticsearch:6.8.22
              ports:
              - containerPort: 9200
                name: http
                protocol: TCP
              - containerPort: 9300
                name: transport
                protocol: TCP
              env:
              - name: "discovery.type" 
                value: "single-node" 
              - name: "ES_JAVA_OPTS" 
                value: "-Xms1024m -Xmx1024m" 
              volumeMounts:
              - mountPath: /usr/share/elasticsearch/data
                name: es-data
            volumes:
            - name: es-data
      $ oc create -f external-es.yaml
      

      Step4. Wait until the pod starts to run.

      $ oc rollout status -n external-es deployment/external-es
      {noformat
      
      Step5. Check the existance of HOME and the result of curl command.
      
      

      $ oc exec -it -n external-es deployment/external-es – /bin/sh -xc 'id; cat /etc/passwd; env | grep -w HOME || echo "HOME is missing"; curl localhost:9200'
      + id
      uid=1000790000 gid=0(root) groups=0(root),1000790000
      + cat /etc/passwd
      root:x:0:0:root:/root:/bin/bash
      bin:x:1:1:bin:/bin:/sbin/nologin
      daemon:x:2:2:daemon:/sbin:/sbin/nologin
      adm:x:3:4:adm:/var/adm:/sbin/nologin
      lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
      sync:x:5:0:sync:/sbin:/bin/sync
      shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
      halt:x:7:0:halt:/sbin:/sbin/halt
      mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
      operator:x:11:0:operator:/root:/sbin/nologin
      games:x:12:100:games:/usr/games:/sbin/nologin
      ftp:x:14:50:FTP User:/var/ftp:/sbin/nologin
      nobody:x:99:99:Nobody:/:/sbin/nologin
      systemd-network:x:192:192:systemd Network Management:/:/sbin/nologin
      dbus:x:81:81:System message bus:/:/sbin/nologin
      elasticsearch:x:1000:1000::/usr/share/elasticsearch:/bin/bash
      + grep -w HOME
      + env
      + echo 'HOME is missing'
      HOME is missing
      + curl localhost:9200
      curl: (27) Out of memory
      command terminated with exit code 27

       

      Actual results:

      • HOME is missing
      • curl failed to run (Although "Out of memory" is a bit confused, it was caused due to HOME missing. It attempts to find the uid from /etc/passwd when HOME is missing, then exits with an error if the uid wasn't found)

      Expected results:

      • HOME is set to "/" as well as when we ran it with runc
      • curl can run

      Additional information:

      set_home_env(), which is defined in crun's source code, returns 0 without setting HOME when uid is not in /etc/passwd[1].
      But it should set HOME, since "If the user was not found, set it to something reasonable"[2]. In other words, jumps to "exit:" label instead of returing 0.

      [1] https://github.com/containers/crun/blob/2b3faef7ee389bdc509c31dec671eb7285b42372/src/libcrun/utils.c#L1446
      [2] https://github.com/containers/crun/blob/2b3faef7ee389bdc509c31dec671eb7285b42372/src/libcrun/utils.c#L1464

      Note

      Please confirm the partner (the ticket reporter) or TAM before you want to change the the JIRA security level or move this ticket to other Jira projects, because you may inadvertently disclose information in this JIRA that the partner considers confidential

            [RHEL-84950] HOME has disappeared after switching the container runtime from runc to crun - [rhel 9.6] 0day

            Alex Jia added a comment -

            This bug has been verified on crun-1.21-1.el9_6 - https://issues.redhat.com/browse/RUN-2738

            Alex Jia added a comment - This bug has been verified on crun-1.21-1.el9_6 - https://issues.redhat.com/browse/RUN-2738

              container-runtime-eng Container Runtime Eng Bot
              tsweeney@redhat.com Tom Sweeney
              Container Runtime Eng Bot Container Runtime Eng Bot
              Alex Jia Alex Jia
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Created:
                Updated: