Container creation with hostUsers=false fails with enabled feature gates - ProcMountType, UserNamespacesSupport, and UserNamespacesPodSecurityStandards

Client Version: 4.19.0-0.ci-2025-02-05-005433
Kustomize Version: v5.4.2
Server Version: 4.19.0-0.ci-2025-02-05-005433
Kubernetes Version: v1.32.1-dirty

    Always

Enable the necessary feature gates:

oc patch FeatureGate cluster --type merge --patch '{"spec":{"featureSet":"CustomNoUpgrade","customNoUpgrade":{"enabled":["ProcMountType","UserNamespacesSupport","UserNamespacesPodSecurityStandards"]}}}'

1.Create an SCC to enforce user namespaces -

cat << EOF | oc apply -f -
apiVersion: security.openshift.io/v1
kind: SecurityContextConstraints
metadata:
  name: test-user-namespaces-scc
priority: null
allowPrivilegeEscalation: true
allowedCapabilities:
- SETUID
- SETGID
fsGroup:
  type: MustRunAs
  ranges:
  - min: 1234
    max: 65534
runAsUser:
  type: MustRunAs
  uid: 1234
seLinuxContext:
  type: MustRunAs
  seLinuxOptions:
    type: container_engine_t
supplementalGroups:
  type: MustRunAs
  ranges:
  - min: 1234
    max: 65534
userNamespaceLevel: RequirePodLevel
EOF

2. Grant the SCC to a non-admin user

oc adm policy add-scc-to-user test-user-namespaces-scc <non-admin-user>

3. Log in as the non-admin user

4. Create a namespace

oc new-project podman-demo

5. Grant the SCC to the default service account

oc adm policy add-scc-to-user test-user-namespaces-scc -z default -n podman-demo

6. Create a Pod

cat << EOF | oc apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: nested-podman
  namespace: podman-demo
  annotations:
    io.kubernetes.cri-o.Devices: "/dev/fuse,/dev/net/tun"
    openshift.io/scc: test-user-namespaces-scc
spec:
  hostUsers: false
  containers:
  - name: nested-podman
    image: quay.io/cgruver0/che/workspace-base:latest
    securityContext:
      allowPrivilegeEscalation: true
      procMount: Unmasked
      capabilities:
        add:
        - "SETUID"
        - "SETGID"
EOF

Observe that the Pod is created, but that container creation will ultimately fail with Context Deadline Exceeded.

Observe the following errors in the journal on the node where the Pod was created:

Feb 05 12:49:16 ocp-4-19-ci-cp-0 crio[2848]: time="2025-02-05T12:49:16.569805261Z" level=info msg="Creating container: podman-demo/nested-podman/nested-podman" id=c9966e3a-5ed6-46c0-ab5c-d7004c2fcf2d name=/runtime.v1.RuntimeService/CreateContainer
Feb 05 12:49:16 ocp-4-19-ci-cp-0 crio[2848]: time="2025-02-05T12:49:16.570530361Z" level=info msg="Allowed annotations are specified for workload [io.containers.trace-syscall io.kubernetes.cri-o.Devices io.kubernetes.cri-o.LinkLogs]"
Feb 05 12:49:16 ocp-4-19-ci-cp-0 crio[2848]: time="2025-02-05T12:49:16.585686693Z" level=info msg="Allowed annotations are specified for workload [io.containers.trace-syscall io.kubernetes.cri-o.Devices io.kubernetes.cri-o.LinkLogs]"
Feb 05 12:49:16 ocp-4-19-ci-cp-0 crio[2848]: time="2025-02-05T12:49:16.585883683Z" level=warning msg="Failed to open /etc/passwd: open /var/lib/containers/storage/overlay/d774cd5bf4daea139cae456ee2537b7075618a04d7d10294bf34beb845f6a868/merged/etc/passwd: no such file or directory"
Feb 05 12:49:16 ocp-4-19-ci-cp-0 crio[2848]: time="2025-02-05T12:49:16.585902675Z" level=warning msg="Failed to open /etc/group: open /var/lib/containers/storage/overlay/d774cd5bf4daea139cae456ee2537b7075618a04d7d10294bf34beb845f6a868/merged/etc/group: no such file or directory"
Feb 05 12:49:16 ocp-4-19-ci-cp-0 crio[2848]: time="2025-02-05T12:49:16.586893697Z" level=info msg="Allowed annotations are specified for workload [io.containers.trace-syscall io.kubernetes.cri-o.Devices io.kubernetes.cri-o.LinkLogs]"
Feb 05 12:49:16 ocp-4-19-ci-cp-0 systemd[1]: Started crio-conmon-27b296756e49b6a4ae77d3c52d3d98960103b87338aa6a79d75bc21684df4fd7.scope.
Feb 05 12:49:16 ocp-4-19-ci-cp-0 systemd[1]: Started libcrun container.
Feb 05 12:49:16 ocp-4-19-ci-cp-0 systemd[1]: crio-27b296756e49b6a4ae77d3c52d3d98960103b87338aa6a79d75bc21684df4fd7.scope: Deactivated successfully.
Feb 05 12:49:16 ocp-4-19-ci-cp-0 systemd[1]: Stopped libcrun container.
Feb 05 12:49:16 ocp-4-19-ci-cp-0 conmon[89005]: conmon 27b296756e49b6a4ae77 <nwarn>: runtime stderr: cannot mkdir `proc`: Permission denied
Feb 05 12:49:16 ocp-4-19-ci-cp-0 conmon[89005]: conmon 27b296756e49b6a4ae77 <error>: Failed to create container: exit status 1
Feb 05 12:49:16 ocp-4-19-ci-cp-0 crio[2848]: time="2025-02-05T12:49:16.625255271Z" level=error msg="Container creation error: cannot mkdir `proc`: Permission denied\n" id=c9966e3a-5ed6-46c0-ab5c-d7004c2fcf2d name=/runtime.v1.RuntimeService/CreateContainer
Feb 05 12:49:16 ocp-4-19-ci-cp-0 systemd[1]: crio-conmon-27b296756e49b6a4ae77d3c52d3d98960103b87338aa6a79d75bc21684df4fd7.scope: Deactivated successfully. 

Feb 05 12:49:16 ocp-4-19-ci-cp-0 crio[2848]: time="2025-02-05T12:49:16.569805261Z" level=info msg="Creating container: podman-demo/nested-podman/nested-podman" id=c9966e3a-5ed6-46c0-ab5c-d7004c2fcf2d name=/runtime.v1.RuntimeService/CreateContainer
Feb 05 12:49:16 ocp-4-19-ci-cp-0 crio[2848]: time="2025-02-05T12:49:16.570530361Z" level=info msg="Allowed annotations are specified for workload [io.containers.trace-syscall io.kubernetes.cri-o.Devices io.kubernetes.cri-o.LinkLogs]"
Feb 05 12:49:16 ocp-4-19-ci-cp-0 crio[2848]: time="2025-02-05T12:49:16.585686693Z" level=info msg="Allowed annotations are specified for workload [io.containers.trace-syscall io.kubernetes.cri-o.Devices io.kubernetes.cri-o.LinkLogs]"
Feb 05 12:49:16 ocp-4-19-ci-cp-0 crio[2848]: time="2025-02-05T12:49:16.585883683Z" level=warning msg="Failed to open /etc/passwd: open /var/lib/containers/storage/overlay/d774cd5bf4daea139cae456ee2537b7075618a04d7d10294bf34beb845f6a868/merged/etc/passwd: no such file or directory"
Feb 05 12:49:16 ocp-4-19-ci-cp-0 crio[2848]: time="2025-02-05T12:49:16.585902675Z" level=warning msg="Failed to open /etc/group: open /var/lib/containers/storage/overlay/d774cd5bf4daea139cae456ee2537b7075618a04d7d10294bf34beb845f6a868/merged/etc/group: no such file or directory"
Feb 05 12:49:16 ocp-4-19-ci-cp-0 crio[2848]: time="2025-02-05T12:49:16.586893697Z" level=info msg="Allowed annotations are specified for workload [io.containers.trace-syscall io.kubernetes.cri-o.Devices io.kubernetes.cri-o.LinkLogs]"
Feb 05 12:49:16 ocp-4-19-ci-cp-0 systemd[1]: Started crio-conmon-27b296756e49b6a4ae77d3c52d3d98960103b87338aa6a79d75bc21684df4fd7.scope.
Feb 05 12:49:16 ocp-4-19-ci-cp-0 systemd[1]: Started libcrun container.
Feb 05 12:49:16 ocp-4-19-ci-cp-0 systemd[1]: crio-27b296756e49b6a4ae77d3c52d3d98960103b87338aa6a79d75bc21684df4fd7.scope: Deactivated successfully.
Feb 05 12:49:16 ocp-4-19-ci-cp-0 systemd[1]: Stopped libcrun container.
Feb 05 12:49:16 ocp-4-19-ci-cp-0 conmon[89005]: conmon 27b296756e49b6a4ae77 <nwarn>: runtime stderr: cannot mkdir `proc`: Permission denied
Feb 05 12:49:16 ocp-4-19-ci-cp-0 conmon[89005]: conmon 27b296756e49b6a4ae77 <error>: Failed to create container: exit status 1
Feb 05 12:49:16 ocp-4-19-ci-cp-0 crio[2848]: time="2025-02-05T12:49:16.625255271Z" level=error msg="Container creation error: cannot mkdir `proc`: Permission denied\n" id=c9966e3a-5ed6-46c0-ab5c-d7004c2fcf2d name=/runtime.v1.RuntimeService/CreateContainer
Feb 05 12:49:16 ocp-4-19-ci-cp-0 systemd[1]: crio-conmon-27b296756e49b6a4ae77d3c52d3d98960103b87338aa6a79d75bc21684df4fd7.scope: Deactivated successfully. 

    Container starts normally and the Pod reaches Running state.