Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-59843

Can't start container on RHEL-8 worker due to "Error: crun: write file `devices.allow`: Operation not permitted: OCI permission denied"

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done-Errata
    • Icon: Critical Critical
    • None
    • 4.18.z
    • None
    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • 3
    • Critical
    • None
    • None
    • None
    • None
    • Customer Escalated, Customer Facing
    • In Progress
    • Bug Fix
    • Hide
      * Before this update, worker scale-up jobs that used Podman v5 with the `container-tools` module for `netavark` failed due to denial of the Open Container Initiative (OCI) permission when writing `devices.allow` with `crun`. As a consequence, the container scaling jobs failed. With this release, the `netavark` dependency for Podman is disabled, and `runc` runtime is used instead. As a result, worker scale-up jobs are now successful. (link:https://issues.redhat.com/browse/OCPBUGS-59843[OCPBUGS-59843])
      Show
      * Before this update, worker scale-up jobs that used Podman v5 with the `container-tools` module for `netavark` failed due to denial of the Open Container Initiative (OCI) permission when writing `devices.allow` with `crun`. As a consequence, the container scaling jobs failed. With this release, the `netavark` dependency for Podman is disabled, and `runc` runtime is used instead. As a result, worker scale-up jobs are now successful. (link: https://issues.redhat.com/browse/OCPBUGS-59843 [ OCPBUGS-59843 ])
    • None
    • None
    • None
    • None

      Description of problem:

      The 4.18 RHEL worker scaleup job failed for containers couldn't be started with the following error:
      
      <ip-10-0-104-104.us-east-2.compute.internal> (1, b'\n{"changed": true, "stdout": "", "stderr": "time=\\"2025-07-25T03:07:58Z\\" level=warning msg=\\"Using cgroups-v1 which is deprecated in favor of cgroups-v2 with Podman v5 and will be removed in a future version. Set environment variable `PODMAN_IGNORE_CGROUPSV1_WARNING` to hide this warning.\\"\\nError: crun: write file `devices.allow`: Operation not permitted: OCI permission denied", "rc": 126, "cmd": ["podman", "run", "--rm", "registry.build11.ci.openshift.org/ci-op-gk87glxb/release@sha256:a85dfcb127052276dad4bbceddd43e4323847a9b498786479b1e90e9c0f31e60", "image", "machine-config-operator"], "start": "2025-07-25 03:07:58.944312", "end": "2025-07-25 03:07:59.376964", "delta": "0:00:00.432652", "failed": true, "msg": "non-zero return code", "invocation": {"module_args": {"_raw_params": "podman run --rm registry.build11.ci.openshift.org/ci-op-gk87glxb/release@sha256:a85dfcb127052276dad4bbceddd43e4323847a9b498786479b1e90e9c0f31e60 image machine-config-operator", "_uses_shell": false, "expand_argument_vars": true, "stdin_add_newline": true, "strip_empty_ends": true, "argv": null, "chdir": null, "executable": null, "creates": null, "removes": null, "stdin": null}}}\n', b'')
      <ip-10-0-104-104.us-east-2.compute.internal> Failed to connect to the host via ssh: 
      fatal: [ip-10-0-104-104.us-east-2.compute.internal]: FAILED! => {
          "changed": true,
          "cmd": [
              "podman",
              "run",
              "--rm",
              "registry.build11.ci.openshift.org/ci-op-gk87glxb/release@sha256:a85dfcb127052276dad4bbceddd43e4323847a9b498786479b1e90e9c0f31e60",
              "image",
              "machine-config-operator"
          ],
          "delta": "0:00:00.432652",
          "end": "2025-07-25 03:07:59.376964",
          "invocation": {
              "module_args": {
                  "_raw_params": "podman run --rm registry.build11.ci.openshift.org/ci-op-gk87glxb/release@sha256:a85dfcb127052276dad4bbceddd43e4323847a9b498786479b1e90e9c0f31e60 image machine-config-operator",
                  "_uses_shell": false,
                  "argv": null,
                  "chdir": null,
                  "creates": null,
                  "executable": null,
                  "expand_argument_vars": true,
                  "removes": null,
                  "stdin": null,
                  "stdin_add_newline": true,
                  "strip_empty_ends": true
              }
          },
          "msg": "non-zero return code",
          "rc": 126,
          "start": "2025-07-25 03:07:58.944312",
          "stderr": "time=\"2025-07-25T03:07:58Z\" level=warning msg=\"Using cgroups-v1 which is deprecated in favor of cgroups-v2 with Podman v5 and will be removed in a future version. Set environment variable `PODMAN_IGNORE_CGROUPSV1_WARNING` to hide this warning.\"\nError: crun: write file `devices.allow`: Operation not permitted: OCI permission denied",
          "stderr_lines": [
              "time=\"2025-07-25T03:07:58Z\" level=warning msg=\"Using cgroups-v1 which is deprecated in favor of cgroups-v2 with Podman v5 and will be removed in a future version. Set environment variable `PODMAN_IGNORE_CGROUPSV1_WARNING` to hide this warning.\"",
              "Error: crun: write file `devices.allow`: Operation not permitted: OCI permission denied"
          ],
          "stdout": "",
          "stdout_lines": []
      }
      
      
      Example failing job: https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.18-e2e-aws-ovn-workers-rhel8/1948566628195110912
       

      Version-Release number of selected component (if applicable):

      registry.ci.openshift.org/ocp/release:4.18.0-0.nightly-2025-07-24-004716
      
      The container related rpm packages on the RHEl worker:
      podman-plugins-5.2.2-1.rhaos4.18.el8.x86_64
      podman-catatonit-5.2.2-1.rhaos4.18.el8.x86_64
      podman-gvproxy-5.2.2-1.rhaos4.18.el8.x86_64
      podman-5.2.2-1.rhaos4.18.el8.x86_64
      cri-o-1.31.10-4.rhaos4.18.git3a7cdab.el8.x86_64
      conmon-2.1.12-2.rhaos4.18.el8.x86_64    

      How reproducible:

      Always    

      Steps to Reproduce:

          1. Add RHEL worker to 4.18 cluster
          2.
          3.
          

      Actual results:

      Playbook failed to execute

      Expected results:

      Playbook run successfully 

      Additional info:

      Also tried with installing podman-4.x packages on RHEL node in job https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/openshift_openshift-ansible/12532/pull-ci-openshift-openshift-ansible-release-4.18-e2e-aws-workers-rhel8/1949007588276310016/artifacts/e2e-aws-workers-rhel8/workers-rhel-scaleup/build-log.txt,
      it failed with the same error, only without the cgroups-v1 vs. Podman v5 warning. 

              Unassigned Unassigned
              rh-ee-gpei Gaoyun Pei
              None
              None
              Gaoyun Pei Gaoyun Pei
              None
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

                Created:
                Updated:
                Resolved: