-
Bug
-
Resolution: Done-Errata
-
Critical
-
None
-
4.18.z
-
None
-
Quality / Stability / Reliability
-
False
-
-
3
-
Critical
-
None
-
None
-
None
-
None
-
Customer Escalated, Customer Facing
-
In Progress
-
Bug Fix
-
-
None
-
None
-
None
-
None
Description of problem:
The 4.18 RHEL worker scaleup job failed for containers couldn't be started with the following error: <ip-10-0-104-104.us-east-2.compute.internal> (1, b'\n{"changed": true, "stdout": "", "stderr": "time=\\"2025-07-25T03:07:58Z\\" level=warning msg=\\"Using cgroups-v1 which is deprecated in favor of cgroups-v2 with Podman v5 and will be removed in a future version. Set environment variable `PODMAN_IGNORE_CGROUPSV1_WARNING` to hide this warning.\\"\\nError: crun: write file `devices.allow`: Operation not permitted: OCI permission denied", "rc": 126, "cmd": ["podman", "run", "--rm", "registry.build11.ci.openshift.org/ci-op-gk87glxb/release@sha256:a85dfcb127052276dad4bbceddd43e4323847a9b498786479b1e90e9c0f31e60", "image", "machine-config-operator"], "start": "2025-07-25 03:07:58.944312", "end": "2025-07-25 03:07:59.376964", "delta": "0:00:00.432652", "failed": true, "msg": "non-zero return code", "invocation": {"module_args": {"_raw_params": "podman run --rm registry.build11.ci.openshift.org/ci-op-gk87glxb/release@sha256:a85dfcb127052276dad4bbceddd43e4323847a9b498786479b1e90e9c0f31e60 image machine-config-operator", "_uses_shell": false, "expand_argument_vars": true, "stdin_add_newline": true, "strip_empty_ends": true, "argv": null, "chdir": null, "executable": null, "creates": null, "removes": null, "stdin": null}}}\n', b'') <ip-10-0-104-104.us-east-2.compute.internal> Failed to connect to the host via ssh: fatal: [ip-10-0-104-104.us-east-2.compute.internal]: FAILED! => { "changed": true, "cmd": [ "podman", "run", "--rm", "registry.build11.ci.openshift.org/ci-op-gk87glxb/release@sha256:a85dfcb127052276dad4bbceddd43e4323847a9b498786479b1e90e9c0f31e60", "image", "machine-config-operator" ], "delta": "0:00:00.432652", "end": "2025-07-25 03:07:59.376964", "invocation": { "module_args": { "_raw_params": "podman run --rm registry.build11.ci.openshift.org/ci-op-gk87glxb/release@sha256:a85dfcb127052276dad4bbceddd43e4323847a9b498786479b1e90e9c0f31e60 image machine-config-operator", "_uses_shell": false, "argv": null, "chdir": null, "creates": null, "executable": null, "expand_argument_vars": true, "removes": null, "stdin": null, "stdin_add_newline": true, "strip_empty_ends": true } }, "msg": "non-zero return code", "rc": 126, "start": "2025-07-25 03:07:58.944312", "stderr": "time=\"2025-07-25T03:07:58Z\" level=warning msg=\"Using cgroups-v1 which is deprecated in favor of cgroups-v2 with Podman v5 and will be removed in a future version. Set environment variable `PODMAN_IGNORE_CGROUPSV1_WARNING` to hide this warning.\"\nError: crun: write file `devices.allow`: Operation not permitted: OCI permission denied", "stderr_lines": [ "time=\"2025-07-25T03:07:58Z\" level=warning msg=\"Using cgroups-v1 which is deprecated in favor of cgroups-v2 with Podman v5 and will be removed in a future version. Set environment variable `PODMAN_IGNORE_CGROUPSV1_WARNING` to hide this warning.\"", "Error: crun: write file `devices.allow`: Operation not permitted: OCI permission denied" ], "stdout": "", "stdout_lines": [] } Example failing job: https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.18-e2e-aws-ovn-workers-rhel8/1948566628195110912
Version-Release number of selected component (if applicable):
registry.ci.openshift.org/ocp/release:4.18.0-0.nightly-2025-07-24-004716 The container related rpm packages on the RHEl worker: podman-plugins-5.2.2-1.rhaos4.18.el8.x86_64 podman-catatonit-5.2.2-1.rhaos4.18.el8.x86_64 podman-gvproxy-5.2.2-1.rhaos4.18.el8.x86_64 podman-5.2.2-1.rhaos4.18.el8.x86_64 cri-o-1.31.10-4.rhaos4.18.git3a7cdab.el8.x86_64 conmon-2.1.12-2.rhaos4.18.el8.x86_64
How reproducible:
Always
Steps to Reproduce:
1. Add RHEL worker to 4.18 cluster 2. 3.
Actual results:
Playbook failed to execute
Expected results:
Playbook run successfully
Additional info:
Also tried with installing podman-4.x packages on RHEL node in job https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/openshift_openshift-ansible/12532/pull-ci-openshift-openshift-ansible-release-4.18-e2e-aws-workers-rhel8/1949007588276310016/artifacts/e2e-aws-workers-rhel8/workers-rhel-scaleup/build-log.txt, it failed with the same error, only without the cgroups-v1 vs. Podman v5 warning.
- links to
-
RHBA-2025:13325 OpenShift Container Platform 4.18.22 bug fix update