The 4.18 RHEL worker scaleup job failed for containers couldn't be started with the following error:

<ip-10-0-104-104.us-east-2.compute.internal> (1, b'\n{"changed": true, "stdout": "", "stderr": "time=\\"2025-07-25T03:07:58Z\\" level=warning msg=\\"Using cgroups-v1 which is deprecated in favor of cgroups-v2 with Podman v5 and will be removed in a future version. Set environment variable `PODMAN_IGNORE_CGROUPSV1_WARNING` to hide this warning.\\"\\nError: crun: write file `devices.allow`: Operation not permitted: OCI permission denied", "rc": 126, "cmd": ["podman", "run", "--rm", "registry.build11.ci.openshift.org/ci-op-gk87glxb/release@sha256:a85dfcb127052276dad4bbceddd43e4323847a9b498786479b1e90e9c0f31e60", "image", "machine-config-operator"], "start": "2025-07-25 03:07:58.944312", "end": "2025-07-25 03:07:59.376964", "delta": "0:00:00.432652", "failed": true, "msg": "non-zero return code", "invocation": {"module_args": {"_raw_params": "podman run --rm registry.build11.ci.openshift.org/ci-op-gk87glxb/release@sha256:a85dfcb127052276dad4bbceddd43e4323847a9b498786479b1e90e9c0f31e60 image machine-config-operator", "_uses_shell": false, "expand_argument_vars": true, "stdin_add_newline": true, "strip_empty_ends": true, "argv": null, "chdir": null, "executable": null, "creates": null, "removes": null, "stdin": null}}}\n', b'')
<ip-10-0-104-104.us-east-2.compute.internal> Failed to connect to the host via ssh: 
fatal: [ip-10-0-104-104.us-east-2.compute.internal]: FAILED! => {
    "changed": true,
    "cmd": [
        "podman",
        "run",
        "--rm",
        "registry.build11.ci.openshift.org/ci-op-gk87glxb/release@sha256:a85dfcb127052276dad4bbceddd43e4323847a9b498786479b1e90e9c0f31e60",
        "image",
        "machine-config-operator"
    ],
    "delta": "0:00:00.432652",
    "end": "2025-07-25 03:07:59.376964",
    "invocation": {
        "module_args": {
            "_raw_params": "podman run --rm registry.build11.ci.openshift.org/ci-op-gk87glxb/release@sha256:a85dfcb127052276dad4bbceddd43e4323847a9b498786479b1e90e9c0f31e60 image machine-config-operator",
            "_uses_shell": false,
            "argv": null,
            "chdir": null,
            "creates": null,
            "executable": null,
            "expand_argument_vars": true,
            "removes": null,
            "stdin": null,
            "stdin_add_newline": true,
            "strip_empty_ends": true
        }
    },
    "msg": "non-zero return code",
    "rc": 126,
    "start": "2025-07-25 03:07:58.944312",
    "stderr": "time=\"2025-07-25T03:07:58Z\" level=warning msg=\"Using cgroups-v1 which is deprecated in favor of cgroups-v2 with Podman v5 and will be removed in a future version. Set environment variable `PODMAN_IGNORE_CGROUPSV1_WARNING` to hide this warning.\"\nError: crun: write file `devices.allow`: Operation not permitted: OCI permission denied",
    "stderr_lines": [
        "time=\"2025-07-25T03:07:58Z\" level=warning msg=\"Using cgroups-v1 which is deprecated in favor of cgroups-v2 with Podman v5 and will be removed in a future version. Set environment variable `PODMAN_IGNORE_CGROUPSV1_WARNING` to hide this warning.\"",
        "Error: crun: write file `devices.allow`: Operation not permitted: OCI permission denied"
    ],
    "stdout": "",
    "stdout_lines": []
}


Example failing job: https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.18-e2e-aws-ovn-workers-rhel8/1948566628195110912
 

registry.ci.openshift.org/ocp/release:4.18.0-0.nightly-2025-07-24-004716

The container related rpm packages on the RHEl worker:
podman-plugins-5.2.2-1.rhaos4.18.el8.x86_64
podman-catatonit-5.2.2-1.rhaos4.18.el8.x86_64
podman-gvproxy-5.2.2-1.rhaos4.18.el8.x86_64
podman-5.2.2-1.rhaos4.18.el8.x86_64
cri-o-1.31.10-4.rhaos4.18.git3a7cdab.el8.x86_64
conmon-2.1.12-2.rhaos4.18.el8.x86_64    

Always    

    1. Add RHEL worker to 4.18 cluster
    2.
    3.
    

Playbook failed to execute

Playbook run successfully 

Also tried with installing podman-4.x packages on RHEL node in job https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/openshift_openshift-ansible/12532/pull-ci-openshift-openshift-ansible-release-4.18-e2e-aws-workers-rhel8/1949007588276310016/artifacts/e2e-aws-workers-rhel8/workers-rhel-scaleup/build-log.txt,
it failed with the same error, only without the cgroups-v1 vs. Podman v5 warning.