Description

RHCOS build 417.94.202511031637-0 (Nov 3, 2025) introduced a regression by switching from OpenShift-specific runc to generic RHEL runc, breaking CPU affinity on Real-Time kernels.

Root Cause

The RHCOS build accidentally changed:

• Working: runc-1.1.14-2.rhaos4.17.el9 (OpenShift-specific with RT patches)
• Broken: runc-1.2.4-2.el9 (Generic RHEL, missing .rhaos4.17 tag)

The generic RHEL runc package lacks the CPU affinity patches required for RT kernel support.

Impact

• Containers on RT-enabled nodes no longer respect CPU affinity settings
• Test "[sig-node] CPU Partitioning node validation" fails on GCP RT upgrade jobs
• This reintroduces the exact issue from OCPBUGS-34902 that was supposedly fixed in July 2024

Timeline

• Last working build: RHCOS 417.94.202510282022-0 (Oct 28, 2025)
• First broken build: RHCOS 417.94.202511031637-0 (Nov 3, 2025)
• Test failures started: Nov 4, 2025 08:00:10

Evidence

Failing job: https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-ci-4.17-upgrade-from-stable-4.16-e2e-gcp-ovn-rt-upgrade/1986290418081337344

RHCOS package comparison:

• Working (Oct 28): https://releases-rhcos--prod-pipeline.apps.int.prod-stable-spoke1-dc-iad2.itup.redhat.com/?release=417.94.202510282022-0&stream=prod%2Fstreams%2F4.17-9.4&arch=x86_64#417.94.202510282022-0
• Broken (Nov 3): https://releases-rhcos--prod-pipeline.apps.int.prod-stable-spoke1-dc-iad2.itup.redhat.com/?release=417.94.202511031637-0&stream=prod%2Fstreams%2F4.17-9.4&arch=x86_64#417.94.202511031637-0

System behavior:
RT node correctly configured via TuneD with isolated_cores=4-7, systemd.cpu_affinity=0,1,2,3. However, containers are not respecting these CPU affinity settings due to the runc regression.

Required Action

Revert to OpenShift-specific runc build (runc-1.1.14-2.rhaos4.17.el9) or port the RT CPU affinity patches to runc-1.2.4.

Related Bugs

• OCPBUGS-34902 - Original issue that was fixed in July 2024 but has now regressed