In OCP 4.18, the ExecPre command to start the kubelet.service has changed. It now scans the whole /var/lib/kubelet filesystem.
ExecStartPre=-/usr/sbin/restorecon -rv /var/lib/kubelet/ /usr/local/bin/kubenswrapper /usr/bin/kubensenter
Used to be this in the previous versions:
4.16
ExecStartPre=-/usr/sbin/restorecon /usr/local/bin/kubenswrapper /usr/bin/kubensenter
4.17
ExecStartPre=-/usr/sbin/restorecon /usr/local/bin/kubenswrapper /usr/bin/kubensenter
The restorecon fails at /var/lib/kubelet, cause is still unknown. But with the failure of restorecon, it results in kubelet.service not able to start. Reboot doesn't help.
=================
Status from kubelet,
[core@ocppoc4 ~]$ sudo systemctl status kubelet
● kubelet.service - Kubernetes Kubelet
Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; preset: disabled)
Drop-In: /etc/systemd/system/kubelet.service.d
└─01-kubens.conf, 10-mco-default-env.conf, 10-mco-default-madv.conf, 10-mco-on-prem-wait-resolv.conf, 20-logging.conf, 20-nodenet.conf
Active: activating (start-pre) since Tue 2025-03-25 17:09:09 UTC; 1min 0s ago
Process: 4119 ExecCondition=/bin/bash -c [ -f /run/resolv-prepender-kni-conf-done ] ||
(code=exited, status=0/SUCCESS)
Process: 4120 ExecStartPre=/bin/mkdir --parents /etc/kubernetes/manifests (code=exited, status=0/SUCCESS)
Cntrl PID: 4121 (restorecon)
Tasks: 1 (limit: 1649287)
Memory: 255.7M
CPU: 1min 423ms
CGroup: /system.slice/kubelet.service
└─4121 /usr/sbin/restorecon -rv /var/lib/kubelet/ /usr/local/bin/kubenswrapper /usr/bin/kubensenter
Mar 25 17:09:09 ocppoc4 restorecon[4121]: /var/lib/kubelet/pods/66b3f206ba2b9bc0e52817b244a5a71a/containers/kube-rbac-proxy-crio/82f7a88c not reset as customized by admin to system_u:object_r:conta>
Mar 25 17:09:09 ocppoc4 restorecon[4121]: /var/lib/kubelet/pods/426f2a4b-ebf6-4d8e-93b2-70cbb52a5691/etc-hosts not reset as customized by admin to system_u:object_r:container_file_t:s0:c24,c27
======================
Journal log errors:
Mar 27 09:58:19 ocppoc4 systemd[1]: Starting Kubernetes Kubelet...
Mar 27 09:59:49 ocppoc4 systemd[1]: kubelet.service: start-pre operation timed out. Terminating. <<<<<<
Mar 27 09:59:49 ocppoc4 systemd[1]: kubelet.service: Failed with result 'timeout'.
Mar 27 09:59:49 ocppoc4 systemd[1]: Failed to start Kubernetes Kubelet.
Mar 27 09:59:49 ocppoc4 systemd[1]: kubelet.service: Consumed 1min 18.348s CPU time.
Mar 27 09:59:59 ocppoc4 systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 1.
Mar 27 09:59:59 ocppoc4 systemd[1]: Stopped Kubernetes Kubelet.
- clones
-
OCPBUGS-54384 Restorecon failure in OCP 4.18, causing kubelet to not start
-
- Closed
-
- depends on
-
OCPBUGS-54384 Restorecon failure in OCP 4.18, causing kubelet to not start
-
- Closed
-
- links to
-
RHSA-2025:4712 OpenShift Container Platform 4.18.13 bug fix and security update