Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-54765

[4.18]Restorecon failure in OCP 4.18, causing kubelet to not start

XMLWordPrintable

    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • Moderate
    • None
    • None
    • None
    • Done
    • Bug Fix
    • Hide
      Currently, the Kubelet service failures due to an incorrect `restorecon` command scope affecting /var/lib/kubelet. This prevents container deployment and management. with this release, the 'ExecStartPre' command was modified to to exclude /var/lib/kubelet/ and only scan pod-resources directory. This change restored file permissions, enabling effective kubelet service start. (link:https://issues.redhat.com/browse/OCPBUGS-54765[OCPBUGS-54765])
      Show
      Currently, the Kubelet service failures due to an incorrect `restorecon` command scope affecting /var/lib/kubelet. This prevents container deployment and management. with this release, the 'ExecStartPre' command was modified to to exclude /var/lib/kubelet/ and only scan pod-resources directory. This change restored file permissions, enabling effective kubelet service start. (link: https://issues.redhat.com/browse/OCPBUGS-54765 [ OCPBUGS-54765 ])
    • None
    • None
    • None
    • None

      In OCP 4.18, the ExecPre command to start the kubelet.service has changed. It now scans the whole /var/lib/kubelet filesystem.

      ExecStartPre=-/usr/sbin/restorecon -rv /var/lib/kubelet/ /usr/local/bin/kubenswrapper /usr/bin/kubensenter

      Used to be this in the previous versions:
      4.16
      ExecStartPre=-/usr/sbin/restorecon /usr/local/bin/kubenswrapper /usr/bin/kubensenter

      4.17
      ExecStartPre=-/usr/sbin/restorecon /usr/local/bin/kubenswrapper /usr/bin/kubensenter

      The restorecon fails at /var/lib/kubelet, cause is still unknown. But with the failure of restorecon, it results in kubelet.service not able to start. Reboot doesn't help.

      =================
      Status from kubelet,
      [core@ocppoc4 ~]$ sudo systemctl status kubelet
      ● kubelet.service - Kubernetes Kubelet
      Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; preset: disabled)
      Drop-In: /etc/systemd/system/kubelet.service.d
      └─01-kubens.conf, 10-mco-default-env.conf, 10-mco-default-madv.conf, 10-mco-on-prem-wait-resolv.conf, 20-logging.conf, 20-nodenet.conf
      Active: activating (start-pre) since Tue 2025-03-25 17:09:09 UTC; 1min 0s ago
      Process: 4119 ExecCondition=/bin/bash -c [ -f /run/resolv-prepender-kni-conf-done ] ||

      { echo "NM resolv-prepender failed"; exit 255; }

      (code=exited, status=0/SUCCESS)
      Process: 4120 ExecStartPre=/bin/mkdir --parents /etc/kubernetes/manifests (code=exited, status=0/SUCCESS)
      Cntrl PID: 4121 (restorecon)
      Tasks: 1 (limit: 1649287)
      Memory: 255.7M
      CPU: 1min 423ms
      CGroup: /system.slice/kubelet.service
      └─4121 /usr/sbin/restorecon -rv /var/lib/kubelet/ /usr/local/bin/kubenswrapper /usr/bin/kubensenter

      Mar 25 17:09:09 ocppoc4 restorecon[4121]: /var/lib/kubelet/pods/66b3f206ba2b9bc0e52817b244a5a71a/containers/kube-rbac-proxy-crio/82f7a88c not reset as customized by admin to system_u:object_r:conta>
      Mar 25 17:09:09 ocppoc4 restorecon[4121]: /var/lib/kubelet/pods/426f2a4b-ebf6-4d8e-93b2-70cbb52a5691/etc-hosts not reset as customized by admin to system_u:object_r:container_file_t:s0:c24,c27

      ======================

      Journal log errors:
      Mar 27 09:58:19 ocppoc4 systemd[1]: Starting Kubernetes Kubelet...
      Mar 27 09:59:49 ocppoc4 systemd[1]: kubelet.service: start-pre operation timed out. Terminating. <<<<<<
      Mar 27 09:59:49 ocppoc4 systemd[1]: kubelet.service: Failed with result 'timeout'.
      Mar 27 09:59:49 ocppoc4 systemd[1]: Failed to start Kubernetes Kubelet.
      Mar 27 09:59:49 ocppoc4 systemd[1]: kubelet.service: Consumed 1min 18.348s CPU time.
      Mar 27 09:59:59 ocppoc4 systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 1.
      Mar 27 09:59:59 ocppoc4 systemd[1]: Stopped Kubernetes Kubelet.

              titzhak Talor Itzhak
              rhn-support-nestoros Nikkie Estorosos
              None
              None
              Cameron Meadors Cameron Meadors
              None
              Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

                Created:
                Updated:
                Resolved: