-
Bug
-
Resolution: Done
-
Normal
-
None
-
4.12.z, 4.11.z, 4.10.z, 4.8.z
-
Quality / Stability / Reliability
-
False
-
-
None
-
None
-
No
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Description of problem:
- During the time of upgrading the cluster, Nodes failed to pull images with errors like following, which blocking the OCP upgrade:
~~~
error creating read-write layer with ID "4372a0c382584d7752da058c5267d1d652d727585457a71c5ef3d4d17a951719": Stat /var/lib/containers/storage/overlay/27abd31f77c1e21b8897140edb61d1e52d48a3cd287c03796dbecb54684871d3: no such file or directory
~~~
- Cu use below method to clean the nodes images storage but it is an temporary fix
~~~
$ sudo -i
- systemctl disable kubelet
- systemctl disable crio
- reboot
$ sudo -i
- rm -rf /var/lib/containers/*
- crio wipe -f
- systemctl enable --now crio
- systemctl enable --now kubelet
~~~
The above shared steps is a temprary fix, they always need to perform the above steps before upgrading cluster and looking for permanent fix.
Cu face this issue when upgrade from 4.8 to 4.10 and also during upgrading from 4.10 to 4.12
Note: Openinig this bug after discussion with Peter Hunt on "https://issues.redhat.com/browse/OCPBUGS-8939"