-
Bug
-
Resolution: Won't Do
-
Normal
-
None
-
4.12.z
-
Moderate
-
No
-
False
-
Description of problem:
Raising this issue to track the potential Kubelet backport of this fix: https://github.com/cri-o/cri-o/commit/41b13e28dd75828ab9cc24806ad4d2d3b4a605b9
i.e. due to the above, in some scenarios we see the Kubelet attempting to garbage-collect images that are already in use.
Sample output of the Kubelet's garbage collector:
Feb 08 20:10:33 node.example.com kubenswrapper[1919]: E0208 20:10:33.420932 1919 kubelet.go:1333] "Image garbage collection failed once. Stats initialization may not have completed yet" err="wanted to free 6220071731 bytes, but freed 2494995641 bytes space with errors in image deletion: [rpc error: code = Unknown desc = image used by 0681759a443b6da954e0a1ab31275c91b0562c2a7817a3abf541d4c4d38ef8ab: image is in use by a container, rpc error: code = Unknown desc = image used by 404779adc43e175342be99b19e6a828b82e30f9a547e9b52aa8ddfa434d1d06d: image is in use by a container [...]
$ cat /host_service_logs/masters/kubelet_service.log | egrep "image_manager.go|garbage" | grep -c "garbage collection succeeded" 9 $ cat ./host_service_logs/masters/kubelet_service.log | egrep "image_manager.go|garbage" | grep -c "garbage collection failed" 380
Version-Release number of selected component (if applicable):
4.12.46
Actual results:
Kubelet attempts to garbage collect images that are in use
Expected results:
Kubelet to garbage collect only images that are not being utilized
