-
Bug
-
Resolution: Done
-
Major
-
4.18
-
Quality / Stability / Reliability
-
False
-
-
None
-
Important
-
None
-
None
-
None
-
OCP Node Sprint 272 (Green), OCP Node Sprint 273 (Green), OCP Node Sprint 274 (green), OCP Node Sprint 275 (green), OCP Node Sprint 276 (green)
-
5
-
Done
-
Release Note Not Required
-
N/A
Nodes in the cluster started indicating DiskPressure.
Draining/rebooting the nodes did not help.
Following cri-o storage clean up procedure worked.
The sosreport indicates that when the nodes started having DiskPressure GarbageCollection started but encountered a fatal error.
~~~
May 24 08:08:22 worker14.wdp-perf-scale.cp.fyre.ibm.com kubenswrapper[2695232]: I0524 08:08:22.152327 2695232 image_gc_manager.go:383] "Disk usage on image filesystem is over the high threshold, trying to free bytes down to the low threshold" usage=88 highThreshold=85 amountToFree=19975571865 lowThreshold=80
May 24 08:08:22 worker14.wdp-perf-scale.cp.fyre.ibm.com kubenswrapper[2695232]: I0524 08:08:22.152405 2695232 image_gc_manager.go:487] "Removing image to free bytes" imageID="bd2fb45ae0cb2e8c4001f84c513f907f7960cdac97e01fdb7214688e4288efa2" size=826271787 runtimeHandler=""
May 24 08:08:22 worker14.wdp-perf-scale.cp.fyre.ibm.com kubenswrapper[2695232]: I0524 08:08:22.498929 2695232 image_gc_manager.go:487] "Removing image to free bytes" imageID="76f6b69fced95c010be36bffeeb722ce144b905c267c99248f79c9defa65a962" size=227488986 runtimeHandler=""
May 24 08:08:22 worker14.wdp-perf-scale.cp.fyre.ibm.com kubenswrapper[2695232]: E0524 08:08:22.499604 2695232 log.go:32] "RemoveImage from image service failed" err="rpc error: code = Unknown desc = delete image: image used by e4c5e303adf168384679d6c1d967349d8086ff2ffbf4bf21a3e60b5cbac5ceda: image is in use by a container" image="bd2fb45ae0cb2e8c4001f84c513f907f7960cdac97e01fdb7214688e4288efa2"
May 24 08:08:22 worker14.wdp-perf-scale.cp.fyre.ibm.com kubenswrapper[2695232]: E0524 08:08:22.499668 2695232 kuberuntime_image.go:150] "Failed to remove image" err="rpc error: code = Unknown desc = delete image: image used by e4c5e303adf168384679d6c1d967349d8086ff2ffbf4bf21a3e60b5cbac5ceda: image is in use by a container" image="bd2fb45ae0cb2e8c4001f84c513f907f7960cdac97e01fdb7214688e4288efa2"
May 24 08:08:22 worker14.wdp-perf-scale.cp.fyre.ibm.com kubenswrapper[2695232]: I0524 08:08:22.499703 2695232 image_gc_manager.go:487] "Removing image to free bytes" imageID="76f6b69fced95c010be36bffeeb722ce144b905c267c99248f79c9defa65a962" size=227488986 runtimeHandler=""
May 24 08:08:22 worker14.wdp-perf-scale.cp.fyre.ibm.com kubenswrapper[2695232]: fatal error: concurrent map writes
May 24 08:08:22 worker14.wdp-perf-scale.cp.fyre.ibm.com kubenswrapper[2695232]: fatal error: concurrent map writes
May 24 08:08:22 worker14.wdp-perf-scale.cp.fyre.ibm.com kubenswrapper[2695232]: goroutine 262 [running]:
....
~~~
- depends on
-
OCPBUGS-60166 GarbageCollection encountered "fatal error: concurrent map writes" [4.19]
-
- Closed
-
- is cloned by
-
OCPBUGS-60165 GarbageCollection encountered "fatal error: concurrent map writes" [4.20]
-
- Closed
-
- links to