-
Bug
-
Resolution: Won't Do
-
Major
-
None
-
4.12.z
-
Quality / Stability / Reliability
-
False
-
-
None
-
Important
-
No
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
We observed multiple cases where kubelet is not removing rotated files of previously failed containers when pod is still running.
When checking for /var/log/pods/{pod}/{container} directory of running container I see next picture:
```
[root@prd-we-datamgt01a-xxxxxxx/]# ls -lh /var/log/pods/openshift-network-observability_flowlogs-pipeline-fzd9g_xxxxxxxxx/flowlogs-pipeline/
total 995M
rw------. 1 root root 12M Jan 26 2023 0.log
rw-rr-. 1 root root 3.2M Jan 23 2023 0.log.20230121-121126.gz
rw-rr-. 1 root root 3.2M Jan 25 2023 0.log.20230123-193016.gz
rw-rr-. 1 root root 3.2M Jan 26 2023 0.log.20230125-185613.gz
rw------. 1 root root 51M Jan 26 2023 0.log.20230126-180202
rw------. 1 root root 21M Feb 22 2023 1.log
rw------. 1 root root 51M Feb 1 2023 1.log.20230201-141918
rw------. 1 root root 3.0M Aug 9 09:04 10.log
rw------. 1 root root 38M Nov 2 07:01 12.log
rw-rr-. 1 root root 3.2M Oct 27 08:49 12.log.20231024-180210.gz
rw-rr-. 1 root root 3.0M Oct 29 19:37 12.log.20231027-084939.gz
rw-rr-. 1 root root 3.0M Oct 31 22:48 12.log.20231029-193751.gz
rw------. 1 root root 51M Oct 31 22:48 12.log.20231031-224858
rw------. 1 root root 30M Nov 27 13:15 13.log
log size rentention can be configured through Kubelet config through containerLogMaxSize (default: 50Mi) & containerLogMaxFiles (default: 5) but we just use default values we we would expect to have 5 files maximum and old files for crashed containers to be removed but file from containers dead several months ago are still there and taking some space.
I saw some related issue than normally fixed this behavior (https://github.com/kubernetes/kubernetes/pull/93333) and for a basic pod crashing, cleanup is working fine but it seems this fix is not covering all the cases as showed above
OCP version is 4.12.36
But from the kubelet.conf file not able to see the `containerLogMaxFiles` parameter.
sh-4.4# grep "containerLog" /etc/kubernetes/kubelet.conf
"containerLogMaxSize": "50Mi",