Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-27051

Old logs of container and pods are not getting deleted automatically if a container get restart

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Won't Do
    • Icon: Major Major
    • None
    • 4.12.z
    • Node / Kubelet
    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • Important
    • No
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      We observed multiple cases where kubelet is not removing rotated files of previously failed containers when pod is still running.

      When checking for /var/log/pods/{pod}/{container} directory of running container I see next picture:
      ```
      [root@prd-we-datamgt01a-xxxxxxx/]# ls -lh /var/log/pods/openshift-network-observability_flowlogs-pipeline-fzd9g_xxxxxxxxx/flowlogs-pipeline/
      total 995M
      rw------. 1 root root  12M Jan 26  2023 0.log
      rw-rr-. 1 root root 3.2M Jan 23  2023 0.log.20230121-121126.gz
      rw-rr-. 1 root root 3.2M Jan 25  2023 0.log.20230123-193016.gz
      rw-rr-. 1 root root 3.2M Jan 26  2023 0.log.20230125-185613.gz
      rw------. 1 root root  51M Jan 26  2023 0.log.20230126-180202
      rw------. 1 root root  21M Feb 22  2023 1.log
      rw------. 1 root root  51M Feb  1  2023 1.log.20230201-141918
      rw------. 1 root root 3.0M Aug  9 09:04 10.log
      rw------. 1 root root  38M Nov  2 07:01 12.log
      rw-rr-. 1 root root 3.2M Oct 27 08:49 12.log.20231024-180210.gz
      rw-rr-. 1 root root 3.0M Oct 29 19:37 12.log.20231027-084939.gz
      rw-rr-. 1 root root 3.0M Oct 31 22:48 12.log.20231029-193751.gz
      rw------. 1 root root  51M Oct 31 22:48 12.log.20231031-224858
      rw------. 1 root root  30M Nov 27 13:15 13.log

      log size rentention can be configured through Kubelet config through containerLogMaxSize (default: 50Mi) & containerLogMaxFiles (default: 5) but we just use default values we we would expect to have 5 files maximum and old files for crashed containers to be removed but file from containers dead several months ago are still there and taking some space.

      I saw some related issue than normally fixed this behavior (https://github.com/kubernetes/kubernetes/pull/93333) and for a basic pod crashing, cleanup is working fine but it seems this fix is not covering all the cases as showed above

      OCP version is 4.12.36

      But from the kubelet.conf file not able to see the `containerLogMaxFiles` parameter.
      sh-4.4# grep "containerLog" /etc/kubernetes/kubelet.conf
        "containerLogMaxSize": "50Mi",

              rh-ee-kehannon Kevin Hannon
              rhn-support-vsolanki Vimal Solanki
              None
              None
              Sunil Choudhary Sunil Choudhary
              None
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated:
                Resolved: