Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-10829

[Reliability]kube-apiserver's memory usage keep increasing to max 3GB in 7 days

    XMLWordPrintable

Details

    • Important
    • No
    • Approved
    • False
    • Hide

      None

      Show
      None

    Description

      Description of problem:

      In 7 day's reliability test, kube-apiserver's memory usage keep increasing. Max is over 3GB.
      In our 4.12 test result, the kube-apiserver's memory usage was stable around 1.7 GB and not keep increasing. 
      I'll redo the test on a 4.12.0 build to see if I can reproduce this issue.
      
      I'll do a longer than 7 days test to see how high the memory can grow up.
      
      About Reliability Test
      https://github.com/openshift/svt/tree/master/reliability-v2

      Version-Release number of selected component (if applicable):

      4.13.0-0.nightly-2023-03-14-053612

      How reproducible:

      Always

      Steps to Reproduce:

      1. Install an AWS cluster with m5.xlarge type
      2. Run reliability test for 7 days
      Reliability Test Configuration example:
      https://github.com/openshift/svt/tree/master/reliability-v2#groups-and-tasks-1
      Config used in this test:
      admin: 1 user
      dev-test: 15 users
      dev-prod: 1 user 
      3. Use dittybopper dashboard to monitor the kube-apiserver's memory usage
      

      Actual results:

      kube-apiserver's memory usage keep increasing. Max is over 3GB

      Expected results:

      kube-apiserver's memory usage should not keep increasing

      Additional info:

      Screenshots are uploaded to shared folder OCPBUGS-10829 - Google Drive

      413-kube-apiserver-memory.png
      413-api-performance-last2d.png - test was stopped on [2023-03-24 04:21:10 UTC]
      412-kube-apiserver-memory.png
      must-gather.local.525817950490593011.tar.gz - 4.13 cluster's must gather

      Attachments

        Issue Links

          Activity

            People

              rhn-support-qili Qiujie Li
              rhn-support-qili Qiujie Li
              Qiujie Li Qiujie Li
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: