Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-27817

High memory usage by Kube APIServer on HostedCluster upgrades

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major Major
    • None
    • 4.14, 4.15, 4.16
    • HyperShift
    • Important
    • No
    • Proposed
    • False
    • Hide

      None

      Show
      None
    • Hide
      Cause - Kube API Server pods are dying because of OOM when a Hosted Cluster, that has a large number of worker nodes, is being upgraded.
      Consequence - ROSA HCP clusters can’t be updated if the total number of worker nodes is over 51.
      Fix - Exposed GOMEMLIMIT and GOGC for kube-apiserver via annotations in HostedCluster.
      Result - ROSA HCP clusters with worker nodes over 51 can be upgraded now.
      Show
      Cause - Kube API Server pods are dying because of OOM when a Hosted Cluster, that has a large number of worker nodes, is being upgraded. Consequence - ROSA HCP clusters can’t be updated if the total number of worker nodes is over 51. Fix - Exposed GOMEMLIMIT and GOGC for kube-apiserver via annotations in HostedCluster. Result - ROSA HCP clusters with worker nodes over 51 can be upgraded now.

      Description of problem:

      When performing upgrades on ROSA HCP clusters with a large number of worker nodes (> 51), the Kube APIServer pods of the cluster use up memory exceeding the capacity of their nodes, resulting in OOMKills.

      Version-Release number of selected component (if applicable):

         4.14, 4.15

      How reproducible:

          always

      Steps to Reproduce:

          1. Create ROSA HCP Cluster
    2. Add 100 workers to Cluster
    3. Upgrade the cluster

      Actual results:

          Kube APIServer pods are OOMKilled

      Expected results:

          Upgrade completes successfully

      Additional info:

              agarcial@redhat.com Alberto Garcia Lamela
              cewong@redhat.com Cesar Wong
              He Liu He Liu
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Created:
                Updated:
                Resolved: