Uploaded image for project: 'OpenShift Request For Enhancement'
  1. OpenShift Request For Enhancement
  2. RFE-2256

Customizing --max-requests-inflight and --max-mutating-requests-inflight on k8s API server

XMLWordPrintable

    • Icon: Feature Request Feature Request
    • Resolution: Done
    • Icon: Normal Normal
    • None
    • None
    • openshift-apiserver
    • False
    • False
    • Undefined

      Modification of specific configuration under  the kube-apiserver CM needs to be done under unsupportedConfigOverrides: 

      $  oc -n openshift-kube-apiserver get configmap config --output 'go-template={{index .data "config.yaml"'}}

       

      There is no supported way to modify --max-requests-inflight and --max-mutating-requests-inflight , so it would be then under unsupported configurations for now: 

       

      $ oc explain --api-version=operator.openshift.io/v1 kubeapiservers.spec

      $ oc edit kubeapiservers.operator.openshift.io/cluster

      ----------------------------

      {{ unsupportedConfigOverrides:}}

      {{ apiServerArguments:}}

      {{ max-mutating-requests-inflight:}}

      {{ - "2000"}}

      {{ max-requests-inflight:}}

      {{ - "6000"}}

      ----------------------------

      $ oc get -o yaml kubeapiservers.operator.openshift.io/cluster

       

      The use case for API tuning:

      We have configured FlowSchemas and PriorityLevelConfigurations for API Fairness to filter API requests for kubectl and our service accounts and split the concurrency shares/limits and queue accordingly. This works well, but we're hitting a ceiling of API QPS in our test and production clusters due to some spammy controllers. We have also increased the CPU/RAM on master nodes but they are hovering around 25-50% CPU/memory usage, so that doesn't seem to be the problem. Master disks latency is also very low (1-2ms) and etcd is performing well. According to https://www.openshift.com/blog/surviving-the-api-storm-with-api-priority-fairness, the current values for --max-requests-inflight and --max-mutating-requests-inflight are 3000 and 1000, for a total of 4000 concurrency limit. Right now we are hovering around 3-3.5k requests per second to the apiserver so things are getting throttled and slow. I have searched in the Openshift 4.7 docs for 'inflight' but there doesn't seem to be an option to configure --max-requests-inflight and --max-mutating-requests-inflight. 

       

      The customer is interested in a supported procedure to tune apiserver without unsupportedConfigOverrides.

              anachand Anandnatraj Chandramohan (Inactive)
              rhn-support-dahernan David Hernandez Fernandez
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: