-
Feature Request
-
Resolution: Done
-
Normal
-
None
-
None
-
False
-
False
-
Undefined
-
Modification of specific configuration under the kube-apiserver CM needs to be done under unsupportedConfigOverrides:
$ oc -n openshift-kube-apiserver get configmap config --output 'go-template={{index .data "config.yaml"'}}
There is no supported way to modify --max-requests-inflight and --max-mutating-requests-inflight , so it would be then under unsupported configurations for now:
$ oc explain --api-version=operator.openshift.io/v1 kubeapiservers.spec
$ oc edit kubeapiservers.operator.openshift.io/cluster
----------------------------
{{ unsupportedConfigOverrides:}}
{{ apiServerArguments:}}
{{ max-mutating-requests-inflight:}}
{{ - "2000"}}
{{ max-requests-inflight:}}
{{ - "6000"}}
----------------------------
$ oc get -o yaml kubeapiservers.operator.openshift.io/cluster
The use case for API tuning:
We have configured FlowSchemas and PriorityLevelConfigurations for API Fairness to filter API requests for kubectl and our service accounts and split the concurrency shares/limits and queue accordingly. This works well, but we're hitting a ceiling of API QPS in our test and production clusters due to some spammy controllers. We have also increased the CPU/RAM on master nodes but they are hovering around 25-50% CPU/memory usage, so that doesn't seem to be the problem. Master disks latency is also very low (1-2ms) and etcd is performing well. According to https://www.openshift.com/blog/surviving-the-api-storm-with-api-priority-fairness, the current values for --max-requests-inflight and --max-mutating-requests-inflight are 3000 and 1000, for a total of 4000 concurrency limit. Right now we are hovering around 3-3.5k requests per second to the apiserver so things are getting throttled and slow. I have searched in the Openshift 4.7 docs for 'inflight' but there doesn't seem to be an option to configure --max-requests-inflight and --max-mutating-requests-inflight.
The customer is interested in a supported procedure to tune apiserver without unsupportedConfigOverrides.