Description of problem:

    During Node density test with 24 worker nodes it's noticeable degradation of API request latencyy

Version-Release number of selected component (if applicable):

    4.19: 4.19.0-0.nightly-2025-04-14-214145

    4.19: 4.19.0-0.nightly-2025-04-17-154552

How reproducible:

    So far 100% (tested on two clusters - details below.

Steps to Reproduce:

Run node density test on IBM or Nutanix cluster with 24 working nodes.

kube-burner: https://github.com/kube-burner/kube-burner-ocp 

 
IBMCloud 4.19 vs 4.18 node density test: https://issues.redhat.com/browse/OCPQE-28923
Nutanix 4.19 vs 4.18 node density test: https://issues.redhat.com/browse/OCPQE-28920
Nutanix 4.18 vs 4.17 node density test: https://issues.redhat.com/browse/OCPQE-26744

IBMCloud:

Builds:
4.18: 4.18.0-0.nightly-2025-04-13-142946
4.19: 4.19.0-0.nightly-2025-04-14-214145

P99 Pod Ready latency
4.18: 77 s
4.19: 36 s (-53%)

Read Only API request P99 latency - resource scoped:
4.18: 142 ms
4.19: 327 ms (+130%)

Maximum Read Only API request P99 latency - resource scoped:
4.18: 610 ms
4.19: 2350 ms (+285%)

Read Only API request P99 latency - namespaced scoped:
4.18: 226 ms
4.19: 519 ms (+129%)

Maximum Read Only API request P99 latency - namespaced scoped:
4.18: 548 ms
4.19: 2760 ms (+403%)

Read Only API request P99 latency - cluster scoped:
4.18: 386 ms
4.19: 231 ms (-40%)

Maximum Read Only API request P99 latency - cluster scoped:
4.18: 1420 ms
4.19: 876 ms (-38%)

Nutanix:

Builds:
4.17: 4.17.0-0.nightly-2024-08-28-022602
4.18: 4.18.0-0.nightly-2025-04-17-181105
4.19: 4.19.0-0.nightly-2025-04-17-154552

P99 Pod Ready latency
4.17: 41 s
4.18: 41 s
4.19: 100 s (+143%)

Maximum Read Only API request P99 latency - resource scoped:
4.17: 303 ms
4.18: 462 ms
4.19: 465 ms

Maximum Read Only API request P99 latency - namespaced scoped:
4.17: 107 ms
4.18: 188 ms
4.19: 372 ms (+98%)

Maximum Read Only API request P99 latency - cluster scoped:
4.17: 658 ms
4.18: 564 ms
4.19: 580 ms (+3%)

IBMCloud 4.19 rehearse and pprof
IBMCloud 4.18 rehearse and pprof
Nutanix 4.19 rehearse and pprof
Nutanix 4.18 rehearse and pprof