-
Bug
-
Resolution: Won't Do
-
Normal
-
None
-
4.18, 4.19
-
None
-
Quality / Stability / Reliability
-
False
-
-
None
-
Important
-
None
-
None
-
None
-
Rejected
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Description of problem:
During Node density test with 24 worker nodes it's noticeable degradation of API request latencyy
Version-Release number of selected component (if applicable):
4.19: 4.19.0-0.nightly-2025-04-14-214145 4.19: 4.19.0-0.nightly-2025-04-17-154552
How reproducible:
So far 100% (tested on two clusters - details below.
Steps to Reproduce:
Run node density test on IBM or Nutanix cluster with 24 working nodes.
kube-burner: https://github.com/kube-burner/kube-burner-ocp
IBMCloud 4.19 vs 4.18 node density test: https://issues.redhat.com/browse/OCPQE-28923
Nutanix 4.19 vs 4.18 node density test: https://issues.redhat.com/browse/OCPQE-28920
Nutanix 4.18 vs 4.17 node density test: https://issues.redhat.com/browse/OCPQE-26744
IBMCloud:
Builds:
4.18: 4.18.0-0.nightly-2025-04-13-142946
4.19: 4.19.0-0.nightly-2025-04-14-214145
P99 Pod Ready latency
4.18: 77 s
4.19: 36 s (-53%)
Read Only API request P99 latency - resource scoped:
4.18: 142 ms
4.19: 327 ms (+130%)
Maximum Read Only API request P99 latency - resource scoped:
4.18: 610 ms
4.19: 2350 ms (+285%)
Read Only API request P99 latency - namespaced scoped:
4.18: 226 ms
4.19: 519 ms (+129%)
Maximum Read Only API request P99 latency - namespaced scoped:
4.18: 548 ms
4.19: 2760 ms (+403%)
Read Only API request P99 latency - cluster scoped:
4.18: 386 ms
4.19: 231 ms (-40%)
Maximum Read Only API request P99 latency - cluster scoped:
4.18: 1420 ms
4.19: 876 ms (-38%)
Nutanix:
Builds:
4.17: 4.17.0-0.nightly-2024-08-28-022602
4.18: 4.18.0-0.nightly-2025-04-17-181105
4.19: 4.19.0-0.nightly-2025-04-17-154552
P99 Pod Ready latency
4.17: 41 s
4.18: 41 s
4.19: 100 s (+143%)
Maximum Read Only API request P99 latency - resource scoped:
4.17: 303 ms
4.18: 462 ms
4.19: 465 ms
Maximum Read Only API request P99 latency - namespaced scoped:
4.17: 107 ms
4.18: 188 ms
4.19: 372 ms (+98%)
Maximum Read Only API request P99 latency - cluster scoped:
4.17: 658 ms
4.18: 564 ms
4.19: 580 ms (+3%)
IBMCloud 4.19 rehearse and pprof
IBMCloud 4.18 rehearse and pprof
Nutanix 4.19 rehearse and pprof
Nutanix 4.18 rehearse and pprof