Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-55390

[OCP 4.19 vs 4.18] Degradation of API request latency

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Won't Do
    • Icon: Normal Normal
    • None
    • 4.18, 4.19
    • kube-apiserver
    • None
    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • Important
    • None
    • None
    • None
    • Rejected
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Description of problem:

          During Node density test with 24 worker nodes it's noticeable degradation of API request latencyy

      Version-Release number of selected component (if applicable):

          4.19: 4.19.0-0.nightly-2025-04-14-214145
      
          4.19: 4.19.0-0.nightly-2025-04-17-154552
      

      How reproducible:

          So far 100% (tested on two clusters - details below.

      Steps to Reproduce:

      Run node density test on IBM or Nutanix cluster with 24 working nodes.

      kube-burner: https://github.com/kube-burner/kube-burner-ocp 

       
      IBMCloud 4.19 vs 4.18 node density test: https://issues.redhat.com/browse/OCPQE-28923
      Nutanix 4.19 vs 4.18 node density test: https://issues.redhat.com/browse/OCPQE-28920
      Nutanix 4.18 vs 4.17 node density test: https://issues.redhat.com/browse/OCPQE-26744

      IBMCloud:

      Builds:
      4.18: 4.18.0-0.nightly-2025-04-13-142946
      4.19: 4.19.0-0.nightly-2025-04-14-214145

      P99 Pod Ready latency
      4.18: 77 s
      4.19: 36 s (-53%)

      Read Only API request P99 latency - resource scoped:
      4.18: 142 ms
      4.19: 327 ms (+130%)

      Maximum Read Only API request P99 latency - resource scoped:
      4.18: 610 ms
      4.19: 2350 ms (+285%)

      Read Only API request P99 latency - namespaced scoped:
      4.18: 226 ms
      4.19: 519 ms (+129%)

      Maximum Read Only API request P99 latency - namespaced scoped:
      4.18: 548 ms
      4.19: 2760 ms (+403%)

      Read Only API request P99 latency - cluster scoped:
      4.18: 386 ms
      4.19: 231 ms (-40%)

      Maximum Read Only API request P99 latency - cluster scoped:
      4.18: 1420 ms
      4.19: 876 ms (-38%)

      Nutanix:

      Builds:
      4.17: 4.17.0-0.nightly-2024-08-28-022602
      4.18: 4.18.0-0.nightly-2025-04-17-181105
      4.19: 4.19.0-0.nightly-2025-04-17-154552

      P99 Pod Ready latency
      4.17: 41 s
      4.18: 41 s
      4.19: 100 s (+143%)

      Maximum Read Only API request P99 latency - resource scoped:
      4.17: 303 ms
      4.18: 462 ms
      4.19: 465 ms

      Maximum Read Only API request P99 latency - namespaced scoped:
      4.17: 107 ms
      4.18: 188 ms
      4.19: 372 ms (+98%)

      Maximum Read Only API request P99 latency - cluster scoped:
      4.17: 658 ms
      4.18: 564 ms
      4.19: 580 ms (+3%)

      IBMCloud 4.19 rehearse and pprof
      IBMCloud 4.18 rehearse and pprof
      Nutanix 4.19 rehearse and pprof
      Nutanix 4.18 rehearse and pprof

       

              Unassigned Unassigned
              skordas Simon Kordas
              None
              None
              Ke Wang Ke Wang
              None
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated:
                Resolved: