Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-22745

Hypershift HC kube-apiserver latency metrics are constant

XMLWordPrintable

    • No
    • False
    • Hide

      None

      Show
      None

      Description of problem:

      As part of Hypershift perf&scale exercise, we run cluster-density workloads which are control-plane intensive as well as KAS, we measure latency metrics to understand their performances. 
      Surprisingly, the api latency metrics like all apiserver_request_duration_seconds_bucket are consistenlty at 990ms even during idle as well as during heavy operation. we suspect its not reporting a right state.
      
      This is the query we used to record the linked snapshot,
      histogram_quantile(0.99, sum(rate(apiserver_request_duration_seconds_bucket{namespace=~"$namespace",resource=~"$resource",subresource!="log",verb!~"WATCH|WATCHLIST|PROXY"}[2m])) by(verb,le))

      snapshot - https://snapshots.raintank.io/dashboard/snapshot/o91PapLH0AxaI7KogIUckfKA0lG9KJwR?orgId=2&viewPanel=57

      Version-Release number of selected component (if applicable):

      4.14.0-rc.6

      How reproducible:

      Always

      Steps to Reproduce:

      1. Deploy a HC
      2. watch apiserver_request_duration_seconds_bucket
      3. They are constant at 990ms
      

      Actual results:

      Latency is constant, although we see spikes during workload.

      Expected results:

      During idle we don't expect 990ms latency for GET, POST, PATHC, APPLY, DELETE calls 

      Additional info:

       

        1. image-2023-11-01-09-27-39-386.png
          326 kB
          Abu H Kashem
        2. image-2023-11-08-08-32-25-283.png
          316 kB
          Abu H Kashem
        3. image-2023-11-08-08-33-54-452.png
          413 kB
          Abu H Kashem
        4. image-2023-11-08-08-47-29-895.png
          608 kB
          Abu H Kashem
        5. image-2023-11-09-12-03-10-155.png
          408 kB
          Abu H Kashem
        6. image-2023-11-22-12-09-33-050.png
          209 kB
          Abu H Kashem
        7. image-2023-11-22-12-10-49-634.png
          569 kB
          Abu H Kashem
        8. image-2023-11-22-12-32-36-695.png
          437 kB
          Abu H Kashem

              Unassigned Unassigned
              mukrishn@redhat.com Murali Krishnasamy
              Ke Wang Ke Wang
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated:
                Resolved: