Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-22745

Hypershift HC kube-apiserver latency metrics are constant

XMLWordPrintable

    • No
    • False
    • Hide

      None

      Show
      None

      Description of problem:

      As part of Hypershift perf&scale exercise, we run cluster-density workloads which are control-plane intensive as well as KAS, we measure latency metrics to understand their performances. 
      Surprisingly, the api latency metrics like all apiserver_request_duration_seconds_bucket are consistenlty at 990ms even during idle as well as during heavy operation. we suspect its not reporting a right state.
      
      This is the query we used to record the linked snapshot,
      histogram_quantile(0.99, sum(rate(apiserver_request_duration_seconds_bucket{namespace=~"$namespace",resource=~"$resource",subresource!="log",verb!~"WATCH|WATCHLIST|PROXY"}[2m])) by(verb,le))

      snapshot - https://snapshots.raintank.io/dashboard/snapshot/o91PapLH0AxaI7KogIUckfKA0lG9KJwR?orgId=2&viewPanel=57

      Version-Release number of selected component (if applicable):

      4.14.0-rc.6

      How reproducible:

      Always

      Steps to Reproduce:

      1. Deploy a HC
      2. watch apiserver_request_duration_seconds_bucket
      3. They are constant at 990ms
      

      Actual results:

      Latency is constant, although we see spikes during workload.

      Expected results:

      During idle we don't expect 990ms latency for GET, POST, PATHC, APPLY, DELETE calls 

      Additional info:

       

            Unassigned Unassigned
            mukrishn@redhat.com Murali Krishnasamy
            Ke Wang Ke Wang
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: