Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-28382

Client side throttling when running the metrics controller

XMLWordPrintable

    • No
    • False
    • Hide

      None

      Show
      None
    • Release Note Not Required
    • In Progress

      This is a clone of issue OCPBUGS-27901. The following is the description of the original issue:

      This is a clone of issue OCPBUGS-27445. The following is the description of the original issue:

      Description of problem:

      Client side throttling observed when running the metrics controller. 

      Steps to Reproduce:

      1. Install an AWS cluster in mint mode
      2. Enable debug log by editing cloudcredential/cluster
      3. Wait for the metrics loop to run for a few times
      4. Check CCO logs

      Actual results:

      // 7s consumed by metrics loop which is caused by client-side throttling 
      time="2024-01-20T19:43:56Z" level=info msg="calculating metrics for all CredentialsRequests" controller=metrics
      I0120 19:43:56.251278       1 request.go:629] Waited for 176.161298ms due to client-side throttling, not priority and fairness, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-cloud-network-config-controller/secrets/cloud-credentials
      I0120 19:43:56.451311       1 request.go:629] Waited for 197.182213ms due to client-side throttling, not priority and fairness, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-cloud-network-config-controller/secrets/cloud-credentials
      I0120 19:43:56.651313       1 request.go:629] Waited for 197.171082ms due to client-side throttling, not priority and fairness, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-cloud-network-config-controller/secrets/cloud-credentials
      I0120 19:43:56.850631       1 request.go:629] Waited for 195.251487ms due to client-side throttling, not priority and fairness, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-cloud-network-config-controller/secrets/cloud-credentials
      ...
      time="2024-01-20T19:44:03Z" level=info msg="reconcile complete" controller=metrics elapsed=7.231061324s

      Expected results:

      No client-side throttling when running the metrics controller. 

              jstuever@redhat.com Jeremiah Stuever
              openshift-crt-jira-prow OpenShift Prow Bot
              Jianping Shu Jianping Shu
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated:
                Resolved: