Uploaded image for project: 'Red Hat Advanced Cluster Management'
  1. Red Hat Advanced Cluster Management
  2. ACM-9269

hcp_capacity_calculation error in hypershift addon agent on startup

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Critical Critical
    • MCE 2.5.0
    • ACM 2.10.0, MCE 2.5.0
    • HyperShift
    • None
    • Important
    • No

      Description of problem:

      The hypershift-addon agent is showing the following error upon startup:

      I0108 18:26:12.896595       1 request.go:682] Waited for 1.020741761s due to client-side throttling, not priority and fairness, request: GET:https://172.30.0.1:443/apis/hive.openshift.io/v1?timeout=32s
      2024-01-08T18:26:21.481Z	INFO	agent.agent-reconciler	install/upgrade.go:72	check if HyperShift operator re-installation is required (startup=true, installfailed=false)
      2024-01-08T18:26:21.481Z	INFO	agent.agent-reconciler	install/upgrade.go:77	hypershift operator re-installation is required
      2024-01-08T18:26:21.481Z	INFO	agent.agent-reconciler	agent/agent.go:318	heath probes server is running...
      2024-01-08T18:26:21.489Z	INFO	agent.agent-reconciler	install/hypershift.go:187	reinstallCheckRequired = false
      2024-01-08T18:26:21.523Z	INFO	agent.controller-manager-setup	agent/agent.go:232	the maximum hosted cluster count set to 80
      2024-01-08T18:26:21.524Z	INFO	agent.controller-manager-setup	agent/agent.go:233	the threshold hosted cluster count set to 60
      2024-01-08T18:26:21.535Z	INFO	agent.agent-reconciler	install/hypershift.go:616	createorupdate the pull secret (hypershift/open-cluster-management-image-pull-credentials)
      2024-01-08T18:26:21.582Z	INFO	agent.agent-reconciler	install/hypershift.go:218	bucket secret(local-cluster/hypershift-operator-oidc-provider-s3-credentials) not found on the hub.
      2024-01-08T18:26:21.593Z	INFO	agent.agent-reconciler	install/hypershift.go:230	private link secret(local-cluster/hypershift-operator-private-link-credentials) not found on the hub.
      2024-01-08T18:26:21.593Z	INFO	agent.agent-reconciler	install/hypershift.go:241	bucket secret(local-cluster/hypershift-operator-oidc-provider-s3-credentials) and private link secret(local-cluster/hypershift-operator-private-link-credentials) not found on the hub, installing hypershift operator for non-AWS platform.
      2024-01-08T18:26:21.602Z	INFO	agent.agent-reconciler	install/hypershift.go:313	external dns secret(local-cluster/hypershift-operator-external-dns-credentials) was not found
      I0108 18:26:22.933230       1 request.go:682] Waited for 1.3504795s due to client-side throttling, not priority and fairness, request: GET:https://172.30.0.1:443/apis/infrastructure.cluster.x-k8s.io/v1beta2?timeout=32s
      2024-01-08T18:26:27.686Z	INFO	agent.agent-reconciler	agent/agent.go:912	this is the initial agent startup and the hypershift CRDs are not installed yet, no matches for kind "HostedCluster" in version "hypershift.openshift.io/v1beta1"
      2024-01-08T18:26:27.686Z	INFO	agent.agent-reconciler	agent/agent.go:913	going to continue updating AddOnPlacementScore and cluster claims with zero HC count
      2024-01-08T18:26:27.693Z	INFO	agent.agent-reconciler	agent/agent.go:991	updated the addOnPlacementScore for local-cluster: 0
      2024-01-08T18:26:27.693Z	INFO	agent.agent-reconciler	agent/clusterclaim.go:73	the hosted cluster count has not reached the maximum 80 yet. current count is 0
      2024-01-08T18:26:27.694Z	INFO	agent.agent-reconciler	install/install_job.go:20	HyperShift install args: [--namespace hypershift --image-refs /tmp/hypershift-operator-imagestream --platform-monitoring OperatorOnly --enable-uwm-telemetry-remote-write --enable-defaulting-webhook --enable-validating-webhook]
      2024-01-08T18:26:27.725Z	INFO	agent.agent-reconciler	install/install_job.go:106	created HyperShift install job: hypershift-install-job-kzvx9
      2024-01-08T18:26:27.725Z	INFO	agent.agent-reconciler	agent/agent.go:1012	updated the hosted cluster cound cluster claims successfully
      2024-01-08T18:26:27.730Z	ERROR	agent.agent-reconciler	agent/agent.go:788	failed to get configmap from the hub. Setting the HCP sizing baseline with default values.	{"error": "configmaps \"hcp-sizing-baseline\" not found"}
      github.com/stolostron/hypershift-addon-operator/pkg/agent.(*agentController).SetHCPSizingBaseline
      	/remote-source/hypershift-addon-operator/app/pkg/agent/agent.go:788
      github.com/stolostron/hypershift-addon-operator/pkg/agent.(*AgentOptions).runControllerManager
      	/remote-source/hypershift-addon-operator/app/pkg/agent/agent.go:244
      github.com/stolostron/hypershift-addon-operator/pkg/agent.NewAgentCommand.func1
      	/remote-source/hypershift-addon-operator/app/pkg/agent/agent.go:82
      github.com/spf13/cobra.(*Command).execute
      	/remote-source/hypershift-addon-operator/app/vendor/github.com/spf13/cobra/command.go:856
      github.com/spf13/cobra.(*Command).ExecuteC
      	/remote-source/hypershift-addon-operator/app/vendor/github.com/spf13/cobra/command.go:974
      github.com/spf13/cobra.(*Command).Execute
      	/remote-source/hypershift-addon-operator/app/vendor/github.com/spf13/cobra/command.go:902
      main.main
      	/remote-source/hypershift-addon-operator/app/cmd/main.go:41
      runtime.main
      	/usr/lib/golang/src/runtime/proc.go:267
      I0108 18:26:32.983146       1 request.go:682] Waited for 5.197303059s due to client-side throttling, not priority and fairness, request: GET:https://172.30.0.1:443/apis/controlplane.operator.openshift.io/v1alpha1?timeout=32s
      2024-01-08T18:26:33.937Z	ERROR	agent.agent-reconciler	agent/hcp_capacity_calculation.go:75	failed to list hosted control planes	{"error": "no matches for kind \"HostedControlPlane\" in version \"hypershift.openshift.io/v1beta1\""}
      github.com/stolostron/hypershift-addon-operator/pkg/agent.(*agentController).calculateCapacitiesToHostHCPs
      	/remote-source/hypershift-addon-operator/app/pkg/agent/hcp_capacity_calculation.go:75
      github.com/stolostron/hypershift-addon-operator/pkg/agent.(*AgentOptions).runControllerManager
      	/remote-source/hypershift-addon-operator/app/pkg/agent/agent.go:246
      github.com/stolostron/hypershift-addon-operator/pkg/agent.NewAgentCommand.func1
      	/remote-source/hypershift-addon-operator/app/pkg/agent/agent.go:82
      github.com/spf13/cobra.(*Command).execute
      	/remote-source/hypershift-addon-operator/app/vendor/github.com/spf13/cobra/command.go:856
      github.com/spf13/cobra.(*Command).ExecuteC
      	/remote-source/hypershift-addon-operator/app/vendor/github.com/spf13/cobra/command.go:974
      github.com/spf13/cobra.(*Command).Execute
      	/remote-source/hypershift-addon-operator/app/vendor/github.com/spf13/cobra/command.go:902
      main.main
      	/remote-source/hypershift-addon-operator/app/cmd/main.go:41
      runtime.main
      	/usr/lib/golang/src/runtime/proc.go:267
      2024-01-08T18:26:33.937Z	INFO	agent.controller-manager-setup	agent/agent.go:248	starting manager
      I0108 18:26:43.009006       1 request.go:682] Waited for 4.188887235s due to client-side throttling, not priority and fairness, request: GET:https://172.30.0.1:443/apis/machine.openshift.io/v1beta1?timeout=32s
      2024-01-08T18:26:44.965Z	INFO	agent.agent-reconciler.external-secret-controller	agent/external_secret_controller.go:54	reconciling klusterlet: klusterlet
      2024-01-08T18:26:44.965Z	INFO	agent.agent-reconciler.external-secret-controller	agent/external_secret_controller.go:58	klusterlet not from a hosted cluster
      2024-01-08T18:26:44.965Z	INFO	agent.agent-reconciler.external-secret-controller	agent/external_secret_controller.go:59	done reconciling klusterlet: klusterlet
      2024-01-08T18:26:45.066Z	INFO	agent.agent-reconciler.addon-status-controller	agent/addon_status_controller.go:65	reconciling Deployment hypershift/operator
      2024-01-08T18:26:45.096Z	INFO	agent.agent-reconciler.addon-status-controller	agent/addon_status_controller.go:98	updated ManagedClusterAddOnStatus 

      This results in the new cluster capacity metrics all showing 0 (since no baselines are set).

      Version-Release number of selected component (if applicable):

      ACM 2.10.0-DOWNSTREAM-2024-01-05-20-32-42

      Hub is on OCP 4.15.0-rc.0 with fips enabled

      How reproducible:

      always

      Steps to Reproduce:

      1. install latest acm/mce 
      2. check hypershift addon agent log
      3. ...

      Actual results:

      Expected results:

      Additional info:

              rokejungrh Roke Jung
              rhn-support-dhuynh David Huynh
              David Huynh David Huynh
              ACM QE Team
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated:
                Resolved: