-
Feature Request
-
Resolution: Unresolved
-
Undefined
-
None
-
None
-
None
-
None
-
Product / Portfolio Work
-
None
-
False
-
-
None
-
None
-
None
-
-
None
-
None
-
None
-
None
-
None
From a Performance & Scale perspective, We want the hosted cluster scheduler to prioritize scheduling pods for the oldest waiting cluster first, so that Hosted Control Plane (HCP) installation times are consistent and predictable, meeting our control plane readiness SLO.
During high-density ROSA Hosted Control Plane (HCP) testing by Perf team, we observed a potential scheduling optimization opportunity that can intermittently increase control plane installation times. We ran a test creating one HCP cluster per minute on a single Management Cluster (with the MC autoscaler disabled). When multiple HCPs were waiting for resources on newly added serving nodes, their kube-apiserver pods were not consistently scheduled in a First-In, First-Out (FIFO) order.
Currently, we do not enforce any specific scheduling order for pods belonging to different hosted clusters. This can lead to a situation where a newer cluster's pods are scheduled before an older cluster's pods, unnecessarily delaying the installation of the older cluster.
Proposal:
We should investigate and potentially modify the scheduler to prioritize pods based on the creation timestamp of their corresponding HostedCluster object. This would ensure that the oldest clusters waiting for resources are scheduled first, leading to a more predictable and efficient installation process.