1. Proposed title of this feature request

Exclude specific nodes from ClusterAutoscaler resource limits via annotation

2. What is the nature and description of the request?

This feature request proposes a mechanism to allow certain nodes to be excluded from the total resource count considered by the Cluster Autoscaler. The proposed solution is to introduce a new annotation that, when applied to a node, would signal the autoscaler to ignore that node's resources (e.g., CPU, memory) when calculating cluster-wide resource limits like cores.max and memory.max. This would give cluster administrators greater control over which resources are accounted for in their scaling policies.

3. Why does the customer need this? (List the business requirements here)

A key challenge for customers, particularly those using managed services like ARO (Azure Red Hat OpenShift) where the control plane is managed by SRE team, is that the control plane's resources are included in the global Cluster Autoscaler configuration. The SRE team can increase the size of the control plane instances at any time for maintenance or performance reasons.

This poses a problem because if a customer sets a cores.max or memory.max limit, any increase in the control plane's resources will consume part of that limit. This can inadvertently prevent the Cluster Autoscaler from scaling up worker nodes, as the cluster may already be at its defined maximum resource capacity even though the customer's workloads are not using all of those resources.

By allowing the control plane nodes to be excluded from the resource count, the customer can set a cores.max and memory.max that applies only to the worker nodes, ensuring that changes to the control plane don't negatively impact their ability to scale their application workloads. 

4. List any affected packages or components.

Cluster Autoscaler: The core logic of the Cluster Autoscaler will need to be modified to recognize and process the new annotation, and to exclude the resources of annotated nodes from its calculations.

Documentation: All relevant documentation for the Cluster Autoscaler, especially concerning configuration and best practices, will need to be updated to reflect this new feature.