1. Proposed title of this feature request

ArgoCD Application controllers sharding mechanisms to also be based on Applications other than being based on target clusters only.

2. What is the nature and description of the request?

Currently ArgoCD Application controllers shards the load (Application syncs) only based on the target cluster thus limiting the scalability in scenarios where there are a lot of applications targeting a single cluster: Basically you can only scale vertically but not horizontally.
Ideally, you should be able to have 1000 apps spreaded unequally across 10 Clusters, and have all the Applications controllers equally managing the apps.

Example:

• Customer has Clusters A and Cluster B, where Cluster A hosts 1000 Applications and Cluster B hosts 10 Applications
• ArgoCD Application controllers are scaled to 2 replicas
• One replica handles Cluster A, hence is reconciling 1000 Applications - While the other replica handles Cluster B, which only reconciles 10 Applications
• Currently, to avoid OOM Kills, you will have to tune RAM and CPU requests and limits to accomodate for 1000 Applications for both replicas,
• This implies wasting resources as the Cluster B application controller doesn't need that much resources, as it's only reconciling 10 Applications

The goal for this RFE would be to enable ArgoCD to equally spread the load on the Application controllers shards, so that each shard handles 505 Applications each (500 from Cluster A, and 5 from Cluster B). This would allow to a much easier scaling (Both Horizontal and Vertical) while efficiently using the available resources.

3. Why does the customer need this?

Customer is managing different clusters and has sharding enabled, but only one of the clusters has lots of applications hence easily causing OOMs on the Application controller shard managing that particular cluster.
They have to vertically scale up all the application-controllers wasting resources for smaller clusters managed by other shards, while it would be more efficient if ArgoCD could spread the Apps equally on all the shards.

4. List any affected packages or components.

OpenShift GitOps - Application Controller