Hive in split control plane for etcd scaling

Feature

Hive in split control plane for etcd scaling

Goal

• Make hive understand two different "control plane API endpoints": one for Jobs, Pods, anything that needs to run on a worker node; another for non-"runnable" objects/CRs like ClusterDeployment, MachinePool, Secret, etc.
• Some background: https://coreos.slack.com/archives/CE3ETN3J8/p1651246764652499

Why is this important?

• ACM would like to run hive in a "split control plane" for etcd scaling purposes. See "Scenarios".

Scenarios

Customer wants to create/manage O(100k) spoke clusters from a single OCP cluster. Due to the number of objects each requires, etcd on one ACM+hive can handle O(1k). ACM proposes an architecture where:

• ACM/MCE/Hive stacks are running in O(100) separate namespaces on a single OCP cluster. We'll call these "control planes".
• Each control plane talks to a corresponding "data plane", a virtualized kube API server running in the same namespace, importantly with its own independent etcd.
• A single "hub of hubs" ACM front end aggregates everything for management and observability. (Does this front end run in the big OCP cluster or elsewhere?) (Hive doesn't know/care about this layer.)
  In this model, all the "runnables" (containers in pods from deployments/statefulsets including controllers; containers in pods created from jobs for e.g. provision/deprovision; maybe other things??) need to be created/reconciled in the control planes; but all the other objects – hive CRs, the Secrets hive consumes, etc. – need to be created/reconciled in the data plane corresponding to that control plane.

This means we need to teach hive to understand two API servers and manage the right objects in the right one.
From a hive API perspective, this is probably just one field in HiveConfig. Internally, we use different clients for the control plane and data plane objects.
The hard part is going to be figuring out which code paths need to use which clients, and validating that we didn't miss any