We've had long running requests to boot into custom pools, starting from https://issues.redhat.com/browse/RFE-1136 (a lot of context in there), but essentially one of our main identified blockers was that the kubelet will not allow you to join the cluster as a new node with a custom role label (e.g. infra), so you will have to join as a worker.

The thing here is, there is nothing stopping you from getting a custom pool ignition config for a new node. You can simply make a request to the MCS endpoint and get e.g. infra and boot with it (although this isn't technically explicitly supported).

Once you do that, the new node actually joins as a worker. It will see the config it booted with is custom-xxx, and then the MCO will immediately update the node into a worker, and then once the labels are processed, then the MCO moves it to the custom pool, causing 3x reboots (bootstrap pivot, custom->worker, worker->custom).

In theory, all it takes for the MCO to support this is to "wait" until the node properly settles into the custom pool. The node can still request to join as a worker, but with the custom pool configs, and the MCO doesn't do the last 2 updates in the above flow. So essentially, the proposal here is:

1. for the MCS to explicitly support non master/worker endpoints
2. for the MCC to understand (maybe with an extra label/file in the MCS-served config) that a new worker node will eventually settle into a custom pool, and not attempt the double update

And that could get us to a point where we can formally support booting into custom pools. Custom role labelling is not considered as part of this, and will require the user to properly set up (e.g. via the machineset)

Note also related: https://github.com/openshift/machine-config-operator/pull/3431 where the assisted installer has support for a similar feature.