Boot new workers directly into custom pool configuration

Feature Overview (aka. Goal Summary)

Today booting directly into custom pools is not possible because it is not permitted. Kubernetes has a NodeRestriction plugin that limits what labels can be requested on node startup. Custom node-role labels are not allowed.

Unfortunately this causes multiple problems:

• Forces an extra reboot in the deployment process, which can be very time consuming on bare metal installs and scaleouts
• Can cause race conditions where workloads are deployed onto nodes before required host configuration changes can be made.

Goals (aka. expected user outcomes)

In order to give administrators the experience they naturally expect and meet the needs of workload placement, we need to be able to boot directly into custom pools without the node needing to join as a "worker" (default pool) worker.

Requirements (aka. Acceptance Criteria):

A list of specific needs or objectives that a feature must deliver in order to be considered complete. Be sure to include nonfunctional requirements such as security, reliability, performance, maintainability, scalability, usability, etc. Initial completion during Refinement status.

Use Cases (Optional):

Include use case diagrams, main success scenarios, alternative flow scenarios. Initial completion during Refinement status.

Questions to Answer (Optional):

Include a list of refinement / architectural questions that may need to be answered before coding can begin. Initial completion during Refinement status.

Out of Scope

High-level list of items that are out of scope. Initial completion during Refinement status.

Background

Provide any additional context is needed to frame the feature. Initial completion during Refinement status.

Customer Considerations

Provide any additional customer-specific considerations that must be made when designing and delivering the Feature. Initial completion during Refinement status.

Documentation Considerations

Provide information that needs to be considered and planned so that documentation will meet customer needs. If the feature extends existing functionality, provide a link to its current documentation. Initial completion during Refinement status.

Interoperability Considerations

Which other projects, including ROSA/OSD/ARO, and versions in our portfolio does this feature impact? What interoperability test scenarios should be factored by the layered products? Initial completion during Refinement status.