When running as a Managed Service, we should expect Nodes in our Kubernetes cluster to be ephemeral. It is likely the the Kubernetes cluster will be scaled up/down aggressively to keep running costs at a minimum.

We need to ensure that the end user experience is not impacted by our need to keep on-top of running costs. If a Node is drained and removed from the Cluster, this should not be visible to the end user.

In particular, when scheduling a KogitoRuntime, I can foresee requirements such:

• Ensure that there are always at-least % of replicas running for a KogitoRuntime in the event of Cluster Administrator actions e.g. drain a node
• Do not schedule all of a customers KogitoRuntimes onto the same node (if we can avoid it). This would be a "soft" scheduling requirement
• Do not schedule all replicas for a KogitoRuntime onto the same node (if we can avoid it). This would be a "soft" scheduling requirement.

There may be other requirements. This feature request is for us to consider how best to expose this ability and where the functionality should reside.