Namespace-based default Secondary Scheduler

1. Proposed title of this feature request

Namespace-based default scheduler

2. What is the nature and description of the request?
The "node-scheduler-profiles" are fixed and predefined by OCP. Using the "secondary-scheduler-operator" requires the Pods to specify the scheduler they like to use.

With Operators and Helm Charts provided by 3rd parties, the option to include such attribute in the pod.spec is usually not possible. In the case of the operator, it might "correct" the Pod definition when the cluster-admin tries to patch it.

The ask is to enable the ability to annotate or label an NS such that all Pods deployed into that NS use a specific preferred scheduler instead of the K8s default scheduler. By not having to modify the Pods specs, this option should work well with Operators.

3. Why does the customer need this? (List the business requirements here)

(see above)

4. List any affected packages or components.

node-scheduler-profiles
secondary-scheduler-operator