ROSA: Additional Security Group(s) for Optional Machine Pools (Day-2)

Feature Overview (aka. Goal Summary)  

This feature strengthens both security and AWS integration themes of ROSA service. Developers deploying workloads in ROSA service often need to connect their workloads running inside OCP to other AWS Services or applications running in other VPCs. This feature will allow cluster administrators to assign optional additional Security Groups to control plane nodes, infra nodes, and all the worker nodes of machine pools (at machine pool granularity). Following will be parts of this feature that will be delivered separately (not necessarily in the phased/ordered way)

1. XCMSTRAT-41 (This): Support for creating additional Security Group IDs on day-two machine pools after ROSA Classic/OSD cluster creation
2. XCMSTRAT-46 : Support for creating additional Security Group IDs on day-one machine pool + Control Plane nodes + Infra nodes during ROSA Classic /OSD cluster creation (Note: XCMSTRAT-374 is split from XCMSTRAT-46 to cover control Plane and Infra nodes)
3. XCMSTRAT-319: Support for changing machine pools to add/remove additional Security Group IDs on ROSA Classic/OSD cluster creation
4. XCMSTRAT-320: Support for Additional Security Group IDs on ROSA HCP clusters

Goals (aka. expected user outcomes)

1. Improve ROSA retention by providing an ability to add additional machine pools with additional Security Group IDs to existing clusters 
2. Support for ROSA Classic and OSD on AWS using OCP 4.11 and above.

Requirements (aka. Acceptance Criteria): 

1. Ability to create machine pool in an existing cluster with additional Security Groups.
2. Ability to apply up to 15 Security Group IDs.
3. Ability to define additional Security Group IDs at the machine pool level and have all the worker nodes of the machine pool consistently use the SG-IDs.
4. Support for clusters created with BYO-VPC. 
5. Support for ROSA and OSD CCS on AWS Clusters (Support for HCP is covered part of XCMSTRAT-319)
6. Support for OCP 4.11 and above.
7. OCM CREATE MACHINE Pool API for CS (api.openshift.com) to support additional SG-IDs.
8. During pre-flight, provided SG is validated for existence and AWS Quota.
9. During pre-flight, number of total SG rules as per AWS quota is validated on top of total number of SGs attached.
10. During pre-flight, the SG-ID tags are validated to ensure OpenShift and ROSA specific tags can be added.
11. When the cluster is upgraded, all the nodes part of machine pool to retain the additional SG-IDs added.
12. When the cluster is deleted, the SG-ID is unaltered/retained. 
13. Consistent UX Support for clients - ROSA CLI, OCM UI and Terraform.
14. OCM API for CS to support displaying/describing cluster with additional SG-IDs. Clusters that don't have additional SG-IDs to list 'None'. 
15. ROSA CLI Describe command list additional security groups on control plane nodes, and infra nodes
16. ROSA CLI for machine pool to display additional SG-IDs assigned .
17. Documentation to support the field in the Cluster Creation page, ROSA CLI reference and Defaults for ROSA Cluster creation pages.
18. Release: CLI, OCM UI and OSDOCS to release/publish on the same day. Terraform release can follow on another day. 

Use Cases (Optional):

Include use case diagrams, main success scenarios, alternative flow scenarios.  Initial completion during Refinement status.

Questions to Answer (Optional):

Include a list of refinement / architectural questions that may need to be answered before coding can begin.  Initial completion during Refinement status.

Out of Scope

High-level list of items that are out of scope.  Initial completion during Refinement status.

Background

Provide any additional context is needed to frame the feature.  Initial completion during Refinement status.

Customer Considerations

Provide any additional customer-specific considerations that must be made when designing and delivering the Feature.  Initial completion during Refinement status.

Documentation Considerations

Provide information that needs to be considered and planned so that documentation will meet customer needs.  Initial completion during Refinement status.

Interoperability Considerations

Which other projects and versions in our portfolio does this feature impact?  What interoperability test scenarios should be factored by the layered products?  Initial completion during Refinement status.

References:

 __  [1] Scenarios for connecting to RDS Database - https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_CommonTasks.Connect.html#CHAP_CommonTasks.Connect.ScenariosForAccess 

 [2]Controlling access to RDS instance using Security Groups - https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Overview.RDSSecurityGroups.html