Feature Overview (aka. Goal Summary)  

This feature strengthens both security and AWS integration themes of ROSA service. Developers deploying workloads in ROSA service often need to connect their workloads running inside OCP to other AWS Services or applications running in other VPCs. This feature will allow cluster administrators to assign optional additional Security Groups to control plane nodes, infra nodes, and all the worker nodes of machine pools (at machine pool granularity). Following will be parts of this feature that will be delivered separately (not necessarily in the phased/ordered way) :

1. XCMSTRAT-46 (This): Support for creating additional Security Group IDs on day-one machine pool + Control Plane nodes + Infra nodes during ROSA Classic /OSD cluster creation
   1. Note: XCMSTRAT-374 is split from XCMSTRAT-46 to cover control Plane and Infra nodes
2. XCMSTRAT-41: Support for creating additional Security Group IDs on day-two machine pools after ROSA Classic/OSD cluster creation
3. XCMSTRAT-319: Support for changing machine pools to add/remove additional Security Group IDs on ROSA Classic/OSD cluster creation
4. XCMSTRAT-320: Support for Additional Security Group IDs on ROSA HCP clusters

Goals (aka. expected user outcomes)

1. Improve ROSA adoption by providing an ability to optionally set additional Security Group IDs at the time of cluster creation
2. Support for ROSA Classic and OSD on AWS cluster installations using OCP 4.14 and above

Requirements (aka. Acceptance Criteria):

A list of specific needs or objectives that a feature must deliver in order to be considered complete.  Be sure to include nonfunctional requirements such as security, reliability, performance, maintainability, scalability, usability, etc.  Initial completion during Refinement status.

1. Ability to set different additional security group IDs to control plane nodes, infra nodes, and day-one machine pool nodes. 
2. Ability to apply up to 15 5 (soft limit) Security Group IDs.
3. Ability to define additional Security Group IDs at the machine pool level and have all the worker nodes of the machine pool consistently use the SG-IDs.
4. Support for day-one machine pool created along with the cluster. (Support for day-two machine pools are covered as part of XCMSTRAT-41)
5. Support for clusters created with BYO-VPC. 
6. Support for ROSA and OSD CCS on AWS Clusters (Support for HCP is covered part of XCMSTRAT-319)
7. Support for OCP 4.14 and above.
8. OCM CREATE API for CS (api.openshift.com) to support additional SG-IDs.
9. OCM API for CS to support displaying/describing cluster with additional SG-IDs. Clusters that don't have additional SG-IDs to list 'None'. 
10. During pre-flight, provided SG is validated for existence and AWS Quota.
11. During pre-flight, number of total SG rules as per AWS quota is validated on top of total number of SGs attached.
12. During pre-flight, the SG-ID tags are validated to ensure OpenShift and ROSA specific tags can be added.
13. When the cluster is upgraded, all the nodes part of machine pool to retain the additional SG-IDs added.
14. When the cluster is deleted, the SG-ID is unaltered/retained. 
15. Consistent UX Support for clients - ROSA CLI, OCM UI and Terraform.
16. Documentation to support the field in the Cluster Creation page, ROSA CLI reference and Defaults for ROSA Cluster creation pages.
17. Release: CLI, OCM UI and OSDOCS to release/publish on the same day. Terraform release can follow on another day. 

Use Cases (Optional):

Include use case diagrams, main success scenarios, alternative flow scenarios.  Initial completion during Refinement status.

1. Connect from ROSA/ OSD applications running on worker machine pools to VPC resources secured by one or more security group. A most common example is to allow a deployment in ROSA service persist data in Amazon Relational Database Service (RDS) databases [1].
2. Define NodePort services that require opening 30000-32767 port range outside the cluster in case of using external Load balancer
3. Use Rosa machine pools in AWS Local Zones where ELBV2 (NLB, ALB) or ELB are not supported

Questions to Answer (Optional):

Include a list of refinement / architectural questions that may need to be answered before coding can begin.  Initial completion during Refinement status.

1. Does https://issues.redhat.com/browse/OCPBUGS-11524 affect delivery of this feature? This is pending discussion with node + network team
2.  

Out of Scope

High-level list of items that are out of scope.  Initial completion during Refinement status.

1. Creation of SG during the cluster installation 
2. Support for ROSA/Installer-provided VPC
3. Validation of rules in the additionalSecurity Group IDs that may or may not reference SGs from peered VPC
4. Changing or removing default SGs created by Installer [Contains necessary rules for cluster functions, governed by Appendix 4]
5. Day-2 and support for HCP (that are covered by other XCMSTRAT JIRAs highlighted in the overview section)

Background

Provide any additional context is needed to frame the feature.  Initial completion during Refinement status.

Customer Considerations

Provide any additional customer-specific considerations that must be made when designing and delivering the Feature.  Initial completion during Refinement status.

Documentation Considerations

Provide information that needs to be considered and planned so that documentation will meet customer needs.  Initial completion during Refinement status.

Interoperability Considerations

Which other projects and versions in our portfolio does this feature impact?  What interoperability test scenarios should be factored by the layered products?  Initial completion during Refinement status.

References:

 __  [1] Scenarios for connecting to RDS Database - https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_CommonTasks.Connect.html#CHAP_CommonTasks.Connect.ScenariosForAccess 

 [2]Controlling access to RDS instance using Security Groups - https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Overview.RDSSecurityGroups.html