This is to spike on the new scope to support IngressController subnet selection at installation time so we can cover the ask coming from RFE-4738

Goal:

• Create a on AWS isolating the API LBs to a different subnets of control plane and worker nodes

Description:

RFE-4738 requests to install API's LBs to a different subnets from worker nodes for IP restrictions/regulations. Currently the API's LBs are created by installer. The EP 1634 proposes to introduce "roles" to compose granular deployments when adding subnets, BYO VPC/unmanaged.

The scope of this spike is to create a cluster to validate the requested scenario prior implementation of proposed in EP, so we can get confidence of any nuance that would exists in that environment.

To reproduce that in the existin codebase, prior EP, it is required to install using UPI-method (create network, LBs, and nodes). Currently the approach of perform UPI installs is using CloudFormation template.

Requirements:

• Validate initial deployment of installer implementation of new subnets API (platform.aws.vpc.subnets)
• Create CloudFormation template, or reuse existing blocks, to create VPC and dependencies, duplicating subnets in existing zones, two subnets in the same zone
• create the API and Ingress LB's to different subnet pairs than the control plane nodes
• Create control plane and worker nodes to the same zone as LB's, but different subnets
• Check CCM behavior in the discover subnets

Nice to have:

• Compose CloudFormation template as stack set templates to reuse blocks

Engineering referece:

• This spikes validates the scenario covered by Enhancement Proposal 1634 : https://github.com/openshift/enhancements/pull/1634
  • the cloudformation templates can be used to validate the feature on installer too, let's document it to help developers to validate it quickly, both on CI or locally when developing the feature.
  • Suggest to use CloudFormation stackset to create templates for variants BYO VPC (two additional subnets in same AZ), using modules, without duplicating cloudfomation logic.
    • Example stackset: https://github.com/mtulio/mtulio.labs/tree/devel/labs/ocp-install-iac/aws-cloudformation-templates 
    • Repo path that can be saved templates:
      • https://github.com/openshift-splat-team/installer-labs/tree/main/installer-upi/aws/cloudformation 
      • https://github.com/openshift/installer/tree/main/upi/aws
      •