IngressController API should support AWS EIPs

Feature Overview (aka. Goal Summary)  

An elevator pitch (value statement) that describes the Feature in a clear, concise way.  Complete during New status.

Users/customers of OpenShift on AWS (ROSA) want to use static IPs (and therefore AWS Elastic IPs) so that they can configure appropriate firewall rules. They want the default AWS Load Balancer that they use (NLB) for their router to use these EIPs.

Kubernetes does define a service annotation for configuring EIP
allocations, which should work in OCP:

    // ServiceAnnotationLoadBalancerEIPAllocations is the annotation used on the
    // service to specify a comma separated list of EIP allocations to use as
    // static IP addresses for the NLB. Only supported on elbv2 (NLB)
    const ServiceAnnotationLoadBalancerEIPAllocations = "service.beta.kubernetes.io/aws-load-balancer-eip-allocations"

Source: https://github.com/openshift/kubernetes/blob/eab9cc98fe4c002916621ace6cdd623afa519203/staging/src/k8s.io/legacy-cloud-providers/aws/aws.go#L227-L230

We do not provide an API field on the IngressController API to configure
this annotation.  

This is a feature request to enhance the IngressController API to be able to support static IPs from install time and upon reconfiguration of the router (may require destroy/recreate LB)

Goals (aka. expected user outcomes)

The observable functionality that the user now has as a result of receiving this feature. Complete during New status.

1. User can provision EIPs and use them with an IngressController via NLB
2. User can ensure EIPs are used with NLB on default router at install time
3. User can reconfigure default router to use EIPs

Requirements (aka. Acceptance Criteria):

A list of specific needs or objectives that a feature must deliver in order to be considered complete.  Be sure to include nonfunctional requirements such as security, reliability, performance, maintainability, scalability, usability, etc.  Initial completion during Refinement status.

1. User can use existing EIPs (one per subnet) for cluster install or router configuration
2. Router NLB and DNS can be inspected to have those (and only those) EIPs attached to the associated ingress.
3. EIPs will survive, be detached and available upon cluster deletion for subsequent cluster reuse

Use Cases (Optional):

Include use case diagrams, main success scenarios, alternative flow scenarios.  Initial completion during Refinement status.

Questions to Answer (Optional):

Include a list of refinement / architectural questions that may need to be answered before coding can begin.  Initial completion during Refinement status.

Out of Scope

High-level list of items that are out of scope.  Initial completion during Refinement status.

1. Management of EIPs (provision/cleanup) outside of selection/association with IngressController
2. Static IP usage with NLBs for API server

Background

Provide any additional context is needed to frame the feature.  Initial completion during Refinement status.

Allowing those EIPs to be provisioned outside and survive the cluster reconfiguration or even creation/deletion, it helps support our "don't treat clusters as pets" philosophy. It also removes additional burden for them to wrap the cluster or our managed service with yet another global IP service that should be unnecessary and bring more complexity. That aligns precisely with their interest in the functionality and we should pursue making this seamless.

Customer Considerations

Provide any additional customer-specific considerations that must be made when designing and delivering the Feature.  Initial completion during Refinement status.

Documentation Considerations

Provide information that needs to be considered and planned so that documentation will meet customer needs.  Initial completion during Refinement status.

Interoperability Considerations

Which other projects and versions in our portfolio does this feature impact?  What interoperability test scenarios should be factored by the layered products?  Initial completion during Refinement status.