Rather than creating an ingresscontroller per az, there would be a loadbalancer created per broker.  There primary motivation is that it eliminates the need to size/scale the ingress router replicas.  More than likely the enterprise case will not restrict the ingress/egress bandwidth, thus brokers may be able to use the full bandwidth of their nodes.  This can result in needing 1 router replica of a similar node bandwidth per 2-3 brokers.  Unless all of those resources come from existing nodes on a customers cluster, it will be cheaper to have load balancer per broker.

Several challenges / considerations:

• We currently use the ha proxy metrics to determine the charge back for ingress/egress.  For enterprise it's not expected that we'll bill for this.
  • There is also an ongoing effort to move away from HAProxy metrics for ingress/egress billing: MGDSTRM-10326
• This will bifurcate the fleet shard operator's configuration of the kafka listeners and the logic managing ingress controllers as this change is not applicable to the managed case.  This will likely require capturing in the managedkafkaagent spec what type of install is being managed.
• This is also highly related to MGDSTRM-9651, which would also eliminate the need for ingresscontrollers, but restricts access to the brokers from only within the OSD instance.
• The LoadBalancer service configuration will likely need to vary by platform - there are specific annotations per platform, such as https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.2/guide/service/annotations/#lb-scheme
• The number of load-balancers per platform account may need adjusted - https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-limits.html