1. Proposed title of this feature request

Network MTU on Installer  

2. What is the nature and description of the request?

AWS Edge locations like Local Zones [1] and Wavelengths [2] and AWS Direct Connect [3] connected private data center locations require different network Maximum Transmission Unit (MTU). This feature will allow customers who bring their own VPC (BYOVPC) to be able to customize cluster-wide MTU in accordance with their needs for running workloads across different AWS locations and their connectivity needs with any private data centers. 

ROSA Clusters - both ones set up in primary regions' AZs and the ones set up in Local Zones - do not provide the knowledge of how these clusters are connected to rest of customers' infrastructure including other VPCs that are are connected/managed centrally using Transit Gateways; connected to their data centers using either VPNs with public/internet traffic or AWS Direct Connect with private AWS traffic. 

Because of these conditions - especially in the BYOVPC- , perhaps we provide an option at the install time for "power" users through to override the cluster-wide MTU? I'd expect that once we have dynamic/route based MTU in OCP/ROSA clusters, this cluster-wide MTU will be an inferior option in performance sense and we can consider deprecating.  OCP 4.13 and 4.14 will cover for BYOVPC and IPI-VPC cases respectively where customers include local zone subnet during install time; but, when customers don't include the LZ subnet at install time but want to add a LZ subnet-based machinepool, they have to do day-2 modification to lower the cluster MTU. Giving an install-time option will mitigate this risk and philosophically, makes customers "own" the decision of lowering the MTU and accepting to the lowered performance. 

3. Why does the customer need this? (List the business requirements here)

Customers will want to add a AWS Local Zone based machine-pool to a ROSA cluster that was initially set up only using the BYOVPC subnets created in parent region's Availability Zones. Such clusters a) will have the default 9001 byte MTU and b) require customers to lower the cluster-wide MTU to 1300 bytes that is supported for traffic between parent  region AZ and AWS Local Zone. 

4. List any affected packages or components.

Installer, OVN/SDN

References:

[1] How AWS Local Zones Work? https://docs.aws.amazon.com/local-zones/latest/ug/how-local-zones-work.html 

 [2] How AWS Wavelength Works? https://docs.aws.amazon.com/wavelength/latest/developerguide/how-wavelengths-work.html

[3] MTU requirements for private and virtual interfaces in AWS Direct Connect https://docs.aws.amazon.com/directconnect/latest/UserGuide/set-jumbo-frames-vif.html 

CC: mak.redhat.com ddharwar@redhat.com rh-ee-adejong