1. Proposed title of this feature request

Secondary Network Interfaces Support in Machine API (MAPI) and Cluster API (CAPI) for IBM Cloud VPC

2. What is the nature and description of the request?

Current machine spec in MAPI and CAPI for IBM Cloud VPC only has a primary network interface spec, but there is no secondary interface spec, so that we cannot create Machine resource that has multiple interfaces through MAPI or CAPI. Of course IBM Cloud VPC API itself supports secondary interfaces creation through `NetworkInterfaces` spec. However, the current machine spec in MAPI and CAPI only implements a small set of IBM Cloud VPC API; in terms of network spec, it prepares `PrimaryNetworkInterface` spec but it does not prepare `NetworkInterfaces` spec. As a result, it is not easy to scale out multiple-interfaces-attached machines in OpenShift on IBM Cloud VPC. In this RFE, we would like to add `NetworkInterfaces` spec into MAPI and CAPI for IBM Cloud. 

3. Why does the customer need this? (List the business requirements here)

We are building and managing a large scale GPU OpenShift Cluster in IBM Cloud. In this cluster, we are enabling RoCE/GDR to maximize network performance between multiple GPUs and worker nodes. The primary network interface is always used for OpenShift control plane (ovn-kube or openshift-sdn), so we have to attach additional interfaces to exploit RoCE/GDR capabilities in the node. Even if we cannot utilize RoCE/GDR (i.e. just using TCP connection), adding multiple interfaces to the node is beneficial to maximize network performance. Thus, customers who want to build a similar large scale GPU or non-GPU OpenShift cluster in IBM Cloud will face the same problem.

OpenShift itself supports to add multiple interfaces to pods via Multus CNI, but we cannot easily launch the IBM Cloud VPC worker nodes with multiple network interfaces based on the above limitation. We have several options to overcome this issue. 

Option 1: customers replace the default machine controller with a custom machine controller that implements `NetworkInterfaces` spec by themselves. 
Option 2: customers scale out machines via MachineSet. Once machines are getting ready, we can manually manipulate IBM Cloud VPC API to attach secondary interfaces to the newly added nodes one-by-one. 

Both of approaches are feasible but these would not be recommended nor scalable approaches in the most of cases. 

4. List any affected packages or components.

Basically there are two components that we need to update. 

1) IBMCloudMachineProviderSpec (MAPI)
Here is the pointer for MAPI where we have to extend the spec.
https://github.com/openshift/machine-api-provider-ibmcloud/blob/main/pkg/apis/ibmcloudprovider/v1/ibmcloudproviderconfig_types.go#L30-L92

The original spec has only primary network type but we would like to add networkinterfaces[] type to support additional secondary interfaces in the node. (actually, we have been internally developed the customer machine controller with the updated spec that inclues networkinterfaces though)

2) IBMVPCMachineSpec (CAPI)
Same as MAPI, only primary network interface spec is available even in CAPI.  
https://doc.crds.dev/github.com/kubernetes-sigs/cluster-api-provider-ibmcloud/infrastructure.cluster.x-k8s.io/IBMVPCMachine/v1beta2@v0.4.0-alpha.2