Multus CNI for Microshift MVP

Goal:

Enable and support Multus CNI for microshift.

Background:

Customers with advanced networking requirement need to be able to attach additional networks to a pod, e.g. for high-performance requirements using SR-IOV or complex VLAN setups etc.

Requirements:

1. opt-in approach: customers can add multus if needed, e.g. by installing/adding "microshift-networking-multus" rpm package to their installation.
2. if possible, it would be good to be able to add multus  an existing installation. If that requires a restart/reboot, that is acceptable. If not possible, it has to be clearly documented. 
3. it is acceptable that once multus has been added to an installation, it can not be removed. If removal can be implemented easily, that would be good. If not possible, then it has to be clearly documented.  
4. Regarding additional networks:
   1. As part of the MVP, the Bridge plugin must be fully supported  
   2. As stretch goal,  macvlan and ipvlan plugins  should be supported
   3. Other plugins, esp. host device and sr-iov are out of scope for the MVP, but will be added with a later version.
   4. Multiple additional networks need to be configurable, e.g. two different bridges leading to two different networks, each consumed by different pods.
5. IP V6 with bridge plugin. Secondary NICs passed to a container via the bridge plugin should work with IP V6, if the consuming pod does support V6. See also "out of scope"
6. Regarding IPAM CNI support for IP address provisioning, static and DHCP must be supported.

Documentation:

1. In the existing "networking" book, we need a new chapter "4. Multiple networks". It can re-use a lot content from OCP doc "https://docs.openshift.com/container-platform/4.14/networking/multiple_networks/understanding-multiple-networks.html", but needs an extra chapter in the beginning "Installing support for multiple networks"{}

Testing:

1. A simple "smoke test" that multus can be added, and a 2nd nic added to a pod (e.g. using host device) is sufficient. No need to replicate all the multus tests from OpenShift, as we assume that if it works there, it works with MicroShift.

Customer Considerations:

• This document contains the MVP requirements of a MicroShift EAP customer that need to be considered.

Out of scope:

• Other plugins, esp. host device and sr-iov are out of scope for the MVP, but will be added with a later version.
• IP V6 support with OVN-K. That is scope of  feature OCPSTRAT-385