Description of problem:

We currently support encryption of internal and admin openstack endpoints at application layer, for which an IdM server needs to be deployed and integrated in order to manage the large amount of certificates required (each endpoint on each overcloud node).

Large customers with internal CA and DNS infrastructure already deployed (especially those using dedicated appliances) also feel against deploying IdM just for the purpose of serving overcloud deployment, and perceive it as a single point of failure.

As an alternative to L7 encryption, the RHOSP control plane could be deployed on top of macsec interfaces, which would take care of L2 encryption at the OS, simplifying the deployment and configuration of RHOSP itself. This way, all RHOSP control plane traffic would be encrypted, including not only the services for which we already support certificate-based encryption, but also services that don't, such as Memcached (currently tech-preview), Keystone middleware and Django. Macsec standard is included and supported out-of-the-box in RHEL8 and 9.

This RFE is to support the deployment of macsec interfaces by TripleO as the foundation of the control plane subnets.

How reproducible:

This change can be done today by deploying an overcloud with bridged interfaces, then manually replacing the slaves (regular nics) with macsec interfaces.

Steps to Reproduce:
1. Deploy undercloud, set MTU of provisioning network to 1468 in advance
2. Deploy overcloud using a bridge for the control plane. Set MTU of all internal vlans to 1468.
3. Stop fencing

On director and each overcloud node:

4. Un-enslave nic from control plane bridge (br-ctlplane or else)
5. Set MTU of nic to 1500
6. Create macsec interface
7. Add TX key and 1 RX key for each other overcloud node
8. Enslave macsec interface to bridge

Actual results:

All traffic between director and overcloud nodes, as well as between the overcloud nodes themselves is encrypted:

~~~
[root@overcloud-controller-0 ~]# tcpdump -i enp2s0 -vv | head -20
dropped privs to tcpdump
tcpdump: listening on enp2s0, link-type EN10MB (Ethernet), capture size 262144 bytes
13:04:26.810962 52:54:01:72:b1:47 (oui Unknown) > 52:54:01:f4:17:cc (oui Unknown), ethertype Unknown (0x88e5), length 98:
0x0000: 2c00 0020 c800 5254 0172 b147 0001 9468 ,.....RT.r.G...h
0x0010: cd54 31de dde0 5aff afae 8f22 256f 070f .T1...Z...."%o..
0x0020: 8b1c 109c 80e7 78f7 68b1 b415 f8f2 0aae ......x.h.......
0x0030: 3ed5 bea5 0cbc fd84 b50b c800 6b64 4aa9 >...........kdJ.
0x0040: c34e e22a 4395 8384 72f1 8e8e f497 e9c2 .N.*C...r.......
0x0050: 2568 c62c %h.,
13:04:26.810980 52:54:01:72:b1:47 (oui Unknown) > 52:54:01:f4:17:cc (oui Unknown), ethertype Unknown (0x88e5), length 98:
0x0000: 2c00 0020 c801 5254 0172 b147 0001 9494 ,.....RT.r.G....
0x0010: f92d a91e e5c6 6066 18ea b3a4 bc4f b85d .-....`f.....O.]
0x0020: 04d2 7759 f6bd b2bf faae 9514 e1ed 2d3b ..wY..........-;
0x0030: 29bd c889 204b 0f66 9457 4021 6836 fa43 )....K.f.W@!h6.C
0x0040: b0b4 2bce bd87 aa77 71a6 3e84 6c5c 0746 ..+....wq.>.l\.F
0x0050: 9414 b711 ....
13:04:26.811051 52:54:01:f4:17:cc (oui Unknown) > 52:54:01:72:b1:47 (oui Unknown), ethertype Unknown (0x88e5), length 98:
0x0000: 2c00 0024 c42d 5254 01f4 17cc 0001 e849 ,..$.-RT.......I
0x0010: 9223 a181 076c 43b4 b667 8ecd 4d7a 28b7 .#...lC..g..Mz(.
0x0020: 2c50 fa0a 492f ae5a 3c44 9952 cbbf afa3 ,P..I/.Z<D.R....
0x0030: 870c f491 5caf 0c21 a72c e5c7 70b3 e8ae ....\..!.,..p...
0x0040: c02c 6a7f 967d 0a5c f52c 077d fad4 6f42 .,j..}.\.,.}..oB
tcpdump: Unable to write output: Broken pipe
[root@overcloud-controller-0 ~]#
~~~

However looking at the macsec interface, all traffic is visible:

~~~
[root@overcloud-controller-0 ~]# tcpdump -i macsec1 -vv | head -20
dropped privs to tcpdump
tcpdump: listening on macsec1, link-type EN10MB (Ethernet), capture size 262144 bytes
13:04:59.790263 IP (tos 0x0, ttl 64, id 23829, offset 0, flags [DF], proto TCP (6), length 100)
overcloud-controller-1.macsec.lab.51423 > overcloud-controller-0.macsec.lab.25672: Flags [P.], cksum 0x8321 (correct), seq 3584495374:3584495422, ack 2595989109, win 5969, options [nop,nop,TS val 1085936595 ecr 4114347458], length 48
13:04:59.790882 IP (tos 0x0, ttl 64, id 4524, offset 0, flags [DF], proto TCP (6), length 52)
overcloud-controller-0.macsec.lab.25672 > overcloud-controller-1.macsec.lab.51423: Flags [.], cksum 0x2ceb (correct), seq 1, ack 48, win 1412, options [nop,nop,TS val 4114347479 ecr 1085936595], length 0
13:04:59.809426 IP (tos 0xc0, ttl 255, id 18294, offset 0, flags [none], proto VRRP (112), length 40)
director.ctlplane.macsec.lab > 224.0.0.18: vrrp director.ctlplane.macsec.lab > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 103, authtype none, intvl 1s, length 20, addrs: 192.168.24.3
13:04:59.809466 IP (tos 0xc0, ttl 255, id 18294, offset 0, flags [none], proto VRRP (112), length 40)
director.ctlplane.macsec.lab > 224.0.0.18: vrrp director.ctlplane.macsec.lab > 224.0.0.18: VRRPv2, Advertisement, vrid 52, prio 103, authtype none, intvl 1s, length 20, addrs: 192.168.24.2
13:04:59.823187 IP (tos 0x0, ttl 64, id 8776, offset 0, flags [DF], proto TCP (6), length 52)
overcloud-controller-0.macsec.lab.memcache > overcloud-controller-2.macsec.lab.49976: Flags [.], cksum 0xe16b (correct), seq 1659315160, ack 1405736867, win 222, options [nop,nop,TS val 876043046 ecr 2359961051], length 0
13:04:59.823192 IP (tos 0x0, ttl 64, id 29077, offset 0, flags [DF], proto TCP (6), length 52)
overcloud-controller-0.macsec.lab.memcache > overcloud-controller-2.macsec.lab.50012: Flags [.], cksum 0x4d78 (correct), seq 3826278804, ack 2854966827, win 222, options [nop,nop,TS val 876043046 ecr 2359961051], length 0
13:04:59.823238 IP (tos 0x0, ttl 64, id 57606, offset 0, flags [DF], proto TCP (6), length 52)
overcloud-controller-0.macsec.lab.memcache > overcloud-controller-2.macsec.lab.50020: Flags [.], cksum 0xf08d (correct), seq 3094920941, ack 1039330693, win 222, options [nop,nop,TS val 876043046 ecr 2359961051], length 0
13:04:59.823283 IP (tos 0x0, ttl 64, id 51912, offset 0, flags [DF], proto TCP (6), length 52)
overcloud.ctlplane.localdomain.mysql > overcloud-controller-2.macsec.lab.33959: Flags [.], cksum 0x4fdb (correct), seq 589975137, ack 1970773866, win 746, options [nop,nop,TS val 1996389862 ecr 1738800647], length 0
13:04:59.823498 IP (tos 0x0, ttl 64, id 40508, offset 0, flags [DF], proto TCP (6), length 52)
overcloud-controller-2.macsec.lab.49976 > overcloud-controller-0.macsec.lab.memcache: Flags [.], cksum 0xd733 (correct), seq 1, ack 1, win 433, options [nop,nop,TS val 2359966107 ecr 875253974], length 0
13:04:59.823517 IP (tos 0x0, ttl 64, id 22546, offset 0, flags [DF], proto TCP (6), length 52)
overcloud-controller-2.macsec.lab.50012 > overcloud-controller-0.macsec.lab.memcache: Flags [.], cksum 0x43be (correct), seq 1, ack 1, win 316, options [nop,nop,TS val 2359966107 ecr 875253965], length 0
tcpdump: Unable to write output: Broken pipe
[root@overcloud-controller-0 ~]#
~~~

Basic functionality such as flavor, image, network, subnet, router server create/delete works, and I have no reason to think that other services would fail as basically RHOSP services are unaware of the L2 encryption. Even an overcloud deploy to update configuration completes successfully, as long as network configuration does not need to be changed.

Additional info:

Test configuration based on the following documentation:

• MACsec: Encryption for the wired LAN - Sabrina Dubroca
  https://legacy.netdevconf.info/1.1/proceedings/papers/MACsec-Encryption-for-the-wired-LAN.pdf

• MACsec: a different solution to encrypt network traffic - Sabrina Dubroca
  https://developers.redhat.com/blog/2016/10/14/macsec-a-different-solution-to-encrypt-network-traffic#

• RHEL8 - Securing Networks / Using MACsec to encrypt layer-2 traffic in the same physical network
  https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/securing_networks/assembly_using-macsec-to-encrypt-layer-2-traffic-in-the-same-physical-network_securing-networks