This spike is to explore, discuss, and make a decision on the best way to deliver stable IP / DNS address for OVN SB connection to EDP nodes.

Context:

Right now, external IP addresses for SB cluster are not stable, which may result in different IP addresses on pod restarts. This breaks connectivity of OVN services running on EDP nodes.

There are several ways to achieve stable addressing, (perhaps more).

a. transform OVN StatefulSet into a DaemonSet, or a set of static Pods, that are assigned particular IP addresses that never change. This may mean enforcing a particular number of cluster nodes (which is in conflict with the dynamic scale-up / scale-down feature already implemented in ovn-operator).

Drawbacks for (a): manual management of pods and their IP address assignments. Requirement to run as DaemonSet means that PVCs are not supported, which implies using HostPaths - and privileged mode for the pods. (There is no technical reason why ovsdb-servers should run as privileged otherwise - these are just databases.)

b. expose the cluster through a MetalLB externalEndpoint.

Drawbacks for (b): when L2 mode is used for MetalLB (the current default), all traffic between EDP ovsdb clients and RAFT members will be proxied through MetalLB endpoint, which is not great for scale. This would be a regression, performance / scale wise, comparing to 17.1 where each client is configured with the list of all RAFT members.

c. expose the cluster through a MetalLB externalEndpoint BUT also require L3 mode.

Drawbacks for (c): BGP fabric assumed.

d. a variation on (b) and (c), e.g. running with L2 mode but allow to use L3 mode.

This spike should result in a decision on the path forward, that will be taken in Q4.