Feature Overview (aka. Goal Summary)  

Customers with large numbers of geographically dispersed locations want a container management solution with a two node footprint.  They require high availability but even "cheap" third nodes represent a significant cost at this scale.  

Goals (aka. expected user outcomes)

Two-node clustering is a solved problem in the traditional HA space. The goal of this feature is to introduce existing RHEL technologies into OpenShift to support a true two-node topology. This requires fencing to ensure node recovery. Hence the name: Two Node OpenShift with Fencing (TNF).

Requirements (aka. Acceptance Criteria):

1. Provide a true two node OCP deployment
2. Support workload in active/passive mode..i.e.. single instance pod where the pods from the failed node are restarted on the 2nd node in a timely manner, or a 2nd pod is already running but passive, ready to take over if the  1st pod fails (e.g.:  psql database in an active/passive setup). This sees CPU utilisation ~50% max.
3. Support workload in active/active workload. Both nodes are load sharing and they are loaded by design to be about 60-75% at full capacity - during failure there is an expectation of service degradation but not service down completely - So if one node fails the other node operates at close to 100% 
4. Both nodes have a fencing device. BMC via redfish/IPMI on GA only. Other fenciing devices, e.g.  power switches controlable via serial port might be added later or on demand. See also requirement #11
5. <60s failover time: if the leading node goes down, the remaining nodes takes over and gains operational state (writable) in less then 60s. Exact parameters (heartbeat interval, missed heartbeats etc. needs to be configurable by users, e.g. to operate on a less aggressive timeline if required (avoid unnecessary failovers due to blip/flukes)..
6. No shared storage available between nodes required. 
7. Be able to scale out to a true three node compact cluster as day2 operation. (Stretch goal, not required for MVP, but constraint to be kept in mind during design and implementation). The resulting cluster should have 3 node etcd quorum, and the same architecture/support statement as a freshly installed 3 node compact cluster
8. Be able to add worker nodes to a two node cluster with fencing as day2 operation. Like we do support with SNO+worker nodes (stretch goal, no required for MVP)
9. Solution fullfills the[ k8s-etcd contract|https://docs.google.com/document/d/1NUZDiJeiIH5vo_FMaTWf0JtrQKCx0kpEaIIuPoj9P6A/edit#heading=h.tlkin1a8b8bl], so that layer mechanism like Leases  work correctly. 
10. support full recovery of the workload when the node comes back online after restoration - total time <15 mins
11. Not all edge devices can have a fencing devices. For those situations, a configuration with a dedicated direct cross over cable between the nodes can be setup. This drastically reduces the risk of split brain situations. In this configuration, if a node does not react to a ping via the direct cross over cable, it is considered to be fenced successfully. --> Removed from the scope, see comments below for reasons.

Questions to Answer (Optional):

1. ...

Out of Scope

1. Storage driver providing RWX shared storage
2. ...

Background

• Two node support is in high demand by telco, industrial and retail customers.
• StarlingX  supports true  two node (docs)

Customer Considerations

Telco Customer requirements:

2-node HA control-plane requirements for Telco

Documentation Considerations

Topology needs to be documented, esp. The requirements of the arbiter node.

Interoperability Considerations

1. OCP Virt needs to be explicitly tested on this scenario to support VM HA (live migration, restart on other node)