Feature Overview (aka. Goal Summary)  

Customers with large numbers of geographically dispersed locations want a container management solution with a two node footprint.  They require high availability but even "cheap" third nodes represent a significant cost at this scale.  

Goals (aka. expected user outcomes)

Two-node clustering is a solved problem in the traditional HA space. The goal of this feature is to introduce existing RHEL technologies into OpenShift to support a true two-node topology. This requires fencing to ensure node recovery. Hence the name: Two Node OpenShift with Fencing (TNF).

Requirements (aka. Acceptance Criteria):

1. Provide a true two node OCP deployment
2. Support workload in active/passive mode..i.e.. single instance pod where the pods from the failed node are restarted on the 2nd node in a timely manner, or a 2nd pod is already running but passive, ready to take over if the  1st pod fails (e.g.:  psql database in an active/passive setup). This sees CPU utilisation ~50% max.
3. Support workload in active/active workload. Both nodes are load sharing and they are loaded by design to be about 60-75% at full capacity - during failure there is an expectation of service degradation but not service down completely - So if one node fails the other node operates at close to 100% 
4. Either both nodes have a fencing device (BMC via redfish, IPMI etc, UPS via serial port), or there is a dedicated direct cross over cable between the nodes to drastically reduce the risk of split brain. BMC via redfish at TP only, other fencing devices probably post-GA. 
5. <60s failover time: if the leading node goes down, the remaining nodes takes over and gains operational state (writable) in less then 60s. Exact parameters (heartbeat interval, missed heartbeats etc. needs to be configurable by users, e.g. to operate on a less aggressive timeline if required (avoid unnecessary failovers due to blip/flukes).  (To be refined after initial numbers observed during TP testing) 
6. No shared storage available between nodes required as fencing device. 
7. Be able to scale out to a true three node compact cluster as day2 operation. (Stretch goal, not required for MVP, but constraint to be kept in mind during design and implementation). The resulting cluster should have 3 node etcd quorum, and the same architecture/support statement as a freshly installed 3 node compact cluster. Out of scope for TP, and probably even for GA, as OCP currently does not control plane topology changes.
8. Be able to add worker nodes to a two node cluster with fencing as day2 operation. Like we do support with SNO+worker nodes (stretch goal, no required for TP or GA)
9. Solution fullfills the[ k8s-etcd contract|https://docs.google.com/document/d/1NUZDiJeiIH5vo_FMaTWf0JtrQKCx0kpEaIIuPoj9P6A/edit#heading=h.tlkin1a8b8bl], so that layer mechanism like Leases  work correctly. 
10. support full recovery of the workload when the node comes back online after restoration - total time <15 mins
11. X86_64 only on initial release, AARCH64 might be added later. 
12. Added: ability to track TNF usage in the fleet of connected clusters via OCP telemetry data (e.g. number of clusters with TNF topology)
13. Added: Be able to install OCP Virt and run VMs with node local storage (e.g. LSO or LVMS) on both nodes.  Deferred to GA 

Questions to Answer (Optional):

1. ...

 Out of Scope

1. Storage driver providing RWX shared storage
2. ...

 Background

• Two node support is in high demand by telco, industrial and retail customers.
• StarlingX  supports true  two node (docs)

 Customer Considerations

Telco Customer requirements:

2-node HA control-plane requirements for Telco

Documentation Considerations

Topology needs to be documented, esp. The requirements of the arbiter node.

Interoperability Considerations

1. OCP Virt needs to be explicitly tested on this scenario to support VM HA (live migration, restart on other node)