Check scripts for the on-premise keepalived static pods only check the haproxy, which only directs to kube-apiserver pod. They do not take into consideration whether the control plane node has a healthy machine-config-server.

This may be a problem because, in a failure scenario, it may be required to rebuild nodes and machine-config-server is required for that (so that ignitions are provided).

One example is the etcd restore procedure (https://docs.openshift.com/container-platform/4.12/backup_and_restore/control_plane_backup_and_restore/disaster_recovery/scenario-2-restoring-cluster-state.html). In our case, the following happened (I'd suggest reading the recovery procedure before this sequence of events):
- Machine config server was healthy in the recovery control plane node but not in the other hosts. 
- At this point, we can only guarantee the health of the recovery control plane node because the non-recovery ones are to be replaced and must be removed first from the cluster (node objects deleted) so that OVN-Kubernetes control plane can work properly.
- The keepalived check scripts were succeeding in the non-recovery control plane nodes because their haproxy pods were up and running. That is fine from kube-apiserver point of view, actually, but does not take machine config server into consideration.
- As the machine-config-server was not reachable, provision of the new masters required by the procedure was impossible.

In parallel to this bug, I'll be raising another bug to improve the restore procedure. Basically, asking to stop the keepalived static pods on the non-recovery control plane nodes. This would prevent the exact situation above.

However, there are other situations where machine-config-server pods may be unhealthy and we should not just be manually stopping keepalived. In such cases, keepalived should take machine-config-server into consideration.

4.12

Under some failure scenarios, where machine-config-server is not healthy in one control plane node.

1. Try to provision new machine for recovery.
2.
3.

Machine-config-server not serving because keepalived assigned the VIP to one node that doesn't have a working machine-config-server pod.

Keepalived to take machine-config-server health into consideration while doing failover.

Possible ideas to fix:
- Create a check script for the machine-config-server check. It may have less weight than the kube-apiserver ones.
- Include machine-config-server endpoint in the haproxy of the kube-apiservers.