setup cluster IPI on nutanix. 2/3 master cannot be ready due to API VIP cannot be accessed

# oc get node
NAME                             STATUS     ROLES                  AGE     VERSION
z-nutanix14-fmxt5-master-0       NotReady   control-plane,master   3h19m   v1.27.4+2c287eb
z-nutanix14-fmxt5-master-1       NotReady   control-plane,master   3h19m   v1.27.4+2c287eb
z-nutanix14-fmxt5-master-2       Ready      control-plane,master   3h19m   v1.27.4+2c287eb
z-nutanix14-fmxt5-worker-tlpqb   Ready      worker                 3h2m    v1.27.4+2c287eb


On this cluster, master-2 get the api vip 10.0.132.16

# ip a s br-ex
5: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 50:6b:8d:d5:01:1a brd ff:ff:ff:ff:ff:ff
    inet 10.0.132.217/24 brd 10.0.132.255 scope global dynamic noprefixroute br-ex
       valid_lft 2147471461sec preferred_lft 2147471461sec
    inet 169.254.169.2/29 brd 169.254.169.7 scope global br-ex
       valid_lft forever preferred_lft forever
    inet 10.0.132.16/32 scope global br-ex:vip
       valid_lft forever preferred_lft forever
    inet6 fe80::7de6:77fd:8917:3903/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever


on master-0 and master-1: apivip cannot be accessed

[core@z-nutanix14-fmxt5-master-1 ~]$ ping 10.0.132.16
PING 10.0.132.16 (10.0.132.16) 56(84) bytes of data.
From 10.0.132.103 icmp_seq=1 Destination Host Unreachable
From 10.0.132.103 icmp_seq=2 Destination Host Unreachable
From 10.0.132.103 icmp_seq=3 Destination Host Unreachable


However the api pod is only running on master-0

[root@z-nutanix14-fmxt5-master-2 ~]# oc get pod -n openshift-kube-apiserver -o wide
NAME                                              READY   STATUS        RESTARTS   AGE     IP            NODE                         NOMINATED NODE   READINESS GATES
installer-3-z-nutanix14-fmxt5-master-0            0/1     Completed     0          3h18m   10.128.0.28   z-nutanix14-fmxt5-master-0   <none>           <none>
installer-4-z-nutanix14-fmxt5-master-1            1/1     Terminating   0          3h16m   10.129.0.60   z-nutanix14-fmxt5-master-1   <none>           <none>
installer-5-z-nutanix14-fmxt5-master-1            0/1     Terminating   0          3h15m   <none>        z-nutanix14-fmxt5-master-1   <none>           <none>
installer-6-z-nutanix14-fmxt5-master-1            0/1     Pending       0          3h6m    <none>        z-nutanix14-fmxt5-master-1   <none>           <none>
kube-apiserver-guard-z-nutanix14-fmxt5-master-0   1/1     Running       0          3h17m   10.128.0.32   z-nutanix14-fmxt5-master-0   <none>           <none>
kube-apiserver-z-nutanix14-fmxt5-master-0         5/5     Running       0          3h17m   10.0.132.76   z-nutanix14-fmxt5-master-0   <none>           <none>
 

4.14.0-0.nightly-2023-09-12-195514

not always

1. IPI on nutanix
2.
3.