The customer is not able to complete an UPI installation because when the OVNKubernetes overlay network starts, the pods sitting on it are not able to reach the kube-apiserver endpoint which has a wrong address, not belonging to the machineNetwork.

The bootstrap node, along with all the other nodes, has 2 nics. The kube-apiserver container on the bootstrap node advertises the wrong IP address even though I tried to define the correct machineNetwork in the install-config.yaml, when the cluster creates the kubernetes service it has a wrong endpoint address.


I noticed that the kube-apiserver pod yaml on the bootstrap node contains the following argument:
--advertise-address=${HOST_IP}

The ${HOST_IP} variable is tied to the downward API "status.hostIP" value, this value is retrieved from the node IP in use by kubelet:
    env:
    - name: HOST_IP
      valueFrom:
        fieldRef:
          fieldPath: status.hostIP

Kubelet is started by the /usr/local/bin/kubelet.sh script that has no "--node-ip" or "--address" arguments:

/usr/bin/hyperkube \
  kubelet \
    --anonymous-auth=false \
    --container-runtime-endpoint=/var/run/crio/crio.sock \
    --runtime-request-timeout="${KUBELET_RUNTIME_REQUEST_TIMEOUT}" \
    --pod-manifest-path=/etc/kubernetes/manifests \
    --minimum-container-ttl-duration=6m0s \
    --cluster-domain=cluster.local \
    --cgroup-driver=systemd \
    --serialize-image-pulls=false \
    --v=2 \
    --volume-plugin-dir=/etc/kubernetes/kubelet-plugins/volume/exec \
    --pod-infra-container-image="${MACHINE_CONFIG_INFRA_IMAGE}"

This leads to possible problems and ambiguity when the bootstrap node has more than one IP assigned, since kubelet can choose a wrong address on its own.

In my opinion this issue can be solved in either two of the following ways or both of them at the same time:
a) by changing the way the kube-apiserver advertises its address, making it consistent with the machineNetwork value.
b) by starting kubelet with the "--node-ip" and "--address" arguments, consistent with the machineNetwork value.

By doing an UPI installation on a bootstrap node with more than one interface. Only reproducible on the customer environment so far, I've been not able to reproduce in lab.

1. Ignite the bootstrap node and reboot
2. once the bootstrap node starts kubelet and then the kubea-apiserver container, check the advertised address

kube-apiserver advertise an IP address from a wrong interface.

kube-apiserver should advertise an address consistent with the defined machineNetwork