pods that are HostNetworked on nodes using routingViaHost:true ipForwarding: global cannot route to default kubernetes service IP

Description of problem:

• We're seeing that on two baremetal nodes where `routingViaHost=true` is enabled (with ipForwarding set properly as Global) the following problem:

• They have set NodeIP Hint to force OVN to bind to a secondary interface at `bond1.2039 `

• we're seeing that specific pods that are hostNetworked can't reach the default kubernetes service IP address; and are failing to initialize as a result (CLBO).

~~~
F0120 03:20:42.221327 879146 driver.go:131] failed to get node "wb02.pdns-edtnabtf-arch01.nfvdev.tlabs.ca" information: Get "https://192.168.0.1:443/api/v1/nodes/wb02.pdns-edtnabtf-arch01.nfvdev.tlabs.ca": dial tcp 192.168.0.1:443: i/o timeout
~~~

other pods on affected node with above config can hit the target service however, pods that are hostNetworked appear to be failing:

$ oc get pod csi-rbdplugin-kpz7n -o yaml | grep hostNetwork
hostNetwork: true

Version-Release number of selected component (if applicable):

4.14

• bare-metal  

How reproducible:

• new cluster, every time

Steps to Reproduce:

We have redeployed the cluster. and have
routingViaHost and ipForwarding both enabled.

We also pushed out a NODEIP_HINT configuraiton to all the nodes to make sure SDN is overlayed on the correct interface.

Default gateway has been moved to bond1.2039on the 2 x baremetal worker nodes.

wb01

wb02

observe that hostNetworked pods crashloop backoff
 

Actual results:

• hostnetworked pods cannot call the default kube service address  

Expected results:

• hostnetworked pods should be able to do so.
   

Additional info:

See the first comment for data samples + must-gathers + sosreports