DaemonSet "/openshift-network-diagnostics/network-check-target" is not available

Seeing intermittently during cluster installs

Network operator stuck in Progressing with 

network                       4.12.0-0.nightly-2022-10-25-210451   True        True          False      117m    DaemonSet "/openshift-network-diagnostics/network-check-target" is not available (awaiting 1 nodes)


MG: http://shell.lab.bos.redhat.com/~anusaxen/must-gather.local.5450303633101217331/

iptables-save on master-2 node - http://shell.lab.bos.redhat.com/~anusaxen/iptables-save


pod events
Events:
  Type     Reason                  Age                   From               Message
  ----     ------                  ----                  ----               -------
  Normal   Scheduled               129m                  default-scheduler  Successfully assigned openshift-network-diagnostics/network-check-target-gnld6 to qe-anurag114e-9xkz4-master-2.c.openshift-qe.internal
  Warning  FailedMount             128m (x7 over 129m)   kubelet            MountVolume.SetUp failed for volume "kube-api-access-kfg5s" : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
  Warning  NetworkNotReady         128m (x18 over 129m)  kubelet            network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?
  Warning  ErrorAddingLogicalPort  127m (x2 over 127m)   controlplane       addLogicalPort failed for openshift-network-diagnostics/network-check-target-gnld6: unable to parse node L3 gw annotation: k8s.ovn.org/l3-gateway-config annotation not found for node "qe-anurag114e-9xkz4-master-2.c.openshift-qe.internal"
  Normal   AddedInterface          127m                  multus             Add eth0 [10.130.0.3/23] from ovn-kubernetes
  Warning  ProbeError              9m (x16 over 71m)     kubelet            Readiness probe error: Get "http://10.130.0.3:8080/": dial tcp 10.130.0.3:8080: i/o timeout (Client.Timeout exceeded while awaiting headers)
body:
  Warning  ProbeError  4m (x717 over 126m)  kubelet  Readiness probe error: Get "http://10.130.0.3:8080/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
body:

4.12.0-0.nightly-2022-10-25-210451

rare

1.Install OCP with OVNKubernetes with HO enabled

defaultNetwork:
    type: OVNKubernetes
    ovnKubernetesConfig:
      hybridOverlayConfig:
        hybridClusterNetwork: []

2.
3.

Installation stuck due to network-check-target issue 

Installation should succeed

Will add additional logs