When the MTU is not set in the `169.254.169.1` route the sporadic failures in the Host Pod to Host Pod on the same node via NodePort service are caused. More often failures occur then it passing. However when the MTU is set to 1400 on the route, we see this issue gets resolved completely and works. The commit that broke this was in OVN-Kubernetes: https://github.com/openshift/ovn-kubernetes/commit/2e052072bc288bd16f0ba5fc95ac620bf95b36f8

A failure is described as no traffic being observed at the iperf3 server or tcpdump on br-ex.

In all cases OvS Hardware Offload is enabled.

This is the log for when the test is passing with MTU 1400 on the route (includes tcpdump and dump flows): https://drive.google.com/file/d/1sc_6hMZJ3kSxNWVLltb2zA0FPwCAjWGp/view?usp=share_link

This is the log for when the test is failing (includes tcpdump and dump flows): https://drive.google.com/file/d/1_esy3L_q7wyYz3ejO3rvK0DWXZDE_61f/view?usp=share_link

Related links: 

https://github.com/ovn-org/ovn-kubernetes/issues/3419

https://bugzilla.redhat.com/show_bug.cgi?id=2169839

https://github.com/openshift/ovn-kubernetes/commit/2e052072bc288bd16f0ba5fc95ac620bf95b36f8

https://issues.redhat.com/browse/OCPBUGS-7609