Throughput regressed in uperf multi-thread pod to pod dataplane testing scenario compared to OCP 4.13

When running dataplane test for pod to pod scenario through uperf workload (with 2 threads configuration), we observed throughput regressed in 4.14 when compared to OCP 4.13

Throughput is (with uperf 2threads 1024 packet size configuration) - 4080 Mbps in 4.14 (OVN-IC) vs 4880 Mbps in 4.13 (OVN).

We see same throughout between the environments when running uperf with 1 thread i.e 4930 Mbps in 4.14 (OVN-IC) vs 4940 in 4.13 (OVN).

We can find more results here https://docs.google.com/spreadsheets/d/1CZKKljsozQ_LeLnq5Jk1bjowG4PVK35PiHHbjZeGG0k/edit?usp=sharing 

OCP 4.14 4.14.0-0.nightly-2023-10-06-234925
OCP 4.13 4.13.0-0.nightly-2023-10-08-090900

Always

1. Deploy a 2 worker node OCP environment
2. Run e2e po2pod dataplane test
git clone https://github.com/cloud-bulldozer/e2e-benchmarking
cd e2e-benchmarking/workloads/network-perf
WORKLOAD=pod2svc ./run.sh 

16% decrease in throughput with uperf 2 threads

Throughput should be closer to 4.13 OVN environment.

Same configuration is used in both the environments i.e
1. worker count - 24 (2 workers sufficient for this testing. This test makes sure that pods are scheduled on different worker nodes before it runs uperf tool)
2. master and worker instance types - m5.2xlarge