Description of problem:

We see a regression in one of our network performance tests. While simulating some traffic between Pod to Pod via Service using TCP, We see very less throughput on 8k packet size in 4.16 when compared to 4.15. The test doesn't use host network, instead it relies on overlay network.

How reproducible: Often. We have run 10 samples and we see this appearing consistently. Variance over multiple runs: https://gist.github.com/vishnuchalla/e87c778f48c1906dd59179a724a9df65 

Steps to Reproduce:

1. We run our performance tests through CI prow.

2. So please spin up a 9 nodes clusters and run this step: https://steps.ci.openshift.org/reference/openshift-qe-network-perf 

Actual results: Less throughput for 8k packet size: https://drive.google.com/file/d/1S0304-NDQxw6gpnZ-puA73j5Z5xAlQGz/view?usp=sharing

Expected results: Throughput needs to be same or higher than 4.15