Hello everyone 

We are experiencing up to 15% network performance degradation, which can be observed mainly with small and medium tcp message sizes.

The most considerable iperf3 performance degradation on Emerald Rapids has occurred between kernels 6.3 and 6.4:

5.14.0-427.13.1.el9_4: 189 Mbits/sec

6.2.0-63.eln126: 186 Mbits/sec

6.3.0-63.eln126: 183 Mbits/sec
6.4.0-59.eln127: 171 Mbits/sec

git log -S copy_user_short_string v6.3..v6.4

shows these two commits from Linus:

commit 427fda2c8a4977d9dbd9bc108bbe6e21ec84648d
    x86: improve on the non-rep 'copy_user' function

commit adfcf4231b8cbc2d9c1e7bfaa965b907e60639eb
    x86: don't use REP_GOOD or ERMS for user memory copies

I think the second commit (adfcf4231b8cbc2d9c1e7bfaa965b907e60639eb) might be the one you also found in your analysis. 

I am wondering if "rep movsb" is actually slower than the manual copy for the small buffer sizes, at least on Emerald Rapids.  I'll build a kernel to test that theory (along with the other fixes).

Problem is visible only with selinux=enforcing  spectre_bhi=on and mitigations enabled. With disabled selinux or spectre_bhi=off performance is equal both on rhel-9.5 and rhel-10

Please note that this Jira is spin-off original jira

https://issues.redhat.com/browse/RHEL-40027

which happened to be mix up of two diferent performance affecting issues and also become long and bloated.

Feel free to drop a comment if you are in doubt. 

Thanks

Adam