Description of problem:

"sockperf pingpong multicast pkey/vlan" test case consistently fails when libvma is tested on MLX5 IB0 devices.

vma test results on rdma-perf-02/rdma-perf-03 & Beaker job J:7283201:
4.18.0-438.el8.x86_64, rdma-core-41.0-1.el8, mlx5, ib0, ConnectX-5 & mlx5_0
Result | Status | Test
-------------------------------------------------
PASS | 0 | sockperf pingpong multicast
PASS | 0 | sockperf throughput multicast
PASS | 0 | sockperf throughput unicast
PASS | 0 | sockperf pingpong unicast
PASS | 0 | sockperf (100 sockets) pingpong multicast
PASS | 0 | sockperf (100 sockets) pingpong unicast
FAIL | 1 | sockperf pingpong multicast pkey/vlan
PASS | 0 | sockperf pingpong unicast pkey/vlan
Checking for failures and known issues:
sockperf pingpong multicast pkey/vlan is NOT a known issue on any environment - consider filing a BZ

++++++++++++++++++++++

This is a regression where, with RHEL-8.7.0, the same test passes.

vma test results on rdma-perf-02/rdma-perf-03 & Beaker job J:7283202:
4.18.0-425.3.1.el8.x86_64, rdma-core-41.0-1.el8, mlx5, ib0, ConnectX-5 & mlx5_0
Result | Status | Test
-------------------------------------------------
PASS | 0 | sockperf pingpong multicast
PASS | 0 | sockperf throughput multicast
PASS | 0 | sockperf throughput unicast
PASS | 0 | sockperf pingpong unicast
PASS | 0 | sockperf (100 sockets) pingpong multicast
PASS | 0 | sockperf (100 sockets) pingpong unicast
PASS | 0 | sockperf pingpong multicast pkey/vlan <<<=============
PASS | 0 | sockperf pingpong unicast pkey/vlan
Checking for failures and known issues:
no test failures

Version-Release number of selected component (if applicable):

Clients: rdma-perf-03
Servers: rdma-perf-02

DISTRO=RHEL-8.8.0-20221120.2

+ [22-11-28 04:25:55] cat /etc/redhat-release
Red Hat Enterprise Linux release 8.8 Beta (Ootpa)

+ [22-11-28 04:25:55] uname -a
Linux rdma-perf-03.rdma.lab.eng.rdu2.redhat.com 4.18.0-438.el8.x86_64 #1 SMP Mon Nov 14 13:08:07 EST 2022 x86_64 x86_64 x86_64 GNU/Linux

+ [22-11-28 04:25:55] cat /proc/cmdline
BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-438.el8.x86_64 root=UUID=9190cd5d-4746-47b4-a4cd-c1d2806f9e12 ro intel_idle.max_cstate=0 processor.max_cstate=0 intel_iommu=on iommu=on console=tty0 rd_NO_PLYMOUTH intel_idle.max_cstate=0 intremap=no_x2apic_optout processor.max_cstate=0 reboot=acpi crashkernel=auto resume=UUID=816ac8a0-cba3-4d20-885c-8dac5877df91 console=ttyS1,115200n81

+ [22-11-28 04:25:55] rpm -q rdma-core linux-firmware
rdma-core-41.0-1.el8.x86_64
linux-firmware-20220726-110.git150864a4.el8.noarch

+ [22-11-28 04:25:55] tail /sys/class/infiniband/mlx5_0/fw_ver /sys/class/infiniband/mlx5_1/fw_ver
==> /sys/class/infiniband/mlx5_0/fw_ver <==
16.33.1048

==> /sys/class/infiniband/mlx5_1/fw_ver <==
16.33.1048

+ [22-11-28 04:25:55] lspci
+ [22-11-28 04:25:55] grep -i -e ethernet -e infiniband -e omni -e ConnectX
03:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
03:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
03:00.2 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
03:00.3 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
07:00.0 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5]
07:00.1 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5

+ [22-11-28 04:25:56] rpm -q libvma
libvma-9.6.4-1.el8.x86_64

How reproducible:
100%

Steps to Reproduce:

+ [22-11-28 04:34:27] export SERVER_IP2=172.31.2.182
+ [22-11-28 04:34:27] SERVER_IP2=172.31.2.182

1. Get both RDMA server & client hosts ready for libvma test with current sockperf, with RHEL-8.8 build as specified above on MLX4 IB0 devices
2. On server host, issue the following commands

LD_PRELOAD=libvma.so
timeout --preserve-status --kill-after=5m 3m sockperf server -i 172.31.2.182

3. On the client hosts, issue the following command

LD_PRELOAD=libvma.so
timeout --preserve-status --kill-after=5m 3m sockperf pp -i 172.31.2.182 -t 10 --msg-size=1472

Actual results:

[0m VMA INFO: ---------------------------------------------------------------------------
[0m[0m VMA INFO: VMA_VERSION: 9.6.4-1 Release built on Aug 17 2022 14:23:56
[0m[0m VMA INFO: Cmd Line: date ++ [%y-%m-%d %H:%M:%S]
[0m[0m VMA INFO: ---------------------------------------------------------------------------
[0m[0m VMA INFO: Log Level INFO [VMA_TRACELEVEL]
[0m[0m VMA INFO: ---------------------------------------------------------------------------
[0m+ [22-11-28 04:38:16] LD_PRELOAD=libvma.so
[0m VMA INFO: ---------------------------------------------------------------------------
[0m[0m VMA INFO: VMA_VERSION: 9.6.4-1 Release built on Aug 17 2022 14:23:56
[0m[0m VMA INFO: Cmd Line: date ++ [%y-%m-%d %H:%M:%S]
[0m[0m VMA INFO: ---------------------------------------------------------------------------
[0m[0m VMA INFO: Log Level INFO [VMA_TRACELEVEL]
[0m[0m VMA INFO: ---------------------------------------------------------------------------
[0m+ [22-11-28 04:38:16] timeout --preserve-status --kill-after=5m 3m sockperf pp -i 172.31.2.182 -t 10 --msg-size=1472
[0m VMA INFO: ---------------------------------------------------------------------------
[0m[0m VMA INFO: VMA_VERSION: 9.6.4-1 Release built on Aug 17 2022 14:23:56
[0m[0m VMA INFO: Cmd Line: timeout --preserve-status --kill-after=5m 3m sockperf pp -i 172.31.2.182 -t 10 --msg-size=1472
[0m[0m VMA INFO: ---------------------------------------------------------------------------
[0m[0m VMA INFO: Log Level INFO [VMA_TRACELEVEL]
[0m[0m VMA INFO: ---------------------------------------------------------------------------
[0m[0m VMA INFO: ---------------------------------------------------------------------------
[0m[0m VMA INFO: VMA_VERSION: 9.6.4-1 Release built on Aug 17 2022 14:23:56
[0m[0m VMA INFO: Cmd Line: sockperf pp -i 172.31.2.182 -t 10 --msg-size=1472
[0m[0m VMA INFO: ---------------------------------------------------------------------------
[0m[0m VMA INFO: Log Level INFO [VMA_TRACELEVEL]
[0m[0m VMA INFO: ---------------------------------------------------------------------------
[0m[2;35m VMA WARNING: *******************************************************************************************************
[0m[2;35m VMA WARNING: * Interface mlx5_ib0 will not be offloaded.
[0m[2;35m VMA WARNING: * Offloaded resources are restricted to root or user with CAP_NET_RAW privileges
[0m[2;35m VMA WARNING: * Read the CAP_NET_RAW and root access section in the VMA's User Manual for more information
[0m[2;35m VMA WARNING: *******************************************************************************************************
[0msockperf: [2;35m== version #3.10-0.git5ebd327da983 == [0m
sockperf[CLIENT] send on:sockperf: using recvfrom() to block on socket(s)

[ 0] IP = 172.31.2.182 PORT = 11111 # UDP
sockperf: Warmup stage (sending a few dummy messages)...
sockperf: Starting test...
sockperf: Test end (interrupted by timer)
sockperf: Test ended
sockperf: No messages were received from the server. Is the server down?
+ [22-11-28 04:38:30] result=0
+ [22-11-28 04:38:30] '[' 0 -ne 0 ']'
+ [22-11-28 04:38:30] grep -qi -e ' error ' -e 'no messages were received' /tmp/vma.txt
+ [22-11-28 04:38:30] return 1
+ [22-11-28 04:38:30] RQA_check_result -r 1 -t 'sockperf pingpong multicast pkey/vlan'

Expected results: On RHEL-8.7.0, the above same testcase result look like the following:

[0m VMA INFO: ---------------------------------------------------------------------------
[0m[0m VMA INFO: VMA_VERSION: 9.6.4-1 Release built on Aug 17 2022 14:23:56
[0m[0m VMA INFO: Cmd Line: date ++ [%y-%m-%d %H:%M:%S]
[0m[0m VMA INFO: ---------------------------------------------------------------------------
[0m[0m VMA INFO: Log Level INFO [VMA_TRACELEVEL]
[0m[0m VMA INFO: ---------------------------------------------------------------------------
[0m+ [22-11-28 06:19:40] LD_PRELOAD=libvma.so
[0m VMA INFO: ---------------------------------------------------------------------------
[0m[0m VMA INFO: VMA_VERSION: 9.6.4-1 Release built on Aug 17 2022 14:23:56
[0m[0m VMA INFO: Cmd Line: date ++ [%y-%m-%d %H:%M:%S]
[0m[0m VMA INFO: ---------------------------------------------------------------------------
[0m[0m VMA INFO: Log Level INFO [VMA_TRACELEVEL]
[0m[0m VMA INFO: ---------------------------------------------------------------------------
[0m+ [22-11-28 06:19:40] timeout --preserve-status --kill-after=5m 3m sockperf pp -i 172.31.2.182 -t 10 --msg-size=1472
[0m VMA INFO: ---------------------------------------------------------------------------
[0m[0m VMA INFO: VMA_VERSION: 9.6.4-1 Release built on Aug 17 2022 14:23:56
[0m[0m VMA INFO: Cmd Line: timeout --preserve-status --kill-after=5m 3m sockperf pp -i 172.31.2.182 -t 10 --msg-size=1472
[0m[0m VMA INFO: ---------------------------------------------------------------------------
[0m[0m VMA INFO: Log Level INFO [VMA_TRACELEVEL]
[0m[0m VMA INFO: ---------------------------------------------------------------------------
[0m[0m VMA INFO: ---------------------------------------------------------------------------
[0m[0m VMA INFO: VMA_VERSION: 9.6.4-1 Release built on Aug 17 2022 14:23:56
[0m[0m VMA INFO: Cmd Line: sockperf pp -i 172.31.2.182 -t 10 --msg-size=1472
[0m[0m VMA INFO: ---------------------------------------------------------------------------
[0m[0m VMA INFO: Log Level INFO [VMA_TRACELEVEL]
[0m[0m VMA INFO: ---------------------------------------------------------------------------
[0m[2;35m VMA WARNING: *******************************************************************************************************
[0m[2;35m VMA WARNING: * Interface mlx5_ib0 will not be offloaded.
[0m[2;35m VMA WARNING: * Offloaded resources are restricted to root or user with CAP_NET_RAW privileges
[0m[2;35m VMA WARNING: * Read the CAP_NET_RAW and root access section in the VMA's User Manual for more information
[0m[2;35m VMA WARNING: *******************************************************************************************************
[0m[2;35m VMA WARNING: *******************************************************************************************************
[0m[2;35m VMA WARNING: * Interface mlx5_ib0 will not be offloaded.
[0m[2;35m VMA WARNING: * Offloaded resources are restricted to root or user with CAP_NET_RAW privileges
[0m[2;35m VMA WARNING: * Read the CAP_NET_RAW and root access section in the VMA's User Manual for more information
[0m[2;35m VMA WARNING: *******************************************************************************************************
[0m[2;35m VMA WARNING: *******************************************************************************************************
[0m[2;35m VMA WARNING: * Interface mlx5_ib0 will not be offloaded.
[0m[2;35m VMA WARNING: * Offloaded resources are restricted to root or user with CAP_NET_RAW privileges
[0m[2;35m VMA WARNING: * Read the CAP_NET_RAW and root access section in the VMA's User Manual for more information
[0m[2;35m VMA WARNING: *******************************************************************************************************
[0m[2;35m VMA WARNING: *******************************************************************************************************
[0m[2;35m VMA WARNING: * Interface mlx5_ib0 will not be offloaded.
[0m[2;35m VMA WARNING: * Offloaded resources are restricted to root or user with CAP_NET_RAW privileges
[0m[2;35m VMA WARNING: * Read the CAP_NET_RAW and root access section in the VMA's User Manual for more information
[0m[2;35m VMA WARNING: *******************************************************************************************************
[0m[2;35m VMA WARNING: *******************************************************************************************************
[0m[2;35m VMA WARNING: * Interface mlx5_ib0 will not be offloaded.
[0m[2;35m VMA WARNING: * Offloaded resources are restricted to root or user with CAP_NET_RAW privileges
[0m[2;35m VMA WARNING: * Read the CAP_NET_RAW and root access section in the VMA's User Manual for more information
[0m[2;35m VMA WARNING: *******************************************************************************************************
[0m[2;35m VMA WARNING: *******************************************************************************************************
[0m[2;35m VMA WARNING: * Interface mlx5_ib0 will not be offloaded.
[0m[2;35m VMA WARNING: * Offloaded resources are restricted to root or user with CAP_NET_RAW privileges
[0m[2;35m VMA WARNING: * Read the CAP_NET_RAW and root access section in the VMA's User Manual for more information
[0m[2;35m VMA WARNING: *******************************************************************************************************
[0m[2;35m VMA WARNING: *******************************************************************************************************
[0m[2;35m VMA WARNING: * Interface mlx5_ib0 will not be offloaded.
[0m[2;35m VMA WARNING: * Offloaded resources are restricted to root or user with CAP_NET_RAW privileges
[0m[2;35m VMA WARNING: * Read the CAP_NET_RAW and root access section in the VMA's User Manual for more information
[0m[2;35m VMA WARNING: *******************************************************************************************************
[0msockperf: [2;35m== version #3.10-0.git5ebd327da983 == [0m
sockperf[CLIENT] send on:sockperf: using recvfrom() to block on socket(s)

[ 0] IP = 172.31.2.182 PORT = 11111 # UDP
sockperf: Warmup stage (sending a few dummy messages)...
sockperf: Starting test...
sockperf: Test end (interrupted by timer)
sockperf: Test ended
sockperf: [Total Run] RunTime=10.000 sec; Warm up time=400 msec; SentMessages=293057; ReceivedMessages=293056
sockperf: ========= Printing statistics for Server No: 0
sockperf: [Valid Duration] RunTime=9.550 sec; SentMessages=283646; ReceivedMessages=283646
sockperf: [2;35m====> avg-latency=16.798 (std-dev=3.707, mean-ad=0.643, median-ad=0.175, siqr=0.120, cv=0.221, std-error=0.007, 99.0% ci=[16.780, 16.816])[0m
sockperf: # dropped messages = 0; # duplicated messages = 0; # out-of-order messages = 0
sockperf: Summary: Latency is 16.798 usec
sockperf: [2;35mTotal 283646 observations[0m; each percentile contains 2836.46 observations
sockperf: ---> <MAX> observation = 189.200
sockperf: ---> percentile 99.999 = 185.468
sockperf: ---> percentile 99.990 = 178.978
sockperf: ---> percentile 99.900 = 34.925
sockperf: ---> percentile 99.000 = 20.270
sockperf: ---> percentile 90.000 = 16.866
sockperf: ---> percentile 75.000 = 16.600
sockperf: ---> percentile 50.000 = 16.491
sockperf: ---> percentile 25.000 = 16.358
sockperf: ---> <MIN> observation = 14.882
+ [22-11-28 06:19:54] result=0
+ [22-11-28 06:19:54] '[' 0 -ne 0 ']'
+ [22-11-28 06:19:54] grep -qi -e ' error ' -e 'no messages were received' /tmp/vma.txt
+ [22-11-28 06:19:54] return 0
+ [22-11-28 06:19:54] RQA_check_result -r 0 -t 'sockperf pingpong multicast pkey/vlan'
Additional info: