Description of problem:

"sockperf pingpong multicast" test case consistently fails when libvma is tested on MLX4 RoCE devices.

vma test results on rdma-virt-00/rdma-virt-01 & Beaker job J:8112258:
5.14.0-339.el9.x86_64, rdma-core-46.0-1.el9, mlx4, roce.45, ConnectX-3 Pro & mlx4_1
Result | Status | Test
-------------------------------------------------
FAIL | 1 | sockperf pingpong multicast <<<==============
PASS | 0 | sockperf throughput multicast
PASS | 0 | sockperf throughput unicast
PASS | 0 | sockperf pingpong unicast
PASS | 0 | sockperf (100 sockets) pingpong multicast
PASS | 0 | sockperf (100 sockets) pingpong unicast
PASS | 0 | sockperf pingpong multicast pkey/vlan
PASS | 0 | sockperf pingpong unicast pkey/vlan

++++++++++++++++++++++++++++++++++

This is a regression from CTC#1 build, RHEL-9.3.0-20230521.45, where the above test case passes.

vma test results on rdma-virt-00/rdma-virt-01 & Beaker job J:8112612:
5.14.0-316.el9.x86_64, rdma-core-44.0-2.el9, mlx4, roce.45, ConnectX-3 Pro & mlx4_1
Result | Status | Test
-------------------------------------------------
PASS | 0 | sockperf pingpong multicast <<<==========
PASS | 0 | sockperf throughput multicast
PASS | 0 | sockperf throughput unicast
PASS | 0 | sockperf pingpong unicast
PASS | 0 | sockperf (100 sockets) pingpong multicast
PASS | 0 | sockperf (100 sockets) pingpong unicast
PASS | 0 | sockperf pingpong multicast pkey/vlan
PASS | 0 | sockperf pingpong unicast pkey/vlan
Checking for failures and known issues:
no test failures

+++++++++++++++++++++++++++

The same results observed in another MLX4 IB0 host pair : rdma-perf-00/01

vma test results on rdma-perf-00/rdma-perf-01 & Beaker job J:8112242:
5.14.0-339.el9.x86_64, rdma-core-46.0-1.el9, mlx4, roce.45, ConnectX-3 & mlx4_0
Result | Status | Test
-------------------------------------------------
FAIL | 1 | sockperf pingpong multicast
PASS | 0 | sockperf throughput multicast
PASS | 0 | sockperf throughput unicast
PASS | 0 | sockperf pingpong unicast
PASS | 0 | sockperf (100 sockets) pingpong multicast
PASS | 0 | sockperf (100 sockets) pingpong unicast
PASS | 0 | sockperf pingpong multicast pkey/vlan
PASS | 0 | sockperf pingpong unicast pkey/vlan
Checking for failures and known issues:
sockperf pingpong multicast is NOT a known issue on any environment - consider filing a BZ

Version-Release number of selected component (if applicable):

Clients: rdma-virt-01
Servers: rdma-virt-00

DISTRO=RHEL-9.3.0-20230718.0

+ [23-07-26 06:53:45] cat /etc/redhat-release
Red Hat Enterprise Linux release 9.3 Beta (Plow)

+ [23-07-26 06:53:45] uname -a
Linux rdma-virt-01.rdma.lab.eng.rdu2.redhat.com 5.14.0-339.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Jul 13 07:33:32 EDT 2023 x86_64 x86_64 x86_64 GNU/Linux

+ [23-07-26 06:53:45] cat /proc/cmdline
BOOT_IMAGE=(hd0,msdos1)/vmlinuz-5.14.0-339.el9.x86_64 root=UUID=63c088f1-90c6-434d-9fad-7d59f30f99c1 ro intel_idle.max_cstate=0 processor.max_cstate=0 intel_iommu=on iommu=on console=tty0 rd_NO_PLYMOUTH crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=UUID=6ec20b3d-43e2-4166-91cd-ae659aee9384 console=ttyS1,115200n81

+ [23-07-26 06:53:45] rpm -q rdma-core linux-firmware
rdma-core-46.0-1.el9.x86_64
linux-firmware-20230625-137.el9.noarch

+ [23-07-26 06:53:45] tail /sys/class/infiniband/mlx4_0/fw_ver /sys/class/infiniband/mlx4_1/fw_ver
==> /sys/class/infiniband/mlx4_0/fw_ver <==
2.42.5000

==> /sys/class/infiniband/mlx4_1/fw_ver <==
2.42.5000

+ [23-07-26 06:53:45] lspci
+ [23-07-26 06:53:45] grep -i -e ethernet -e infiniband -e omni -e ConnectX
02:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 Gigabit Ethernet PCIe
02:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 Gigabit Ethernet PCIe
03:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 Gigabit Ethernet PCIe
03:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 Gigabit Ethernet PCIe
04:00.0 Network controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro]
04:00.1 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
04:00.2 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
04:00.3 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
04:00.4 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
04:00.5 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
04:00.6 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
04:00.7 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
06:00.0 Network controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro]

+ [23-07-26 06:53:46] rpm -q libvma
libvma-9.8.20-1.el9.x86_64

How reproducible:

100%

Steps to Reproduce:
1. Get both RDMA server & client hosts ready for libvma test with current sockperf, with RHEL-9.3 build as specified above on MLX4 IB0 devices
2. On server host, issue the following commands

LD_PRELOAD=libvma.so
timeout --preserve-status --kill-after=5m 3m sockperf server -i 172.31.40.200

3. On the client hosts, issue the following command

LD_PRELOAD=libvma.so
imeout --preserve-status --kill-after=5m 3m sockperf pp -i 172.31.40.200 -t 10 --msg-size=1472

Actual results:

Server host:

[0m VMA INFO: ---------------------------------------------------------------------------
[0m[0m VMA INFO: VMA_VERSION: 9.8.20-1 Release built on Jun 1 2023 00:00:00
[0m[0m VMA INFO: Cmd Line: date ++ [%y-%m-%d %H:%M:%S]
[0m[0m VMA INFO: ---------------------------------------------------------------------------
[0m[0m VMA INFO: Log Level INFO [VMA_TRACELEVEL]
[0m[0m VMA INFO: ---------------------------------------------------------------------------
[0m+ [23-07-26 07:01:27] LD_PRELOAD=libvma.so
[0m VMA INFO: ---------------------------------------------------------------------------
[0m[0m VMA INFO: VMA_VERSION: 9.8.20-1 Release built on Jun 1 2023 00:00:00
[0m[0m VMA INFO: Cmd Line: date ++ [%y-%m-%d %H:%M:%S]
[0m[0m VMA INFO: ---------------------------------------------------------------------------
[0m[0m VMA INFO: Log Level INFO [VMA_TRACELEVEL]
[0m[0m VMA INFO: ---------------------------------------------------------------------------
[0m+ [23-07-26 07:01:27] timeout --preserve-status --kill-after=5m 3m sockperf server -i 172.31.40.200
[0m VMA INFO: ---------------------------------------------------------------------------
[0m[0m VMA INFO: VMA_VERSION: 9.8.20-1 Release built on Jun 1 2023 00:00:00
[0m[0m VMA INFO: Cmd Line: timeout --preserve-status --kill-after=5m 3m sockperf server -i 172.31.40.200
[0m[0m VMA INFO: ---------------------------------------------------------------------------
[0m[0m VMA INFO: Log Level INFO [VMA_TRACELEVEL]
[0m[0m VMA INFO: ---------------------------------------------------------------------------
[0m[0m VMA INFO: ---------------------------------------------------------------------------
[0m[0m VMA INFO: VMA_VERSION: 9.8.20-1 Release built on Jun 1 2023 00:00:00
[0m[0m VMA INFO: Cmd Line: sockperf server -i 172.31.40.200
[0m[0m VMA INFO: ---------------------------------------------------------------------------
[0m[0m VMA INFO: Log Level INFO [VMA_TRACELEVEL]
[0m[0m VMA INFO: ---------------------------------------------------------------------------
[0m[2;35m VMA WARNING: *******************************************************************************************************
[0m[2;35m VMA WARNING: * IPoIB mode of interface 'mlx4_ib0' is "connected" !
[0m[2;35m VMA WARNING: * Please change it to datagram: "echo datagram > /sys/class/net/mlx4_ib0/mode" before loading your application with VMA library
[0m[2;35m VMA WARNING: * VMA doesn't support IPoIB in connected mode.
[0m[2;35m VMA WARNING: * Please refer to VMA Release Notes for more information
[0m[2;35m VMA WARNING: *******************************************************************************************************
[0m[2;35m VMA WARNING: *******************************************************************************************************
[0m[2;35m VMA WARNING: * IPoIB mode of interface 'mlx4_ib0.8004' is "connected" !
[0m[2;35m VMA WARNING: * Please change it to datagram: "echo datagram > /sys/class/net/mlx4_ib0.8004/mode" before loading your application with VMA library
[0m[2;35m VMA WARNING: * VMA doesn't support IPoIB in connected mode.
[0m[2;35m VMA WARNING: * Please refer to VMA Release Notes for more information
[0m[2;35m VMA WARNING: *******************************************************************************************************
[0m[2;35m VMA WARNING: *******************************************************************************************************
[0m[2;35m VMA WARNING: * IPoIB mode of interface 'mlx4_ib0.8002' is "connected" !
[0m[2;35m VMA WARNING: * Please change it to datagram: "echo datagram > /sys/class/net/mlx4_ib0.8002/mode" before loading your application with VMA library
[0m[2;35m VMA WARNING: * VMA doesn't support IPoIB in connected mode.
[0m[2;35m VMA WARNING: * Please refer to VMA Release Notes for more information
[0m[2;35m VMA WARNING: *******************************************************************************************************
[0m[2;35m VMA WARNING: *******************************************************************************************************
[0m[2;35m VMA WARNING: * IPoIB mode of interface 'mlx4_ib0.8010' is "connected" !
[0m[2;35m VMA WARNING: * Please change it to datagram: "echo datagram > /sys/class/net/mlx4_ib0.8010/mode" before loading your application with VMA library
[0m[2;35m VMA WARNING: * VMA doesn't support IPoIB in connected mode.
[0m[2;35m VMA WARNING: * Please refer to VMA Release Notes for more information
[0m[2;35m VMA WARNING: *******************************************************************************************************
[0m[2;35m VMA WARNING: *******************************************************************************************************
[0m[2;35m VMA WARNING: * IPoIB mode of interface 'mlx4_ib0.8012' is "connected" !
[0m[2;35m VMA WARNING: * Please change it to datagram: "echo datagram > /sys/class/net/mlx4_ib0.8012/mode" before loading your application with VMA library
[0m[2;35m VMA WARNING: * VMA doesn't support IPoIB in connected mode.
[0m[2;35m VMA WARNING: * Please refer to VMA Release Notes for more information
[0m[2;35m VMA WARNING: *******************************************************************************************************
[0msockperf: [2;35m== version #3.10-11.gitca16d475d6fa == [0m
sockperf: [SERVER] listen on:
[ 0] IP = 172.31.40.200 PORT = 11111 # UDP
sockperf: Warmup stage (sending a few dummy messages)...
sockperf: [tid 61498] using recvfrom() to block on socket(s)
Terminated

client hosts:

+ [23-07-26 07:00:16] sleep 10
+ [23-07-26 07:00:26] vma_test sockperf pp -i 172.31.40.200 -t 10 --msg-size=1472
+ [23-07-26 07:00:26] rm -rf /tmp/vma.txt
+ [23-07-26 07:00:26] tee /tmp/vma.txt
[0m VMA INFO: ---------------------------------------------------------------------------
[0m[0m VMA INFO: VMA_VERSION: 9.8.20-1 Release built on Jun 1 2023 00:00:00
[0m[0m VMA INFO: Cmd Line: date ++ [%y-%m-%d %H:%M:%S]
[0m[0m VMA INFO: ---------------------------------------------------------------------------
[0m[0m VMA INFO: Log Level INFO [VMA_TRACELEVEL]
[0m[0m VMA INFO: ---------------------------------------------------------------------------
[0m+ [23-07-26 07:00:26] LD_PRELOAD=libvma.so
[0m VMA INFO: ---------------------------------------------------------------------------
[0m[0m VMA INFO: VMA_VERSION: 9.8.20-1 Release built on Jun 1 2023 00:00:00
[0m[0m VMA INFO: Cmd Line: date ++ [%y-%m-%d %H:%M:%S]
[0m[0m VMA INFO: ---------------------------------------------------------------------------
[0m[0m VMA INFO: Log Level INFO [VMA_TRACELEVEL]
[0m[0m VMA INFO: ---------------------------------------------------------------------------
[0m+ [23-07-26 07:00:26] timeout --preserve-status --kill-after=5m 3m sockperf pp -i 172.31.40.200 -t 10 --msg-size=1472
[0m VMA INFO: ---------------------------------------------------------------------------
[0m[0m VMA INFO: VMA_VERSION: 9.8.20-1 Release built on Jun 1 2023 00:00:00
[0m[0m VMA INFO: Cmd Line: timeout --preserve-status --kill-after=5m 3m sockperf pp -i 172.31.40.200 -t 10 --msg-size=1472
[0m[0m VMA INFO: ---------------------------------------------------------------------------
[0m[0m VMA INFO: Log Level INFO [VMA_TRACELEVEL]
[0m[0m VMA INFO: ---------------------------------------------------------------------------
[0m[0m VMA INFO: ---------------------------------------------------------------------------
[0m[0m VMA INFO: VMA_VERSION: 9.8.20-1 Release built on Jun 1 2023 00:00:00
[0m[0m VMA INFO: Cmd Line: sockperf pp -i 172.31.40.200 -t 10 --msg-size=1472
[0m[0m VMA INFO: ---------------------------------------------------------------------------
[0m[0m VMA INFO: Log Level INFO [VMA_TRACELEVEL]
[0m[0m VMA INFO: ---------------------------------------------------------------------------
[0m[2;35m VMA WARNING: *******************************************************************************************************
[0m[2;35m VMA WARNING: * IPoIB mode of interface 'mlx4_ib0' is "connected" !
[0m[2;35m VMA WARNING: * Please change it to datagram: "echo datagram > /sys/class/net/mlx4_ib0/mode" before loading your application with VMA library
[0m[2;35m VMA WARNING: * VMA doesn't support IPoIB in connected mode.
[0m[2;35m VMA WARNING: * Please refer to VMA Release Notes for more information
[0m[2;35m VMA WARNING: *******************************************************************************************************
[0m[2;35m VMA WARNING: *******************************************************************************************************
[0m[2;35m VMA WARNING: * IPoIB mode of interface 'mlx4_ib0.8004' is "connected" !
[0m[2;35m VMA WARNING: * Please change it to datagram: "echo datagram > /sys/class/net/mlx4_ib0.8004/mode" before loading your application with VMA library
[0m[2;35m VMA WARNING: * VMA doesn't support IPoIB in connected mode.
[0m[2;35m VMA WARNING: * Please refer to VMA Release Notes for more information
[0m[2;35m VMA WARNING: *******************************************************************************************************
[0m[2;35m VMA WARNING: *******************************************************************************************************
[0m[2;35m VMA WARNING: * IPoIB mode of interface 'mlx4_ib0.8002' is "connected" !
[0m[2;35m VMA WARNING: * Please change it to datagram: "echo datagram > /sys/class/net/mlx4_ib0.8002/mode" before loading your application with VMA library
[0m[2;35m VMA WARNING: * VMA doesn't support IPoIB in connected mode.
[0m[2;35m VMA WARNING: * Please refer to VMA Release Notes for more information
[0m[2;35m VMA WARNING: *******************************************************************************************************
[0m[2;35m VMA WARNING: *******************************************************************************************************
[0m[2;35m VMA WARNING: * IPoIB mode of interface 'mlx4_ib0.8010' is "connected" !
[0m[2;35m VMA WARNING: * Please change it to datagram: "echo datagram > /sys/class/net/mlx4_ib0.8010/mode" before loading your application with VMA library
[0m[2;35m VMA WARNING: * VMA doesn't support IPoIB in connected mode.
[0m[2;35m VMA WARNING: * Please refer to VMA Release Notes for more information
[0m[2;35m VMA WARNING: *******************************************************************************************************
[0m[2;35m VMA WARNING: *******************************************************************************************************
[0m[2;35m VMA WARNING: * IPoIB mode of interface 'mlx4_ib0.8012' is "connected" !
[0m[2;35m VMA WARNING: * Please change it to datagram: "echo datagram > /sys/class/net/mlx4_ib0.8012/mode" before loading your application with VMA library
[0m[2;35m VMA WARNING: * VMA doesn't support IPoIB in connected mode.
[0m[2;35m VMA WARNING: * Please refer to VMA Release Notes for more information
[0m[2;35m VMA WARNING: *******************************************************************************************************
[0msockperf: [2;35m== version #3.10-11.gitca16d475d6fa == [0m
sockperf[CLIENT] send on:sockperf: using recvfrom() to block on socket(s)

[ 0] IP = 172.31.40.200 PORT = 11111 # UDP
sockperf: Warmup stage (sending a few dummy messages)...
sockperf: Starting test...
sockperf: Test end (interrupted by timer)
sockperf: Test ended
sockperf: No messages were received from the server. Is the server down?
+ [23-07-26 07:00:40] result=0
+ [23-07-26 07:00:40] '[' 0 -ne 0 ']'
+ [23-07-26 07:00:40] grep -qi -e ' error ' -e 'no messages were received' /tmp/vma.txt
+ [23-07-26 07:00:40] return 1
+ [23-07-26 07:00:40] RQA_check_result -r 1 -t 'sockperf pingpong multicast'

Expected results:

[0m VMA INFO: ---------------------------------------------------------------------------
[0m[0m VMA INFO: VMA_VERSION: 9.6.4-1 Release built on Aug 3 2022 00:00:00
[0m[0m VMA INFO: Cmd Line: date ++ [%y-%m-%d %H:%M:%S]
[0m[0m VMA INFO: ---------------------------------------------------------------------------
[0m[0m VMA INFO: Log Level INFO [VMA_TRACELEVEL]
[0m[0m VMA INFO: ---------------------------------------------------------------------------
[0m+ [23-07-26 08:23:41] LD_PRELOAD=libvma.so
[0m VMA INFO: ---------------------------------------------------------------------------
[0m[0m VMA INFO: VMA_VERSION: 9.6.4-1 Release built on Aug 3 2022 00:00:00
[0m[0m VMA INFO: Cmd Line: date ++ [%y-%m-%d %H:%M:%S]
[0m[0m VMA INFO: ---------------------------------------------------------------------------
[0m[0m VMA INFO: Log Level INFO [VMA_TRACELEVEL]
[0m[0m VMA INFO: ---------------------------------------------------------------------------
[0m+ [23-07-26 08:23:41] timeout --preserve-status --kill-after=5m 3m sockperf server -i 172.31.40.200
[0m VMA INFO: ---------------------------------------------------------------------------
[0m[0m VMA INFO: VMA_VERSION: 9.6.4-1 Release built on Aug 3 2022 00:00:00
[0m[0m VMA INFO: Cmd Line: timeout --preserve-status --kill-after=5m 3m sockperf server -i 172.31.40.200
[0m[0m VMA INFO: ---------------------------------------------------------------------------
[0m[0m VMA INFO: Log Level INFO [VMA_TRACELEVEL]
[0m[0m VMA INFO: ---------------------------------------------------------------------------
[0m[0m VMA INFO: ---------------------------------------------------------------------------
[0m[0m VMA INFO: VMA_VERSION: 9.6.4-1 Release built on Aug 3 2022 00:00:00
[0m[0m VMA INFO: Cmd Line: sockperf server -i 172.31.40.200
[0m[0m VMA INFO: ---------------------------------------------------------------------------
[0m[0m VMA INFO: Log Level INFO [VMA_TRACELEVEL]
[0m[0m VMA INFO: ---------------------------------------------------------------------------
[0m[2;35m VMA WARNING: *******************************************************************************************************
[0m[2;35m VMA WARNING: * IPoIB mode of interface 'mlx4_ib0' is "connected" !
[0m[2;35m VMA WARNING: * Please change it to datagram: "echo datagram > /sys/class/net/mlx4_ib0/mode" before loading your application with VMA library
[0m[2;35m VMA WARNING: * VMA doesn't support IPoIB in connected mode.
[0m[2;35m VMA WARNING: * Please refer to VMA Release Notes for more information
[0m[2;35m VMA WARNING: *******************************************************************************************************
[0msockperf: [2;35m== version #3.10-11.gitca16d475d6fa == [0m
sockperf: [SERVER] listen on:
[ 0] IP = 172.31.40.200 PORT = 11111 # UDP
sockperf: Warmup stage (sending a few dummy messages)...
sockperf: [tid 60923] using recvfrom() to block on socket(s)
Terminated

Client host:

+ [23-07-26 08:23:57] sleep 10
+ [23-07-26 08:24:07] vma_test sockperf pp -i 172.31.40.200 -t 10 --msg-size=1472
+ [23-07-26 08:24:07] rm -rf /tmp/vma.txt
+ [23-07-26 08:24:07] tee /tmp/vma.txt
[0m VMA INFO: ---------------------------------------------------------------------------
[0m[0m VMA INFO: VMA_VERSION: 9.6.4-1 Release built on Aug 3 2022 00:00:00
[0m[0m VMA INFO: Cmd Line: date ++ [%y-%m-%d %H:%M:%S]
[0m[0m VMA INFO: ---------------------------------------------------------------------------
[0m[0m VMA INFO: Log Level INFO [VMA_TRACELEVEL]
[0m[0m VMA INFO: ---------------------------------------------------------------------------
[0m+ [23-07-26 08:24:07] LD_PRELOAD=libvma.so
[0m VMA INFO: ---------------------------------------------------------------------------
[0m[0m VMA INFO: VMA_VERSION: 9.6.4-1 Release built on Aug 3 2022 00:00:00
[0m[0m VMA INFO: Cmd Line: date ++ [%y-%m-%d %H:%M:%S]
[0m[0m VMA INFO: ---------------------------------------------------------------------------
[0m[0m VMA INFO: Log Level INFO [VMA_TRACELEVEL]
[0m[0m VMA INFO: ---------------------------------------------------------------------------
[0m+ [23-07-26 08:24:07] timeout --preserve-status --kill-after=5m 3m sockperf pp -i 172.31.40.200 -t 10 --msg-size=1472
[0m VMA INFO: ---------------------------------------------------------------------------
[0m[0m VMA INFO: VMA_VERSION: 9.6.4-1 Release built on Aug 3 2022 00:00:00
[0m[0m VMA INFO: Cmd Line: timeout --preserve-status --kill-after=5m 3m sockperf pp -i 172.31.40.200 -t 10 --msg-size=1472
[0m[0m VMA INFO: ---------------------------------------------------------------------------
[0m[0m VMA INFO: Log Level INFO [VMA_TRACELEVEL]
[0m[0m VMA INFO: ---------------------------------------------------------------------------
[0m[0m VMA INFO: ---------------------------------------------------------------------------
[0m[0m VMA INFO: VMA_VERSION: 9.6.4-1 Release built on Aug 3 2022 00:00:00
[0m[0m VMA INFO: Cmd Line: sockperf pp -i 172.31.40.200 -t 10 --msg-size=1472
[0m[0m VMA INFO: ---------------------------------------------------------------------------
[0m[0m VMA INFO: Log Level INFO [VMA_TRACELEVEL]
[0m[0m VMA INFO: ---------------------------------------------------------------------------
[0m[2;35m VMA WARNING: *******************************************************************************************************
[0m[2;35m VMA WARNING: * IPoIB mode of interface 'mlx4_ib0' is "connected" !
[0m[2;35m VMA WARNING: * Please change it to datagram: "echo datagram > /sys/class/net/mlx4_ib0/mode" before loading your application with VMA library
[0m[2;35m VMA WARNING: * VMA doesn't support IPoIB in connected mode.
[0m[2;35m VMA WARNING: * Please refer to VMA Release Notes for more information
[0m[2;35m VMA WARNING: *******************************************************************************************************
[0msockperf: [2;35m== version #3.10-11.gitca16d475d6fa == [0m
sockperf[CLIENT] send on:sockperf: using recvfrom() to block on socket(s)

[ 0] IP = 172.31.40.200 PORT = 11111 # UDP
sockperf: Warmup stage (sending a few dummy messages)...
sockperf: Starting test...
sockperf: Test end (interrupted by timer)
sockperf: Test ended
sockperf: [Total Run] RunTime=10.000 sec; Warm up time=400 msec; SentMessages=1574119; ReceivedMessages=1574118
sockperf: ========= Printing statistics for Server No: 0
sockperf: [Valid Duration] RunTime=9.531 sec; SentMessages=1506246; ReceivedMessages=1506246
sockperf: [2;35m====> avg-latency=3.144 (std-dev=0.332, mean-ad=0.090, median-ad=0.033, siqr=0.025, cv=0.106, std-error=0.000, 99.0% ci=[3.143, 3.145])[0m
sockperf: # dropped messages = 0; # duplicated messages = 0; # out-of-order messages = 0
sockperf: Summary: Latency is 3.144 usec
sockperf: [2;35mTotal 1506246 observations[0m; each percentile contains 15062.46 observations
sockperf: ---> <MAX> observation = 94.753
sockperf: ---> percentile 99.999 = 57.166
sockperf: ---> percentile 99.990 = 5.478
sockperf: ---> percentile 99.900 = 4.607
sockperf: ---> percentile 99.000 = 4.045
sockperf: ---> percentile 90.000 = 3.226
sockperf: ---> percentile 75.000 = 3.126
sockperf: ---> percentile 50.000 = 3.094
sockperf: ---> percentile 25.000 = 3.075
sockperf: ---> <MIN> observation = 2.986
+ [23-07-26 08:24:21] result=0
+ [23-07-26 08:24:21] '[' 0 -ne 0 ']'
+ [23-07-26 08:24:21] grep -qi -e ' error ' -e 'no messages were received' /tmp/vma.txt
+ [23-07-26 08:24:21] return 0
+ [23-07-26 08:24:21] RQA_check_result -r 0 -t 'sockperf pingpong multicast'
+ [23-07-26 08:24:21] local test_pass=0
+ [23-07-26 08:24:21] local test_skip=777
+ [23-07-26 08:24:21] test 4 -gt 0
+ [23-07-26 08:24:21] case $1 in
+ [23-07-26 08:24:21] local rc=0
+ [23-07-26 08:24:21] shift
+ [23-07-26 08:24:21] shift
+ [23-07-26 08:24:21] test 2 -gt 0
+ [23-07-26 08:24:21] case $1 in
+ [23-07-26 08:24:21] local 'msg=sockperf pingpong multicast'
+ [23-07-26 08:24:21] shift
+ [23-07-26 08:24:21] shift
+ [23-07-26 08:24:21] test 0 -gt 0

Additional info: