Description of problem:

SIGABRT core files were observed during the fabtests on RHEL-9.2.0 Beta compose.

Version-Release number of selected component (if applicable):

Clients: rdma-dev-21
Servers: rdma-dev-20

DISTRO=RHEL-9.2.0-20230309.10
+ [23-03-11 08:59:05] cat /etc/redhat-release
Red Hat Enterprise Linux release 9.2 Beta (Plow)

+ [23-03-11 08:59:05] uname -a
Linux rdma-dev-21.rdma.lab.eng.rdu2.redhat.com 5.14.0-284.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Mon Feb 27 20:08:54 EST 2023 x86_64 x86_64 x86_64 GNU/Linux

+ [23-03-11 08:59:05] cat /proc/cmdline
BOOT_IMAGE=(hd0,msdos1)/vmlinuz-5.14.0-284.el9.x86_64 root=UUID=941c727f-9f57-43d6-8de9-0af7db8bf888 ro intel_idle.max_cstate=0 processor.max_cstate=0 intel_iommu=on iommu=on console=tty0 rd_NO_PLYMOUTH crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=UUID=11327d46-3e02-467d-b44e-086447bf8566 console=ttyS1,115200n81

+ [23-03-11 08:59:05] rpm -q rdma-core linux-firmware
rdma-core-44.0-2.el9.x86_64
linux-firmware-20230210-132.el9.noarch

+ [23-03-11 08:59:05] tail /sys/class/infiniband/mlx5_0/fw_ver /sys/class/infiniband/mlx5_1/fw_ver /sys/class/infiniband/mlx5_2/fw_ver
==> /sys/class/infiniband/mlx5_0/fw_ver <==
12.28.2006

==> /sys/class/infiniband/mlx5_1/fw_ver <==
12.28.2006

==> /sys/class/infiniband/mlx5_2/fw_ver <==
12.28.2006
+ [23-03-11 08:59:05] lspci
+ [23-03-11 08:59:05] grep -i -e ethernet -e infiniband -e omni -e ConnectX
01:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 Gigabit Ethernet PCIe
01:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 Gigabit Ethernet PCIe
02:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 Gigabit Ethernet PCIe
02:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 Gigabit Ethernet PCIe
04:00.0 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4]
82:00.0 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4]
82:00.1 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4]

Installed:
fabtests-1.17.0-2.el9.x86_64
python3-attrs-20.3.0-7.el9.noarch
python3-iniconfig-1.1.1-7.el9.noarch
python3-packaging-20.9-5.el9.noarch
python3-pluggy-0.13.1-7.el9.noarch
python3-py-1.10.0-6.el9.noarch
python3-pyparsing-2.4.7-9.el9.noarch
python3-pytest-6.2.2-6.el9.noarch
python3-toml-0.10.2-6.el9.noarch
ruby-3.0.4-160.el9_0.x86_64
ruby-default-gems-3.0.4-160.el9_0.noarch
ruby-libs-3.0.4-160.el9_0.x86_64
rubygem-bigdecimal-3.0.0-160.el9_0.x86_64
rubygem-bundler-2.2.33-160.el9_0.noarch
rubygem-io-console-0.5.7-160.el9_0.x86_64
rubygem-json-2.5.1-160.el9_0.x86_64
rubygem-psych-3.3.2-160.el9_0.x86_64
rubygem-rdoc-6.3.3-160.el9_0.noarch
rubygems-3.2.33-160.el9_0.noarch

How reproducible:

Steps to Reproduce:
1. run the fatests with the above packages on the above MLX5 IB0
2.
3.

Actual results:

In both RDMA server and client hosts, the following core files were observed.

TIME PID UID GID SIG COREFILE EXE SIZE
Sat 2023-03-11 09:53:13 EST 64015 0 0 SIGABRT present /usr/bin/fi_av_xfer 272.6K
Sat 2023-03-11 09:53:20 EST 64064 0 0 SIGABRT present /usr/bin/fi_av_xfer 274.0K
Sat 2023-03-11 09:53:31 EST 64182 0 0 SIGABRT present /usr/bin/fi_cq_data 274.6K
Sat 2023-03-11 09:53:38 EST 64228 0 0 SIGABRT present /usr/bin/fi_cq_data 275.2K
Sat 2023-03-11 09:53:45 EST 64274 0 0 SIGABRT present /usr/bin/fi_dgram 271.5K
Sat 2023-03-11 09:53:52 EST 64318 0 0 SIGABRT present /usr/bin/fi_dgram_waitset 270.0K
Sat 2023-03-11 09:54:04 EST 64470 0 0 SIGABRT present /usr/bin/fi_poll 271.8K
Sat 2023-03-11 09:54:10 EST 64518 0 0 SIGABRT present /usr/bin/fi_poll 272.1K
Sat 2023-03-11 09:54:17 EST 64564 0 0 SIGABRT present /usr/bin/fi_rdm 270.8K
Sat 2023-03-11 09:54:24 EST 64608 0 0 SIGABRT present /usr/bin/fi_rdm 271.6K
Sat 2023-03-11 09:54:31 EST 64652 0 0 SIGABRT present /usr/bin/fi_rdm_rma_event 269.7K
Sat 2023-03-11 09:54:39 EST 64696 0 0 SIGABRT present /usr/bin/fi_rdm_rma_trigger 274.2K
Sat 2023-03-11 09:54:50 EST 64813 0 0 SIGABRT present /usr/bin/fi_shared_ctx 676.3K
Sat 2023-03-11 09:55:05 EST 65038 0 0 SIGABRT present /usr/bin/fi_shared_ctx 675.2K
Sat 2023-03-11 09:55:12 EST 65084 0 0 SIGABRT present /usr/bin/fi_rdm_tagged_peek 270.5K
Sat 2023-03-11 09:55:21 EST 65168 0 0 SIGABRT present /usr/bin/fi_rdm_shared_av 270.6K
Sat 2023-03-11 09:55:46 EST 65293 0 0 SIGABRT present /usr/bin/fi_multi_mr 304.6K
Sat 2023-03-11 09:55:58 EST 65374 0 0 SIGABRT present /usr/bin/fi_multi_ep 715.9K
Sat 2023-03-11 09:56:07 EST 65419 0 0 SIGABRT present /usr/bin/fi_recv_cancel 303.8K
Sat 2023-03-11 09:56:17 EST 65500 0 0 SIGABRT present /usr/bin/fi_unexpected_msg 275.5K
Sat 2023-03-11 09:56:26 EST 65544 0 0 SIGABRT present /usr/bin/fi_msg_inject 303.7K
Sat 2023-03-11 09:56:34 EST 65591 0 0 SIGABRT present /usr/bin/fi_msg_inject 304.6K
Sat 2023-03-11 09:56:42 EST 65637 0 0 SIGABRT present /usr/bin/fi_msg_inject 304.4K
Sat 2023-03-11 09:56:51 EST 65681 0 0 SIGABRT present /usr/bin/fi_msg_inject 304.3K
Sat 2023-03-11 09:56:59 EST 65726 0 0 SIGABRT present /usr/bin/fi_bw 307.2K
Sat 2023-03-11 09:57:07 EST 65770 0 0 SIGABRT present /usr/bin/fi_bw 306.1K
Sat 2023-03-11 09:57:16 EST 65851 0 0 SIGABRT present /usr/bin/fi_rdm_multi_client 273.6K
Sat 2023-03-11 09:57:23 EST 65896 0 0 SIGABRT present /usr/bin/fi_rdm_multi_client 274.4K
Sat 2023-03-11 09:57:46 EST 66267 0 0 SIGABRT present /usr/bin/fi_rma_bw 302.1K
Sat 2023-03-11 09:57:55 EST 66312 0 0 SIGABRT present /usr/bin/fi_rma_bw 301.6K
Sat 2023-03-11 09:58:03 EST 66356 0 0 SIGABRT present /usr/bin/fi_rma_bw 301.8K
Sat 2023-03-11 09:58:11 EST 66401 0 0 SIGABRT present /usr/bin/fi_rma_bw 302.0K
Sat 2023-03-11 09:58:20 EST 66446 0 0 SIGABRT present /usr/bin/fi_rma_bw 304.7K
Sat 2023-03-11 09:58:28 EST 66490 0 0 SIGABRT present /usr/bin/fi_rma_bw 303.7K
Sat 2023-03-11 09:58:37 EST 66534 0 0 SIGABRT present /usr/bin/fi_rdm_atomic 305.7K
Sat 2023-03-11 09:58:45 EST 66579 0 0 SIGABRT present /usr/bin/fi_rdm_atomic 304.9K
Sat 2023-03-11 09:58:55 EST 66662 0 0 SIGABRT present /usr/bin/fi_multi_recv 275.3K
Sat 2023-03-11 09:59:06 EST 66743 0 0 SIGABRT present /usr/bin/fi_rdm_pingpong 304.3K
Sat 2023-03-11 09:59:14 EST 66790 0 0 SIGABRT present /usr/bin/fi_rdm_pingpong 305.9K
Sat 2023-03-11 09:59:23 EST 66834 0 0 SIGABRT present /usr/bin/fi_rdm_pingpong 304.2K
Sat 2023-03-11 09:59:31 EST 66881 0 0 SIGABRT present /usr/bin/fi_rdm_pingpong 304.2K
Sat 2023-03-11 09:59:47 EST 67074 0 0 SIGABRT present /usr/bin/fi_rdm_tagged_pingpong 305.5K
Sat 2023-03-11 09:59:55 EST 67119 0 0 SIGABRT present /usr/bin/fi_rdm_tagged_pingpong 305.4K
Sat 2023-03-11 10:00:03 EST 67165 0 0 SIGABRT present /usr/bin/fi_rdm_tagged_pingpong 305.6K
Sat 2023-03-11 10:00:12 EST 67211 0 0 SIGABRT present /usr/bin/fi_rdm_tagged_pingpong 306.1K
Sat 2023-03-11 10:00:20 EST 67255 0 0 SIGABRT present /usr/bin/fi_rdm_tagged_bw 303.1K
Sat 2023-03-11 10:00:29 EST 67300 0 0 SIGABRT present /usr/bin/fi_rdm_tagged_bw 306.0K
Sat 2023-03-11 10:00:37 EST 67344 0 0 SIGABRT present /usr/bin/fi_rdm_tagged_bw 303.6K
Sat 2023-03-11 10:00:46 EST 67389 0 0 SIGABRT present /usr/bin/fi_rdm_tagged_bw 303.6K
Sat 2023-03-11 10:00:55 EST 67434 0 0 SIGABRT present /usr/bin/fi_dgram_pingpong 306.3K
Sat 2023-03-11 10:01:05 EST 67535 0 0 SIGABRT present /usr/bin/fi_multinode 273.8K
Sat 2023-03-11 10:01:05 EST 67529 0 0 SIGABRT present /usr/bin/fi_multinode 273.9K
Sat 2023-03-11 10:01:13 EST 67647 0 0 SIGABRT present /usr/bin/fi_multinode 271.9K
Sat 2023-03-11 10:01:13 EST 67622 0 0 SIGABRT present /usr/bin/fi_multinode 276.0K
Sat 2023-03-11 10:17:41 EST 71058 0 0 SIGABRT present /usr/bin/fi_unexpected_msg 274.9K
Sat 2023-03-11 15:03:42 EST 164478 0 0 SIGABRT present /usr/bin/fi_av_xfer 274.1K
Sat 2023-03-11 15:03:50 EST 164537 0 0 SIGABRT present /usr/bin/fi_av_xfer 273.9K
Sat 2023-03-11 15:04:00 EST 164692 0 0 SIGABRT present /usr/bin/fi_cq_data 273.3K
Sat 2023-03-11 15:04:07 EST 164750 0 0 SIGABRT present /usr/bin/fi_cq_data 271.1K
Sat 2023-03-11 15:04:14 EST 164810 0 0 SIGABRT present /usr/bin/fi_dgram 271.2K
Sat 2023-03-11 15:04:21 EST 164868 0 0 SIGABRT present /usr/bin/fi_dgram_waitset 270.5K
Sat 2023-03-11 15:04:33 EST 165073 0 0 SIGABRT present /usr/bin/fi_poll 271.8K
Sat 2023-03-11 15:04:40 EST 165131 0 0 SIGABRT present /usr/bin/fi_poll 271.4K
Sat 2023-03-11 15:04:47 EST 165189 0 0 SIGABRT present /usr/bin/fi_rdm 271.8K
Sat 2023-03-11 15:04:54 EST 165246 0 0 SIGABRT present /usr/bin/fi_rdm 270.1K
Sat 2023-03-11 15:05:01 EST 165304 0 0 SIGABRT present /usr/bin/fi_rdm_rma_event 270.6K
Sat 2023-03-11 15:05:09 EST 165361 0 0 SIGABRT present /usr/bin/fi_rdm_rma_trigger 275.5K
Sat 2023-03-11 15:05:19 EST 165515 0 0 SIGABRT present /usr/bin/fi_shared_ctx 677.7K
Sat 2023-03-11 15:05:35 EST 165813 0 0 SIGABRT present /usr/bin/fi_shared_ctx 680.3K
Sat 2023-03-11 15:05:42 EST 165870 0 0 SIGABRT present /usr/bin/fi_rdm_tagged_peek 271.0K
Sat 2023-03-11 15:05:51 EST 165976 0 0 SIGABRT present /usr/bin/fi_rdm_shared_av 270.5K
Sat 2023-03-11 15:06:15 EST 166130 0 0 SIGABRT present /usr/bin/fi_multi_mr 304.5K
Sat 2023-03-11 15:06:28 EST 166237 0 0 SIGABRT present /usr/bin/fi_multi_ep 718.1K
Sat 2023-03-11 15:06:34 EST 166293 0 0 SIGABRT present /usr/bin/fi_recv_cancel 305.0K
Sat 2023-03-11 15:06:45 EST 166400 0 0 SIGABRT present /usr/bin/fi_unexpected_msg 275.3K
Sat 2023-03-11 15:06:53 EST 166458 0 0 SIGABRT present /usr/bin/fi_msg_inject 303.7K
Sat 2023-03-11 15:07:02 EST 166516 0 0 SIGABRT present /usr/bin/fi_msg_inject 306.0K
Sat 2023-03-11 15:07:10 EST 166575 0 0 SIGABRT present /usr/bin/fi_msg_inject 306.8K
Sat 2023-03-11 15:07:18 EST 166633 0 0 SIGABRT present /usr/bin/fi_msg_inject 304.8K
Sat 2023-03-11 15:07:27 EST 166690 0 0 SIGABRT present /usr/bin/fi_bw 307.7K
Sat 2023-03-11 15:07:35 EST 166747 0 0 SIGABRT present /usr/bin/fi_bw 307.1K
Sat 2023-03-11 15:07:43 EST 166855 0 0 SIGABRT present /usr/bin/fi_rdm_multi_client 274.2K
Sat 2023-03-11 15:07:50 EST 166912 0 0 SIGABRT present /usr/bin/fi_rdm_multi_client 274.2K
Sat 2023-03-11 15:08:14 EST 167402 0 0 SIGABRT present /usr/bin/fi_rma_bw 304.3K
Sat 2023-03-11 15:08:22 EST 167461 0 0 SIGABRT present /usr/bin/fi_rma_bw 303.0K
Sat 2023-03-11 15:08:31 EST 167518 0 0 SIGABRT present /usr/bin/fi_rma_bw 301.9K
Sat 2023-03-11 15:08:39 EST 167576 0 0 SIGABRT present /usr/bin/fi_rma_bw 304.6K
Sat 2023-03-11 15:08:48 EST 167633 0 0 SIGABRT present /usr/bin/fi_rma_bw 302.3K
Sat 2023-03-11 15:08:56 EST 167692 0 0 SIGABRT present /usr/bin/fi_rma_bw 301.8K
Sat 2023-03-11 15:09:04 EST 167749 0 0 SIGABRT present /usr/bin/fi_rdm_atomic 305.7K
Sat 2023-03-11 15:09:13 EST 167808 0 0 SIGABRT present /usr/bin/fi_rdm_atomic 303.9K
Sat 2023-03-11 15:09:22 EST 167913 0 0 SIGABRT present /usr/bin/fi_multi_recv 274.4K
Sat 2023-03-11 15:09:33 EST 168019 0 0 SIGABRT present /usr/bin/fi_rdm_pingpong 306.8K
Sat 2023-03-11 15:09:42 EST 168076 0 0 SIGABRT present /usr/bin/fi_rdm_pingpong 303.0K
Sat 2023-03-11 15:09:50 EST 168134 0 0 SIGABRT present /usr/bin/fi_rdm_pingpong 304.5K
Sat 2023-03-11 15:09:59 EST 168192 0 0 SIGABRT present /usr/bin/fi_rdm_pingpong 302.9K
Sat 2023-03-11 15:10:14 EST 168445 0 0 SIGABRT present /usr/bin/fi_rdm_tagged_pingpong 303.9K
Sat 2023-03-11 15:10:22 EST 168502 0 0 SIGABRT present /usr/bin/fi_rdm_tagged_pingpong 306.0K
Sat 2023-03-11 15:10:31 EST 168559 0 0 SIGABRT present /usr/bin/fi_rdm_tagged_pingpong 306.9K
Sat 2023-03-11 15:10:39 EST 168617 0 0 SIGABRT present /usr/bin/fi_rdm_tagged_pingpong 306.3K
Sat 2023-03-11 15:10:48 EST 168676 0 0 SIGABRT present /usr/bin/fi_rdm_tagged_bw 306.3K
Sat 2023-03-11 15:10:56 EST 168733 0 0 SIGABRT present /usr/bin/fi_rdm_tagged_bw 303.4K
Sat 2023-03-11 15:11:04 EST 168791 0 0 SIGABRT present /usr/bin/fi_rdm_tagged_bw 305.8K
Sat 2023-03-11 15:11:13 EST 168849 0 0 SIGABRT present /usr/bin/fi_rdm_tagged_bw 307.0K
Sat 2023-03-11 15:11:22 EST 168906 0 0 SIGABRT present /usr/bin/fi_dgram_pingpong 308.1K
Sat 2023-03-11 15:11:32 EST 169022 0 0 SIGABRT present /usr/bin/fi_multinode 275.1K
Sat 2023-03-11 15:11:32 EST 169019 0 0 SIGABRT present /usr/bin/fi_multinode 272.6K
Sat 2023-03-11 15:11:40 EST 169126 0 0 SIGABRT present /usr/bin/fi_multinode 273.3K
Sat 2023-03-11 15:11:40 EST 169124 0 0 SIGABRT present /usr/bin/fi_multinode 276.2K
Sat 2023-03-11 15:28:08 EST 172866 0 0 SIGABRT present /usr/bin/fi_unexpected_msg 273.9K
total 35012

Expected results:

Additional info: