Description of problem:

Version-Release number of selected component (if applicable):
$ rpm -q fabtests
fabtests-1.15.1-1.el9.x86_64

07:00.0 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5]

How reproducible:
Always

Steps to Reproduce:

cd /usr/share/fabtests/pytest; pytest --provider="verbs\;ofi_rxd" --server-id=172.31.0.182 --client-id=172.31.0.183 -v -rfE --tb=short -m 'unit or multinode or functional or standard or ubertest_all' --timeout=120 default
============================= test session starts ==============================
platform linux – Python 3.9.16, pytest-6.2.2, py-1.10.0, pluggy-0.13.1 – /usr/bin/python3
cachedir: .pytest_cache
rootdir: /usr/share/fabtests/pytest, configfile: pytest.ini
collecting ... collected 138 items / 52 deselected / 86 selected

default/test_av.py::test_av SKIPPED (ENODATA) [ 1%]
default/test_av.py::test_av_xfer[rdm] PASSED [ 2%]
default/test_av.py::test_av_xfer[dgram] SKIPPED (ENODATA) [ 3%]
default/test_cm.py::test_cm_data SKIPPED (ENODATA) [ 4%]
default/test_cntr.py::test_cntr PASSED [ 5%]
default/test_cq.py::test_cq PASSED [ 6%]
default/test_cq.py::test_cq_data[msg] SKIPPED (ENODATA) [ 8%]
default/test_cq.py::test_cq_data[rdm] PASSED [ 9%]
default/test_cq.py::test_cq_data[dgram] SKIPPED (ENODATA) [ 10%]
default/test_dgram.py::test_dgram_g00n13s SKIPPED (ENODATA) [ 11%]
default/test_dgram.py::test_dgram SKIPPED (ENODATA) [ 12%]
default/test_dgram.py::test_dgram_waitset SKIPPED (ENODATA) [ 13%]
default/test_dgram.py::test_dgram_pingpong[with_prefix-standard] SKIPPED [ 15%]
default/test_dgram.py::test_dgram_pingpong[wout_prefix-standard] SKIPPED [ 16%]
default/test_dom.py::test_dom PASSED [ 17%]
default/test_eq.py::test_eq PASSED [ 18%]
default/test_getinfo.py::test_getinfo PASSED [ 19%]
default/test_mr.py::test_mr PASSED [ 20%]
default/test_mr.py::test_multi_mr[msg] SKIPPED (ENODATA) [ 22%]
default/test_mr.py::test_multi_mr[rdm] SKIPPED (ENOSYS) [ 23%]
default/test_msg.py::test_msg_g00n13s SKIPPED (ENODATA) [ 24%]
default/test_msg.py::test_msg SKIPPED (ENODATA) [ 25%]
default/test_msg.py::test_msg_epoll SKIPPED (ENODATA) [ 26%]
default/test_msg.py::test_msg_sockets SKIPPED (ENODATA) [ 27%]
default/test_msg.py::test_msg_pingpong[with_prefix-with_datacheck-standard] SKIPPED [ 29%]
default/test_msg.py::test_msg_pingpong[wout_prefix-with_datacheck-standard] SKIPPED [ 30%]
default/test_msg.py::test_msg_pingpong[wout_prefix-wout_datacheck-standard] SKIPPED [ 31%]
default/test_msg.py::test_msg_pingpong[with_prefix-wout_datacheck-standard] SKIPPED [ 32%]
default/test_msg.py::test_msg_bw[wout_datacheck-standard] SKIPPED (E...) [ 33%]
default/test_msg.py::test_msg_bw[with_datacheck-standard] SKIPPED (E...) [ 34%]
default/test_msg_inject.py::test_msg_inject[inject-sendmsg] FAILED [ 36%]
default/test_msg_inject.py::test_msg_inject[inject-post_tx] FAILED [ 37%]
default/test_msg_inject.py::test_msg_inject[inj_complete-sendmsg] PASSED [ 38%]
default/test_msg_inject.py::test_msg_inject[inj_complete-post_tx] PASSED [ 39%]
default/test_multi_recv.py::test_multi_recv[standard-rdm] PASSED [ 40%]
default/test_multi_recv.py::test_multi_recv[standard-msg] SKIPPED (E...) [ 41%]
default/test_multinode.py::test_multinode[msg] PASSED [ 43%]
default/test_multinode.py::test_multinode[rma] PASSED [ 44%]
default/test_multinode.py::test_multinode_coll PASSED [ 45%]
default/test_poll.py::test_poll[queue] PASSED [ 46%]
default/test_poll.py::test_poll[counter] PASSED [ 47%]
default/test_rdm.py::test_rdm_g00n13s SKIPPED (ENODATA) [ 48%]
default/test_rdm.py::test_rdm[transmit_complete] PASSED [ 50%]
default/test_rdm.py::test_rdm_bw_functional[transmit_complete] PASSED [ 51%]
default/test_rdm.py::test_rdm_atomic[transmit_complete-standard] PASSED [ 52%]
default/test_rdm.py::test_rdm_pingpong[with_prefix-with_datacheck-transmit_complete-standard] SKIPPED [ 53%]
default/test_rdm.py::test_rdm_pingpong[with_prefix-with_datacheck-delivery_complete-standard] SKIPPED [ 54%]
default/test_rdm.py::test_rdm_pingpong[wout_prefix-with_datacheck-delivery_complete-standard] PASSED [ 55%]
default/test_rdm.py::test_rdm_pingpong[wout_prefix-wout_datacheck-delivery_complete-standard] PASSED [ 56%]
default/test_rdm.py::test_rdm_pingpong[wout_prefix-wout_datacheck-transmit_complete-standard] PASSED [ 58%]
default/test_rdm.py::test_rdm_pingpong[with_prefix-wout_datacheck-delivery_complete-standard] SKIPPED [ 59%]
default/test_rdm.py::test_rdm_tagged_pingpong[wout_datacheck-delivery_complete-standard] PASSED [ 60%]
default/test_rdm.py::test_rdm_tagged_bw[wout_datacheck-delivery_complete-standard] PASSED [ 61%]
default/test_rdm.py::test_rdm_pingpong[with_prefix-wout_datacheck-transmit_complete-standard] SKIPPED [ 62%]
default/test_rdm.py::test_rdm_tagged_pingpong[wout_datacheck-transmit_complete-standard] PASSED [ 63%]
default/test_rdm.py::test_rdm_tagged_bw[wout_datacheck-transmit_complete-standard] PASSED [ 65%]
default/test_rdm.py::test_rdm_pingpong[wout_prefix-with_datacheck-transmit_complete-standard] PASSED [ 66%]
default/test_rdm.py::test_rdm_tagged_pingpong[with_datacheck-delivery_complete-standard] PASSED [ 67%]
default/test_rdm.py::test_rdm_tagged_bw[with_datacheck-delivery_complete-standard] PASSED [ 68%]
default/test_rdm.py::test_rdm[delivery_complete] PASSED [ 69%]
default/test_rdm.py::test_rdm_bw_functional[delivery_complete] PASSED [ 70%]
default/test_rdm.py::test_rdm_atomic[delivery_complete-standard] PASSED [ 72%]
default/test_rdm.py::test_rdm_tagged_pingpong[with_datacheck-transmit_complete-standard] PASSED [ 73%]
default/test_rdm.py::test_rdm_tagged_bw[with_datacheck-transmit_complete-standard] PASSED [ 74%]
default/test_rdm.py::test_rdm_rma_event PASSED [ 75%]
default/test_rdm.py::test_rdm_rma_trigger SKIPPED (ENODATA) [ 76%]
default/test_rdm.py::test_rdm_tagged_peek PASSED [ 77%]
default/test_rdm.py::test_rdm_shared_av SKIPPED (ENODATA) [ 79%]
default/test_rdm.py::test_rdm_cntr_pingpong[standard] PASSED [ 80%]
default/test_recv_cancel.py::test_recv_cancel PASSED [ 81%]
default/test_rma_bw.py::test_rma_bw[transmit_complete-standard-msg-read] SKIPPED [ 82%]
default/test_rma_bw.py::test_rma_bw[transmit_complete-standard-msg-writedata] SKIPPED [ 83%]
default/test_rma_bw.py::test_rma_bw[transmit_complete-standard-msg-write] SKIPPED [ 84%]
default/test_rma_bw.py::test_rma_bw[transmit_complete-standard-rdm-read] PASSED [ 86%]
default/test_rma_bw.py::test_rma_bw[transmit_complete-standard-rdm-writedata] PASSED [ 87%]
default/test_rma_bw.py::test_rma_bw[transmit_complete-standard-rdm-write] PASSED [ 88%]
default/test_rma_bw.py::test_rma_bw[delivery_complete-standard-msg-read] SKIPPED [ 89%]
default/test_rma_bw.py::test_rma_bw[delivery_complete-standard-msg-writedata] SKIPPED [ 90%]
default/test_rma_bw.py::test_rma_bw[delivery_complete-standard-msg-write] SKIPPED [ 91%]
default/test_rma_bw.py::test_rma_bw[delivery_complete-standard-rdm-read] PASSED [ 93%]
default/test_rma_bw.py::test_rma_bw[delivery_complete-standard-rdm-writedata] PASSED [ 94%]
default/test_rma_bw.py::test_rma_bw[delivery_complete-standard-rdm-write] PASSED [ 95%]
default/test_scalable_ep.py::test_scalable_ep SKIPPED (ENODATA) [ 96%]
default/test_ubertest.py::test_ubertest[all] SKIPPED (no config file) [ 97%]
default/test_unexpected_msg.py::test_unexpected_msg[msg] SKIPPED (EN...) [ 98%]
default/test_unexpected_msg.py::test_unexpected_msg[rdm] PASSED [100%]

=================================== FAILURES ===================================
_______________________ test_msg_inject[inject-sendmsg] ________________________
default/test_msg_inject.py:14: in test_msg_inject
test.run()
common.py:208: in run
check_returncode(client_process.returncode, strict)
common.py:29: in check_returncode
pytest.fail(error_msg)
E Failed: returncode 1
----------------------------- Captured stdout call -----------------------------

server_command: ssh -n -o StrictHostKeyChecking=no -o ConnectTimeout=2 -o BatchMode=yes 172.31.0.182 timeout 120 fi_msg_inject -N -A inject -p "verbs\;ofi_rxd" -s 172.31.0.182
server_stdout:
Start testing FI_INJECT
GOOD: Completed FI_INJECT Testing

client_command: ssh -n -o StrictHostKeyChecking=no -o ConnectTimeout=2 -o BatchMode=yes 172.31.0.183 timeout 120 fi_msg_inject -N -A inject -p "verbs\;ofi_rxd" -s 172.31.0.183 172.31.0.182
client_stdout:
ERROR: op ctx=0x564848d806a0 cq_ctx=(nil)
ft_cq_read_verify(): functional/msg_inject.c:83, ret=-256 (Unspecified error)
Start testing FI_INJECT

_______________________ test_msg_inject[inject-post_tx] ________________________
default/test_msg_inject.py:14: in test_msg_inject
test.run()
common.py:208: in run
check_returncode(client_process.returncode, strict)
common.py:29: in check_returncode
pytest.fail(error_msg)
E Failed: returncode 1
----------------------------- Captured stdout call -----------------------------

server_command: ssh -n -o StrictHostKeyChecking=no -o ConnectTimeout=2 -o BatchMode=yes 172.31.0.182 timeout 120 fi_msg_inject -A inject -p "verbs\;ofi_rxd" -s 172.31.0.182
server_stdout:
Start testing FI_INJECT
GOOD: Completed FI_INJECT Testing

client_command: ssh -n -o StrictHostKeyChecking=no -o ConnectTimeout=2 -o BatchMode=yes 172.31.0.183 timeout 120 fi_msg_inject -A inject -p "verbs\;ofi_rxd" -s 172.31.0.183 172.31.0.182
client_stdout:
ERROR: op ctx=0x558586b836a0 cq_ctx=(nil)
ft_cq_read_verify(): functional/msg_inject.c:83, ret=-256 (Unspecified error)
Start testing FI_INJECT

=========================== short test summary info ============================
FAILED default/test_msg_inject.py::test_msg_inject[inject-sendmsg] - Failed: ...
FAILED default/test_msg_inject.py::test_msg_inject[inject-post_tx] - Failed: ...
===== 2 failed, 45 passed, 39 skipped, 52 deselected in 213.08s (0:03:33) ======
~

Actual results:

Expected results:

Additional info: