-
Bug
-
Resolution: Unresolved
-
Major
-
None
-
rhel-8.4.0
-
None
-
Yes
-
None
-
rhel-storage-io-2
-
None
-
False
-
False
-
-
None
-
None
-
None
-
None
-
Unspecified
-
Unspecified
-
Unspecified
-
-
x86_64
-
None
What were you trying to do that didn't work?
On a system provisioned with RHEL-8.4, I noticed that our automation was failing as the system was not able to connect to the nvme namespaces. Below is the job:
https://beaker.engineering.redhat.com/jobs/11956031
Upon further investigation, I triggered a manual discovery and observed the following in the logs:
Nov 20 14:15:08 storageqe-04.fast.eng.rdu2.dc.redhat.com systemd[1]: Started NVMf auto-connect scan upon nvme discovery controller Events.
Nov 20 14:15:08 storageqe-04.fast.eng.rdu2.dc.redhat.com kernel: nvme nvme0: NVME-FC{0}: create association : host wwpn 0x100000109b579dc6 rport wwpn 0x20e100a098cbcac6: NQN "nqn.2014-08.org.nvmexpress.discovery"
Nov 20 14:15:08 storageqe-04.fast.eng.rdu2.dc.redhat.com systemd[1]: Started NVMf auto-connect scan upon nvme discovery controller Events.
Nov 20 14:15:08 storageqe-04.fast.eng.rdu2.dc.redhat.com systemd[1]: Started NVMf auto-connect scan upon nvme discovery controller Events.
Nov 20 14:15:08 storageqe-04.fast.eng.rdu2.dc.redhat.com systemd[1]: Started NVMf auto-connect scan upon nvme discovery controller Events.
Nov 20 14:15:08 storageqe-04.fast.eng.rdu2.dc.redhat.com kernel: nvme nvme0: queue_size 128 > ctrl maxcmd 32, reducing to maxcmd
Nov 20 14:15:08 storageqe-04.fast.eng.rdu2.dc.redhat.com kernel: nvme nvme0: NVME-FC{0}: controller connect complete
Nov 20 14:15:08 storageqe-04.fast.eng.rdu2.dc.redhat.com kernel: nvme nvme0: NVME-FC{0}: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery"
Nov 20 14:15:08 storageqe-04.fast.eng.rdu2.dc.redhat.com kernel: nvme nvme1: NVME-FC{1}: create association : host wwpn 0x100000109b579dc6 rport wwpn 0x20de00a098cbcac6: NQN "nqn.2014-08.org.nvmexpress.discovery"
Nov 20 14:15:09 storageqe-04.fast.eng.rdu2.dc.redhat.com kernel: nvme nvme1: queue_size 128 > ctrl maxcmd 32, reducing to maxcmd
Nov 20 14:15:09 storageqe-04.fast.eng.rdu2.dc.redhat.com kernel: nvme nvme1: NVME-FC{1}: controller connect complete
Nov 20 14:15:09 storageqe-04.fast.eng.rdu2.dc.redhat.com kernel: nvme nvme1: NVME-FC{1}: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery"
Nov 20 14:15:09 storageqe-04.fast.eng.rdu2.dc.redhat.com kernel: nvme nvme2: NVME-FC{2}: create association : host wwpn 0x100000109b579dc7 rport wwpn 0x20df00a098cbcac6: NQN "nqn.2014-08.org.nvmexpress.discovery"
Nov 20 14:15:10 storageqe-04.fast.eng.rdu2.dc.redhat.com kernel: nvme nvme2: queue_size 128 > ctrl maxcmd 32, reducing to maxcmd
Nov 20 14:15:10 storageqe-04.fast.eng.rdu2.dc.redhat.com kernel: nvme nvme2: NVME-FC{2}: controller connect complete
Nov 20 14:15:10 storageqe-04.fast.eng.rdu2.dc.redhat.com kernel: nvme nvme2: NVME-FC{2}: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery"
Nov 20 14:15:10 storageqe-04.fast.eng.rdu2.dc.redhat.com kernel: nvme nvme3: NVME-FC{3}: create association : host wwpn 0x100000109b579dc7 rport wwpn 0x20e000a098cbcac6: NQN "nqn.2014-08.org.nvmexpress.discovery"
Nov 20 14:15:11 storageqe-04.fast.eng.rdu2.dc.redhat.com kernel: nvme nvme3: queue_size 128 > ctrl maxcmd 32, reducing to maxcmd
Nov 20 14:15:11 storageqe-04.fast.eng.rdu2.dc.redhat.com kernel: nvme nvme3: NVME-FC{3}: controller connect complete
Nov 20 14:15:11 storageqe-04.fast.eng.rdu2.dc.redhat.com kernel: nvme nvme3: NVME-FC{3}: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery"
Nov 20 14:15:11 storageqe-04.fast.eng.rdu2.dc.redhat.com kernel: nvme nvme1: Removing ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery"
Nov 20 14:15:11 storageqe-04.fast.eng.rdu2.dc.redhat.com kernel: nvme nvme0: Removing ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery"
Nov 20 14:15:11 storageqe-04.fast.eng.rdu2.dc.redhat.com sh[13834]: skipping unsupported subtype 3
Nov 20 14:15:11 storageqe-04.fast.eng.rdu2.dc.redhat.com systemd[1]: nvmf-connect@--device\x3dnone\t--transport\x3dfc\t--traddr\x3dnn-0x20dd00a098cbcac6:pn-0x20de00a098cbcac6\t--trsvcid\x3dnone\t--host-traddr\x3dnn-0x200000109b579dc6:pn-0x100000109b579dc6.service: Succeeded.
Nov 20 14:15:11 storageqe-04.fast.eng.rdu2.dc.redhat.com sh[13828]: skipping unsupported subtype 3
Nov 20 14:15:11 storageqe-04.fast.eng.rdu2.dc.redhat.com systemd[1]: nvmf-connect@--device\x3dnone\t--transport\x3dfc\t--traddr\x3dnn-0x20dd00a098cbcac6:pn-0x20e100a098cbcac6\t--trsvcid\x3dnone\t--host-traddr\x3dnn-0x200000109b579dc6:pn-0x100000109b579dc6.service: Succeeded.
Nov 20 14:15:11 storageqe-04.fast.eng.rdu2.dc.redhat.com kernel: nvme nvme2: Removing ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery"
Nov 20 14:15:11 storageqe-04.fast.eng.rdu2.dc.redhat.com sh[13838]: skipping unsupported subtype 3
Nov 20 14:15:11 storageqe-04.fast.eng.rdu2.dc.redhat.com systemd[1]: nvmf-connect@--device\x3dnone\t--transport\x3dfc\t--traddr\x3dnn-0x20dd00a098cbcac6:pn-0x20df00a098cbcac6\t--trsvcid\x3dnone\t--host-traddr\x3dnn-0x200000109b579dc7:pn-0x100000109b579dc7.service: Succeeded.
Nov 20 14:15:11 storageqe-04.fast.eng.rdu2.dc.redhat.com kernel: nvme nvme3: Removing ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery"
Nov 20 14:15:11 storageqe-04.fast.eng.rdu2.dc.redhat.com sh[13842]: skipping unsupported subtype 3
Nov 20 14:15:11 storageqe-04.fast.eng.rdu2.dc.redhat.com systemd[1]: nvmf-connect@--device\x3dnone\t--transport\x3dfc\t--traddr\x3dnn-0x20dd00a098cbcac6:pn-0x20e000a098cbcac6\t--trsvcid\x3dnone\t--host-traddr\x3dnn-0x200000109b579dc7:pn-0x100000109b579dc7.service: Succeeded.
'nvme list' does not return any entries:
# nvme list Node SN Model Namespace Usage Format FW Rev ---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
Below is the output of 'nvme discover':
# nvme discover --transport=fc --traddr=nn-0x20dd00a098cbcac6:pn-0x20de00a098cbcac6 --host-traddr=nn-0x200000109b579dc6:pn-0x100000109b579dc6Discovery Log Number of Records 4, Generation counter 26 =====Discovery Log Entry 0====== trtype: fc adrfam: fibre-channel subtype: unrecognized treq: not specified portid: 5 trsvcid: none subnqn: nqn.1992-08.com.netapp:sn.4cd37e89c7e211ecb5b300a098cbcac6:discovery traddr: nn-0x20dd00a098cbcac6:pn-0x20de00a098cbcac6 =====Discovery Log Entry 1====== trtype: fc adrfam: fibre-channel subtype: unrecognized treq: not specified portid: 7 trsvcid: none subnqn: nqn.1992-08.com.netapp:sn.4cd37e89c7e211ecb5b300a098cbcac6:discovery traddr: nn-0x20dd00a098cbcac6:pn-0x20e000a098cbcac6 =====Discovery Log Entry 2====== trtype: fc adrfam: fibre-channel subtype: unrecognized treq: not specified portid: 8 trsvcid: none subnqn: nqn.1992-08.com.netapp:sn.4cd37e89c7e211ecb5b300a098cbcac6:discovery traddr: nn-0x20dd00a098cbcac6:pn-0x20e100a098cbcac6 =====Discovery Log Entry 3====== trtype: fc adrfam: fibre-channel subtype: unrecognized treq: not specified portid: 6 trsvcid: none subnqn: nqn.1992-08.com.netapp:sn.4cd37e89c7e211ecb5b300a098cbcac6:discovery traddr: nn-0x20dd00a098cbcac6:pn-0x20df00a098cbcac6
I provisioned a similar system with RHEL-8.5 and did not see any issues. FYI, since we last tested NVMe-FC on RHEL-8.4, the NetApp A700 storage array has been upgraded numerous times.
What is the impact of this issue to you?
Unable to utilize nvme-fc namespaces in RHEL-8.4
Please provide the package NVR for which the bug is seen:
[root@storageqe-04 ~]# uname -r 4.18.0-305.el8.x86_64 [root@storageqe-04 ~]# rpm -qa nvme-cli nvme-cli-1.12-3.el8.x86_64