Details
-
Bug
-
Resolution: Done-Errata
-
Major
-
None
-
4.14
-
None
-
No
-
5
-
NHE Sprint 237
-
1
-
False
-
Description
Description of problem:
When enabling hardware offload in NIC mode, bf2-switch-mode.sh is run to switch the nic into NIC mode. The changes are made via mstconfig, and then applied by rebooting the nic with mstfwreset(). This command is failing with the following error message:
May 24 18:21:56 worker-225 bash[8026]: Switching to NIC mode. [17/1481] May 24Failed to find catalog entry: Invalid argument 18:21:56 worker-225 bash[8026]: Device #1: May 24 18:21:56 worker-225 bash[8026]: ---------- May 24 18:21:56 worker-225 bash[8026]: Device type: BlueField2 May 24 18:21:56 worker-225 bash[8026]: Name: MBF2H332A-AEEO_Ax_Bx May 24 18:21:56 worker-225 bash[8026]: Description: BlueField-2 P-Series DPU 25GbE Dual-Port SFP56; PCIe Gen4 x8; Crypto Enabled; 16GB on-board DDR; 1GbE OOB management; HHHL May 24 18:21:56 worker-225 bash[8026]: Device: 0000:ca:00.0 May 24 18:21:56 worker-225 bash[8026]: Configurations: Next Boot New May 24 18:21:56 worker-225 bash[8026]: INTERNAL_CPU_MODEL EMBEDDED_CPU(1) EMBEDDED_CPU(1) May 24 18:21:56 worker-225 bash[8026]: INTERNAL_CPU_PAGE_SUPPLIER ECPF(0) EXT_HOST_PF(1) May 24 18:21:56 worker-225 bash[8026]: INTERNAL_CPU_ESWITCH_MANAGER ECPF(0) EXT_HOST_PF(1) May 24 18:21:56 worker-225 bash[8026]: INTERNAL_CPU_IB_VPORT0 ECPF(0) EXT_HOST_PF(1) May 24 18:21:56 worker-225 bash[8026]: INTERNAL_CPU_OFFLOAD_ENGINE ENABLED(0) DISABLED(1) May 24 18:21:56 worker-225 bash[8026]: Apply new Configuration? (y/n) [n] : y May 24 18:21:56 worker-225 bash[8026]: Applying... Done! May 24 18:21:56 worker-225 bash[8026]: -I- Please reboot machine to load new configurations. May 24 18:21:56 worker-225 bash[8026]: Minimal reset level for device, 0000:ca:00.0: May 24 18:21:56 worker-225 bash[8026]: 3: Driver restart and PCI reset May 24 18:21:56 worker-225 bash[8026]: Continue with reset?[y/N] y May 24 18:21:56 worker-225 bash[8026]: Failed May 24 18:21:56 worker-225 bash[8026]: -E- failed to run 'setpci -s 0000:c9:02.0 0x0.w'. May 24 18:21:56 worker-225 bash[8026]: time="2023-05-24T18:21:56Z" level=fatal msg="execing command in container: command terminated with exit code 1" May 24 18:21:56 worker-225 sudo[7523]: pam_unix(sudo:session): session closed for user root May 24 18:21:56 worker-225 systemd[1]: Finished Switch BlueField2 card to NIC/DPU mode. May 24 18:21:56 worker-225 systemd[1]: session-c1.scope: Deactivated successfully. May 24 18:21:56 worker-225 systemd[1]: Reached target Multi-User System. May 24 18:21:56 worker-225 systemd[1]: Reached target Graphical Interface. May 24 18:21:56 worker-225 systemd[1]: Afterburn (Check In) was skipped because no trigger condition checks were met. May 24 18:21:56 worker-225 systemd[1]: Afterburn (Firstboot Check In) was skipped because of an unmet condition check (ConditionFirstBoot=yes). May 24 18:21:56 worker-225 systemd[1]: Starting Record Runlevel Change in UTMP... May 24 18:21:56 worker-225 systemd[1]: systemd-update-utmp-runlevel.service: Deactivated successfully. May 24 18:21:56 worker-225 systemd[1]: Finished Record Runlevel Change in UTMP. May 24 18:21:56 worker-225 systemd[1]: Startup finished in 1.711s (kernel) + 9.102s (initrd) + 2min 31.545s (userspace) = 2min 42.359s.
In previous versions of the bf firmware, we would recover from this failure, as the node would be rebooted at some point in the process and apply the changes. However, with the newest firmware update (5/18/2023) changes are not being applied on reboot and require mstfwreset() succeed, which results in the nodes getting stuck in a boot loop when the sriov pool is applied.
This results in switchdev-configuration-before-nm.service failing
Version-Release number of selected component (if applicable):
How reproducible:
Every time
Steps to Reproduce:
1. Install an OCP cluster w/ BF2 in default state 2. Install Sriov Network Operator 3. Enable hw-offload (apply sriov-workload-node-policy, management policy, sriov pool config, nad)
Actual results:
Failed Units: 2 NetworkManager-wait-online.service switchdev-configuration-before-nm.service [root@worker-225 /]# python get_mode nic [root@worker-225 /]# mstconfig -e -d 0000:ca:00.0 q Device #1: ---------- Device type: BlueField2 Name: MBF2H332A-AEEO_Ax_Bx Description: BlueField-2 P-Series DPU 25GbE Dual-Port SFP56; PCIe Gen4 x8; Crypto Enabled; 16GB on-board DDR; 1GbE OOB management; HHHL Device: 0000:ca:00.0 Configurations: Default Current Next Boot MEMIC_BAR_SIZE 0 0 0 MEMIC_SIZE_LIMIT _256KB(1) _256KB(1) _256KB(1) HOST_CHAINING_MODE DISABLED(0) DISABLED(0) DISABLED(0) HOST_CHAINING_CACHE_DISABLE False(0) False(0) False(0) HOST_CHAINING_DESCRIPTORS Array[0..7] Array[0..7] Array[0..7] HOST_CHAINING_TOTAL_BUFFER_SIZE Array[0..7] Array[0..7] Array[0..7] INTERNAL_CPU_MODEL EMBEDDED_CPU(1) EMBEDDED_CPU(1) EMBEDDED_CPU(1) * INTERNAL_CPU_PAGE_SUPPLIER ECPF(0) ECPF(0) EXT_HOST_PF(1) * INTERNAL_CPU_ESWITCH_MANAGER ECPF(0) ECPF(0) EXT_HOST_PF(1) * INTERNAL_CPU_IB_VPORT0 ECPF(0) ECPF(0) EXT_HOST_PF(1) * INTERNAL_CPU_OFFLOAD_ENGINE ENABLED(0) ENABLED(0) DISABLED(1)
Expected results:
[root@worker-225 /]# mstconfig -e -d 0000:ca:00.0 q Device #1: ---------- Device type: BlueField2 Name: MBF2H332A-AEEO_Ax_Bx Description: BlueField-2 P-Series DPU 25GbE Dual-Port SFP56; PCIe Gen4 x8; Crypto Enabled; 16GB on-board DDR; 1GbE OOB management; HHHL Device: 0000:ca:00.0 Configurations: Default Current Next Boot MEMIC_BAR_SIZE 0 0 0 MEMIC_SIZE_LIMIT _256KB(1) _256KB(1) _256KB(1) HOST_CHAINING_MODE DISABLED(0) DISABLED(0) DISABLED(0) HOST_CHAINING_CACHE_DISABLE False(0) False(0) False(0) HOST_CHAINING_DESCRIPTORS Array[0..7] Array[0..7] Array[0..7] HOST_CHAINING_TOTAL_BUFFER_SIZE Array[0..7] Array[0..7] Array[0..7] INTERNAL_CPU_MODEL EMBEDDED_CPU(1) EMBEDDED_CPU(1) EMBEDDED_CPU(1) * INTERNAL_CPU_PAGE_SUPPLIER ECPF(0) EXT_HOST_PF(1) EXT_HOST_PF(1) * INTERNAL_CPU_ESWITCH_MANAGER ECPF(0) EXT_HOST_PF(1) EXT_HOST_PF(1) * INTERNAL_CPU_IB_VPORT0 ECPF(0) EXT_HOST_PF(1) EXT_HOST_PF(1) * INTERNAL_CPU_OFFLOAD_ENGINE ENABLED(0) DISABLED(1) DISABLED(1)
Additional info:
