BF-2 on newest firmware 24.40.1000 fails to switch to NIC mode via SriovNetworkOperator

    On the newest BF-2 firmware, the sriov-network-operator no longer succeeds in switching the BF-2 to nic mode: https://github.com/openshift/sriov-network-operator/blob/master/bindata/scripts/bf2-switch-mode.sh

    24.40.1000 tested with sriov-network-operator build from master branch on 2/22/24 on OCP 4.14 nightly

    Everytime

    1.deploy sriov network operator
    2. create MCP / MC to set BF-2 to nicmode via systemd. Apply label "--overwrite=true feature.node.kubernetes.io/network-sriov.capable=true" to workers with BF-2


apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfigPool
metadata:
  name: sriov
spec:
  machineConfigSelector:
    matchExpressions:
      - {key: machineconfiguration.openshift.io/role, operator: In, values: [worker,sriov]}
  nodeSelector:
    matchLabels:
      feature.node.kubernetes.io/network-sriov.capable: "true"

apiVersion: machineconfiguration.openshift.io/v1kind: MachineConfigmetadata:  labels:    machineconfiguration.openshift.io/role: sriov  name: 99-bf2-dpuspec:  config:    ignition:      version: 3.2.0    storage:      files:      - contents:          source: data:text/plain;charset=utf-8;base64,ZmluZF9jb250YWluZXIoKSB7CiAgY3JpY3RsIHBzIC1vIGpzb24gfCBqcSAtciAnLmNvbnRhaW5lcnNbXSB8IHNlbGVjdCgubWV0YWRhdGEubmFtZT09InNyaW92LW5ldHdvcmstY29uZmlnLWRhZW1vbiIpIHwgLmlkJwp9CnVudGlsIG91dHB1dD0kKGZpbmRfY29udGFpbmVyKTsgW1sgLW4gIiRvdXRwdXQiIF1dOyBkbwogIGVjaG8gIndhaXRpbmcgZm9yIGNvbnRhaW5lciB0byBjb21lIHVwIgogIHNsZWVwIDE7CmRvbmUKISBzdWRvIGNyaWN0bCBleGVjICRvdXRwdXQgL2JpbmRhdGEvc2NyaXB0cy9iZjItc3dpdGNoLW1vZGUuc2ggIiRAIgo=        mode: 0755        overwrite: true        path: /etc/default/switch_in_sriov_config_daemon.sh    systemd:      units:        - name: dpu-switch.service          enabled: true          contents: |            [Unit]            Description=Switch BlueField2 card to NIC/DPU mode            RequiresMountsFor=%t/containers            Wants=network.target            After=network-online.target kubelet.service            [Service]            SuccessExitStatus=0 120            RemainAfterExit=True            ExecStart=/bin/bash -c '/etc/default/switch_in_sriov_config_daemon.sh nic || shutdown -r now'            Type=oneshot            [Install]            WantedBy=multi-user.target


    3. After reboot the BF-2 will silently fail to switch to NIC mode resulting in a bootloop if the sriov network node policy is applied
    

    BF-2 remains in DPU mode

    BF-2 will be set to NIC mode on reboot