Customer upgraded OCP 4.12.29 to OCP 4.12.45 and 'sriov-network-config-daemon' pod no longer recognizes the Intel XXV710 NICs in the PCIe slots. We can manually 'echo' the number of VFs into the NIC but the SRIOV Pods are not automatically doing it.

OCP Version: 4.12.45
SRIOV Operator: sriov-network-operator.v4.12.0-202306261055

This is the first cluster upgraded to 4.12.45 and this previously worked on OCP 4.12.29 but there aren't any nodes stuck in Draining, MCPs are up to date and all have been rebooted. The nodes have the proper labels but the SRIOV pods on the node complain 'no such device' for the PCIe device where our SRIOV NIC is.

$ oc adm upgrade --to='4.12.45'
$ oc get node <sriovcapablenode> -o yaml | yq -e '.status.capacity'
$ ssh core@<node> 'sudo cat /sys/class/net/ens3f0/device/sriov_numvfs'
$ ssh core@<node> 'sudo cat /sys/class/net/ens3f1/device/sriov_numvfs'

$ oc logs -n openshift-sriov-network-operator sriov-network-config-daemon-qjfg6 -c sriov-network-config-daemon
2023-12-13T16:51:52.603041880Z I1213 16:51:52.603037 3186691 utils.go:488] getNetdevMTU(): get MTU for device 0000:d8:00.0
2023-12-13T16:51:52.603078629Z I1213 16:51:52.603074 3186691 utils.go:483] tryGetInterfaceName(): name is ens3f0
2023-12-13T16:51:52.603134920Z I1213 16:51:52.603130 3186691 utils.go:483] tryGetInterfaceName(): name is ens3f0
2023-12-13T16:51:52.603139913Z I1213 16:51:52.603134 3186691 utils.go:509] getNetDevMac(): get Mac for device ens3f0
2023-12-13T16:51:52.603159638Z I1213 16:51:52.603155 3186691 utils.go:521] getNetDevLinkSpeed(): get LinkSpeed for device ens3f0
2023-12-13T16:51:52.603182523Z I1213 16:51:52.603178 3186691 utils.go:668] getLinkType(): Device 0000:d8:00.0
2023-12-13T16:51:52.603300686Z I1213 16:51:52.603291 3186691 utils.go:721] GetNicSriovMode(): device 0000:d8:00.0
2023-12-13T16:51:52.603355659Z W1213 16:51:52.603349 3186691 utils.go:128] DiscoverSriovDevices(): unable to get device mode 0000:d8:00.0 "no such device"
2023-12-13T16:51:52.603405837Z I1213 16:51:52.603400 3186691 utils.go:488] getNetdevMTU(): get MTU for device 0000:d8:00.1
2023-12-13T16:51:52.603441971Z I1213 16:51:52.603437 3186691 utils.go:483] tryGetInterfaceName(): name is ens3f1
2023-12-13T16:51:52.603493758Z I1213 16:51:52.603489 3186691 utils.go:483] tryGetInterfaceName(): name is ens3f1
2023-12-13T16:51:52.603498801Z I1213 16:51:52.603493 3186691 utils.go:509] getNetDevMac(): get Mac for device ens3f1
2023-12-13T16:51:52.603521851Z I1213 16:51:52.603517 3186691 utils.go:521] getNetDevLinkSpeed(): get LinkSpeed for device ens3f1
2023-12-13T16:51:52.603545198Z I1213 16:51:52.603540 3186691 utils.go:668] getLinkType(): Device 0000:d8:00.1
2023-12-13T16:51:52.603654714Z I1213 16:51:52.603645 3186691 utils.go:721] GetNicSriovMode(): device 0000:d8:00.1
2023-12-13T16:51:52.603708731Z W1213 16:51:52.603703 3186691 utils.go:128] DiscoverSriovDevices(): unable to get device mode 0000:d8:00.1 "no such device"
2023-12-13T16:51:52.608317321Z I1213 16:51:52.608301 3186691 writer.go:173] setNodeStateStatus(): syncStatus: InProgress, lastSyncError: 

$ egrep "d8:00.|Initial" sosreport-worker-18-2023-12-14-vgtkjuw/lspci
d8:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller XXV710 for 25GbE SFP28 [8086:158b] (rev 02)
        Initial VFs: 64, Total VFs: 64, Number of VFs: 0, Function Dependency Link: 00
d8:00.1 Ethernet controller [0200]: Intel Corporation Ethernet Controller XXV710 for 25GbE SFP28 [8086:158b] (rev 02)
        Initial VFs: 64, Total VFs: 64, Number of VFs: 0, Function Dependency Link: 01 

$ cat sosreport-worker-18-2023-12-14-vgtkjuw/sos_commands/networking/ethtool_-i_ens3f0 
driver: i40e
version: 4.18.0-372.82.1.el8_6.x86_64
firmware-version: 7.20 0x80007fdf 1.2585.0
bus-info: 0000:d8:00.0 

$ cat sosreport-worker-18-2023-12-14-vgtkjuw/sos_commands/networking/ethtool_-i_ens3f1
driver: i40e
version: 4.18.0-372.82.1.el8_6.x86_64
firmware-version: 7.20 0x80007fdf 1.2585.0
bus-info: 0000:d8:00.1 

$ echo 64 > /sys/class/net/ens3f0/device/sriov_numvfs 

$ oc delete pod -n openshift-sriov-network-operator $(oc get pods -n openshift-sriov-network-operator -o wide | awk '/worker-18/ {print $1}' | xargs echo) 

$ oc logs -n openshift-sriov-network-operator $(oc get pods -n openshift-sriov-network-operator -l app=sriov-network-config-daemon -o wide | awk '/worker-18/ {print $1}') -c sriov-network-config-daemon 

$ oc get SriovNetworkNodePolicy -n openshift-sriov-network-operator ens3f0resspk -o yaml
spec:
deviceType: vfio-pci
isRdma: false
nicSelector:
 deviceID: 158b
 pfNames:
 - ens3f0  ## THIS WAS ens3f0#10-60
 vendor: "8086"
nodeSelector:
 feature.node.kubernetes.io/network-sriov.capable: "true"
numVfs: 32  ## THIS WAS 64
priority: 99
resourceName: ens3f0resspk 

$ oc get SriovNetworkNodePolicy -n openshift-sriov-network-operator ens3f1resspk -o yaml
spec:
deviceType: vfio-pci
isRdma: false
nicSelector:
 deviceID: 158b
 pfNames:
 - ens3f1  ## THIS WAS ens3f1#10-60
 vendor: "8086"
nodeSelector:
 feature.node.kubernetes.io/network-sriov.capable: "true"
numVfs: 32  ## THIS WAS 64
priority: 99
resourceName: ens3f1resspk 

$ oc adm drain worker-18.<redacted> --force --grace-period=0 --ignore-daemonsets --delete-emptydir-data --disable-eviction
$ oc adm uncordon worker-18.<redacted> 

The VFs should be created on the nodes based on the SriovNetworkNodePolicy but instead, it's complaining about "no such device" for the PCIe NICs that are actually in the location it complains about

- ens3f0 lives at PCI device 0000:d8:00.0 and can support up to 64VFs as Intel XXV710 for 25GbE SFP28 [8086:158b]
- ens3f1 lives at PCI device 0000:d8:00.1 and can support up to 64VFs as Intel XXV710 for 25GbE SFP28 [8086:158b]
- SRIOV capabilities are enabled in the BIOS
- SRIOV capabilities are enabled in the KernelArgs
- The cluster was just upgraded from OCP 4.12.29 to OCP 4.12.45
- The cluster is using 'sriov-network-operator.v4.12.0-202306261055' and on 'stable' channel