Errata Tool added a comment - 2024/10/09 1:37 AM Since the problem described in this issue should be resolved in a recent advisory, it has been closed. For information on the advisory (OpenShift Container Platform 4.16.16 security and extras update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2024:7598

Franck Baudin added a comment - 2024/06/21 10:19 AM apanatto@redhat.com can you confirm that this is not a regression but a bug that has always been there? Thanks!

Andrea Panattoni added a comment - 2024/06/07 2:14 PM As a workaround, deleting and recreating all the remaining SriovNetworkNodePolicies solves the issue. This should help reduce the bug's severity. We should make it before 4.16 GA. I'll update the release notes fields as soon as I imagine to miss the deadline

Andrea Panattoni added a comment - 2024/06/06 11:35 AM TL;DR The problem is that the sriov-network-config-daemon is not able to reconcile the desired state with the current configuration when a vfio-pci policy is deleted, but other policies on the same PF are still in place. Working on a fix. Detailed analysis: SriovNetworkNodeState spec: ... - name: ens2f3 numVfs: 48 pciAddress: "0000:12:00.3" vfGroups: - deviceType: vfio-pci policyName: snnp1 resourceName: snnp1 vfRange: 32-33 config daemon logs after setting `SriovOperatorConfig.Spec.LogLevel = 2`. VF 34 and 35 have been previously configured by the deleted snnp2. snnp1 keeps `numVfs = 48`: 2024-06-06T10:13:25.89985124Z LEVEL(-2) generic/generic_plugin.go:130 generic plugin needDrainNode() {"current": [ ... {"name":"ens2f3","mac":"b4:96:91:d2:e6:49","driver":"ice","pciAddress":"0000:12:00.3","vendor":"8086","deviceID":"1593","mtu":1500,"numVfs":48,"linkSpeed":"-1 Mb/s","linkType":"ETH","eSwitchMode":"legacy","totalvfs":64,"Vfs":[ ... {"name":"ens2f3v31","mac":"62:1c:60:56:a0:55","driver":"iavf","pciAddress":"0000:12:1c.7","vendor":"8086","deviceID":"1889","mtu":1500,"vfID":31}, {"driver":"vfio-pci","pciAddress":"0000:12:1d.0","vendor":"8086","deviceID":"1889","vfID":32}, {"driver":"vfio-pci","pciAddress":"0000:12:1d.1","vendor":"8086","deviceID":"1889","vfID":33}, {"driver":"vfio-pci","pciAddress":"0000:12:1d.2","vendor":"8086","deviceID":"1889","vfID":34}, {"driver":"vfio-pci","pciAddress":"0000:12:1d.3","vendor":"8086","deviceID":"1889","vfID":35}, {"name":"ens2f3v36","mac":"ea:09:b2:75:dd:ee","driver":"iavf","pciAddress":"0000:12:1d.4","vendor":"8086","deviceID":"1889","mtu":1500,"vfID":36}, ... ]}], "desired": [ ... {"pciAddress":"0000:12:00.3","numVfs":48,"name":"ens2f3","vfGroups":[ {"resourceName":"snnp1","deviceType":"vfio-pci","vfRange":"32-33","policyName":"snnp1"}]}]} and they trigger the reconfiguration in NeedToUpdateSriov(...) but they doesn't get reconfigured ( configSriovVFDevices(...) ). The problem does not occur if the deleted SriovNetworkNodePolicy has `spec.deviceType: netdevice`: # Starting with no snnp2 [kni@registry.kni-qe-0 ~]$ oc get sriovnetworknodepolicy NAME AGE pci-sriov-net-f1 47h snnp1 18h sriov-nnp-du-fh 22h sriov-nnp-du-mh 22h # creating vfio-pci snnp2 [kni@registry.kni-qe-0 ~]$ cat <<EOF | oc create -f - > apiVersion: sriovnetwork.openshift.io/v1 > kind: SriovNetworkNodePolicy > metadata: > name: snnp2 > namespace: openshift-sriov-network-operator > spec: > deviceType: vfio-pci > isRdma: false > nicSelector: > pfNames: > - ens2f3#34-35 > nodeSelector: > node-role.kubernetes.io/master: "" > numVfs: 48 > resourceName: snnp2 > EOF sriovnetworknodepolicy.sriovnetwork.openshift.io/snnp2 created # device plugin works correctly [kni@registry.kni-qe-0 ~]$ oc get pods NAME READY STATUS RESTARTS AGE ... sriov-device-plugin-7mftg 1/1 Running 0 2m56s # deleting the vfio-pci policy [kni@registry.kni-qe-0 ~]$ oc delete sriovnetworknodepolicy snnp2 sriovnetworknodepolicy.sriovnetwork.openshift.io "snnp2" deleted # device plugin problem occurs [kni@registry.kni-qe-0 ~]$ oc get pods NAME READY STATUS RESTARTS AGE ... sriov-device-plugin-sd2p2 0/1 Pending 0 1s sriov-device-plugin-ztzm4 0/1 Terminating 0 5s # creating a netdevice policy [kni@registry.kni-qe-0 ~]$ cat <<EOF | oc create -f - > apiVersion: sriovnetwork.openshift.io/v1 > kind: SriovNetworkNodePolicy > metadata: > name: snnp2-netdevice > namespace: openshift-sriov-network-operator > spec: > deviceType: netdevice > isRdma: false > nicSelector: > pfNames: > - ens2f3#34-35 > nodeSelector: > node-role.kubernetes.io/master: "" > numVfs: 48 > resourceName: snnp2 > EOF sriovnetworknodepolicy.sriovnetwork.openshift.io/snnp2-netdevice created # device plugin works correctly [kni@registry.kni-qe-0 ~]$ oc get pods NAME READY STATUS RESTARTS AGE sriov-device-plugin-dtv8m 1/1 Running 0 9s ... # deleting the netdevice policy [kni@registry.kni-qe-0 ~]$ oc delete sriovnetworknodepolicy snnp2-netdevice sriovnetworknodepolicy.sriovnetwork.openshift.io "snnp2-netdevice" deleted # device plugin is restarted and works correctly [kni@registry.kni-qe-0 ~]$ oc get pods NAME READY STATUS RESTARTS AGE sriov-device-plugin-bvlxj 1/1 Running 0 8s