Description of problem:  

OCP 4.22-ec3 VF-to-VF communication on the same PF fails when the VFs are configured with MTU 500 on an Intel E810 NIC. The same topology works correctly when the VFs use MTU 9000.

Two pods scheduled on the same node and attached to different VFs on the same PF are unable to communicate when the VF MTU is set to 500. Packet captures show that ARP requests leave the transmitting VF but are never received by the peer VF.

Potential datapath or VF switching issue in the ice/iavf driver path when MTU values are less than 871 configured on VFs.

Version-Release number of selected component (if applicable):

5.14.0-686.el9.x86_64 is the version we tested on, where the VFs with low MTU can't talk each other.
https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=3955992

How reproducible: Easily

Steps to Reproduce:
1. Create sriovneetworknodepolicy with mtu 500
2. Create two pods with secondary interfaces using the VFs from a single PF.
3. Send pings between the pods. Ping fails
Actual results:

pod with VF from sriovnetworknodepolicy with mtu 500 should is unable to send packets over the interface.

Expected results:

pod with VF from sriovnetworknodepolicy with mtu 500 should be able to send packets over the interface.

Additional info:

apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetworkNodePolicy
metadata:
  name: ipv4-policy-pf1-mtu500
  namespace: openshift-sriov-network-operator
spec:
  bridge: {}
  deviceType: netdevice
  isRdma: false
  mtu: 500
  nicSelector:
    pfNames:
    - ens3f0#0-4
  nodeSelector:
    node-role.kubernetes.io/workercnf: ""
  numVfs: 10
  priority: 1
  resourceName: sriovpf1mtu500