Description of problem:
Ping fails between 2 CNV guest VMs secondary interfaces, which are backed by an SR-IOV VF and VLAN tag.

Version-Release number of selected component (if applicable):
CNV-4.13.0
ose-sriov-cni@sha256:1e71da4022477787ff8d3f2ff53fc7e86c4f6827734815291b136ef219a0c7a7 (sriov-cni-container-v4.13.0-202304211716.p0.g08b4f6a.assembly.stream, https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=2475985)
ose-sriov-network-operator@sha256:f9433618ed10282ef39e0d8267fed539be0688d555c38826e3390bfdb48a27ba (sriov-network-operator-container-v4.13.0-202304211716.p0.g8471529.assembly.stream, https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=2475986)

How reproducible:
100%

Steps to Reproduce:
1. On a bare-metal cluster with VLAN and SR-IOV supported - apply an SriovNetworkNodePolicy like the sriov-network-node-policy.yaml attached.

2. Apply an SriovNetwork like the attached sriov-network.yaml.
Make sure to select a VLAN tag which is enabled on your cluster.

3. Make sure a matching NetworkAttachmentDefinition was created:
$ oc get net-attach-def -n sriov-test-sriov sriov-test-network-vlan -o yaml
apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
annotations:
k8s.v1.cni.cncf.io/resourceName: openshift.io/sriov_net_with_vlan
creationTimestamp: "2023-05-14T09:53:45Z"
generation: 1
name: sriov-test-network-vlan
namespace: sriov-test-sriov
resourceVersion: "19078001"
uid: d7498ca2-566a-4b2e-ab23-5c312ef7f1ae
spec:
config: '{ "cniVersion":"0.3.1", "name":"sriov-test-network-vlan","type":"sriov","vlan":1000,"vlanQoS":0,"ipam":{}
}'

4. Create 2 VMs like the attached VM manifests (vm3.yaml and vm4.yaml), with secondary
interfaces which are based on the NetworkAttachmentDefinition, and IP addresses on the same subnet.

5. Login to one of the VMs, and try pinging the IP address of the secondary interface of the other VM:
$ virtctl console sriov-vm3-1684058028-2034686
Successfully connected to sriov-vm3-1684058028-2034686 console. The escape sequence is ^]

[fedora@sriov-vm3-1684058028-2034686 ~]$ ping 10.200.3.2
PING 10.200.3.2 (10.200.3.2) 56(84) bytes of data.
From 10.200.3.1 icmp_seq=1 Destination Host Unreachable
From 10.200.3.1 icmp_seq=2 Destination Host Unreachable
From 10.200.3.1 icmp_seq=3 Destination Host Unreachable

— 10.200.3.2 ping statistics —
5 packets transmitted, 0 received, +3 errors, 100% packet loss, time 4099ms
pipe 4

Actual results:
Ping fails.

Expected results:
Ping should succeed and verify connectivity.

Additional info:
1. Same setup without VLAN tag (i.e. the secondary interfaces on the VMs are backed by SR-IOV connections without VLAN) works successfully.
2. Same setup without SR-IOV (i.e. the secondary interfaces on the VM reside on VLAN tagged network) works successfully, so we know VLAN is supported on the cluster(s).
3. The same issue was found on 2 different bare-metal clusters.