Description of problem:
Attach direct type interface with VF and vlan setting failed for the second time

Version-Release number of selected component (if applicable):
libvirt-8.10.0-2.el9.x86_64
qemu-kvm-7.2.0-2.el9.x86_64
kernel-5.14.0-226.el9.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Initialize the vfs, then check the mac address.

1. echo 0 >/sys/class/net/ens1f0/device/sriov_numvfs
2. echo 1 >/sys/class/net/ens1f0/device/sriov_numvfs

2. Prepare the interface xml.

1. cat interface.xml
   <interface type='direct'>
   <mac address='7a:3d:d6:18:19:76'/>
   <vlan>
   <tag id='50'/>
   </vlan>
   <source dev='enp59s2f16v0' mode='passthrough'/>
   <address type='pci' domain='0x0000' bus='0x09' slot='0x00' function='0x0'/>
   </interface>

3. Attach the interface to the guest.

1. virsh attach-device lmn interface.xml
   Device attached successfully
2. virsh dumpxml lmn | xmllint --xpath //interface -
   <interface type="direct">
   <mac address="7a:3d:d6:18:19:76"/>
   <source dev="enp59s2f16v0" mode="passthrough"/>
   <vlan>
   <tag id="50"/>
   </vlan>
   <target dev="macvtap12"/>
   <model type="rtl8139"/>
   <alias name="net0"/>
   <address type="pci" domain="0x0000" bus="0x09" slot="0x00" function="0x0"/>
   </interface>

4. Detach the interface.

1. virsh detach-device lmn interface.xml
   Device detached successfully
2. virsh dumpxml lmn | xmllint --xpath //interface -
   XPath set is empty

5. Attach the interface again.

1. virsh attach-device lmn interface.xml
   error: Failed to attach device from interface.xml
   error: Cannot set interface flags on 'enp59s2f16v0': No such device

1. ip l show enp59s2f16v0
   36: enp59s2f16v0: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
   link/ether e6:9b:12:18:87:ee brd ff:ff:ff:ff:ff:ff

Actual results:
Attach failed as above step 5.

Expected results:
Attach successfully.

Additional info:
1) Can attach the interface without vlan setting successfully in step 5.
2) libvirtd.log:
2023-01-12 03:59:27.023+0000: 7347: error : virNetDevSetIFFlag:621 : Cannot set interface flags on 'enp59s2f16v0': No such device