What were you trying to do that didn't work?

Modify the mac address or combine different mtu size on VM with vdpa interface, then do migration, migrate will fail. Refer to qemu https://issues.redhat.com/browse/RHEL-923, it is already supported by qemu.

Please provide the package NVR for which bug is seen:

libvirt-9.5.0-7.el9_3.x86_64
qemu-kvm-8.0.0-16.el9_3.x86_64
guest kernel: 5.14.0-362.4.1.el9_3.x86_64

How reproducible:

100%

Steps to reproduce

1. Configure the vdpa device on source and target host, prepare the migration env on source and target host;
   On source host:

   # vdpa dev config show vdpa0
   vdpa0: mac 52:54:00:cb:45:00 link up link_announce false max_vq_pairs 8 mtu 1500 
   
   # virsh dumpxml rh --xpath //interface 
   <interface type="vdpa">
     <mac address="52:54:00:cb:45:00"/>
     <source dev="/dev/vhost-vdpa-0"/>
     <model type="virtio"/>
     <driver queues="8"/>
     <acpi index="1"/>
     <address type="pci" domain="0x0000" bus="0x07" slot="0x00" function="0x0"/>
   </interface>
   (vm set with vcpu number as "10")
   

1. Start vm and login the vm, check the network works well.
   Check and combine the queue size to be a new value:

   [root@localhost ~]# ethtool -l eno1
   Channel parameters for eno1:
   Pre-set maximums:
   RX:		n/a
   TX:		n/a
   Other:		n/a
   Combined:	8
   Current hardware settings:
   RX:		n/a
   TX:		n/a
   Other:		n/a
   Combined:	8
   [root@localhost ~]# ethtool -L eno1 combined 4
   [root@localhost ~]# ethtool -l eno1 
   Channel parameters for eno1:
   Pre-set maximums:
   RX:		n/a
   TX:		n/a
   Other:		n/a
   Combined:	8
   Current hardware settings:
   RX:		n/a
   TX:		n/a
   Other:		n/a
   Combined:	4
   

1. Check on host, the queue size keeps as "8". Try migration, it fails when migration process to 100%:

   # vdpa dev config show
   vdpa0: mac 52:54:00:cb:45:00 link up link_announce false max_vq_pairs 8 mtu 1500 
     negotiated_features CSUM GUEST_CSUM HOST_TSO4 HOST_TSO6 STATUS CTRL_VQ MQ CTRL_MAC_ADDR VERSION_1 ACCESS_PLATFORM
   
   # virsh migrate rh qemu+ssh://dell-per740-38.lab.eng.pek2.redhat.com/system --live --verbose --p2p
   Migration: [100.00 %]error: operation failed: job 'migration out' unexpectedly failed
   

1. Check on guest, can not ping since the migration starts, after migration failed, the connection can not recover.

   [root@localhost ~]# ping 192.168.0.1
   64 bytes from 192.168.0.1: icmp_seq=12 ttl=64 time=1.20 ms
   64 bytes from 192.168.0.1: icmp_seq=13 ttl=64 time=1.30 ms
   64 bytes from 192.168.0.1: icmp_seq=14 ttl=64 time=213 ms
   From 192.168.2.119 icmp_seq=31 Destination Host Unreachable
   From 192.168.2.119 icmp_seq=32 Destination Host Unreachable
   

2. Check qemu log shows:

   2023-09-25 17:15:06.077+0000: initiating migration
   vhost VQ 2 ring restore failed: -1: Operation not permitted (1)
   vhost VQ 3 ring restore failed: -1: Operation not permitted (1)
   vhost VQ 4 ring restore failed: -1: Operation not permitted (1)
   vhost VQ 5 ring restore failed: -1: Operation not permitted (1)
   vhost VQ 6 ring restore failed: -1: Operation not permitted (1)
   vhost VQ 7 ring restore failed: -1: Operation not permitted (1)
   vhost VQ 8 ring restore failed: -1: Operation not permitted (1)
   vhost VQ 9 ring restore failed: -1: Operation not permitted (1)
   vhost VQ 10 ring restore failed: -1: Operation not permitted (1)
   vhost VQ 11 ring restore failed: -1: Operation not permitted (1)
   vhost VQ 12 ring restore failed: -1: Operation not permitted (1)
   vhost VQ 13 ring restore failed: -1: Operation not permitted (1)
   vhost VQ 14 ring restore failed: -1: Operation not permitted (1)
   vhost VQ 15 ring restore failed: -1: Operation not permitted (1)
   2023-09-25T17:15:18.014467Z qemu-kvm: vhost_set_log_base failed: Inappropriate ioctl for device (25)
   2023-09-25T17:15:18.042058Z qemu-kvm: unable to start vhost net: 25: falling back on userspace virtio
   

Another similar scenario is to update the mac address in the VM, then do a migration. It fails as well.

1. Start vm with the xml and settings just as above, then login vm and update the mac address:

   [root@localhost ~]# ip l show
   1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
       link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
   2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
       link/ether 52:54:00:cb:45:00 brd ff:ff:ff:ff:ff:ff
       altname enp8s0
   [root@localhost ~]# ip l set dev eno1 down
   [root@localhost ~]# ip l set dev eno1 address 52:54:00:cb:77:33
   [root@localhost ~]# ip l set dev eno1 up
   [root@localhost ~]# ip a show eno1 
   2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
       link/ether 52:54:00:cb:77:33 brd ff:ff:ff:ff:ff:ff permaddr 52:54:00:cb:45:00
       altname enp8s0
       inet 192.168.2.35/22 brd 192.168.3.255 scope global dynamic noprefixroute eno1
          valid_lft 572sec preferred_lft 572sec
       inet6 fe80::2897:13a4:e692:c246/64 scope link noprefixroute 
          valid_lft forever preferred_lft forever
   (After mac updates, ping still succeed)
   [root@localhost ~]# ping 192.168.0.1
   PING 192.168.0.1 (192.168.0.1) 56(84) bytes of data.
   64 bytes from 192.168.0.1: icmp_seq=1 ttl=64 time=4.55 ms
   64 bytes from 192.168.0.1: icmp_seq=2 ttl=64 time=0.895 ms
   

   On host, check the vdpa device mac changes

   # vdpa dev config show 
   vdpa0: mac 52:54:00:cb:77:33 link up link_announce false max_vq_pairs 8 mtu 1500 
     negotiated_features CSUM GUEST_CSUM HOST_TSO4 HOST_TSO6 STATUS CTRL_VQ MQ CTRL_MAC_ADDR VERSION_1 ACCESS_PLATFORM
   

2. Try migrate

   # virsh migrate rh qemu+ssh://dell-per740-38.lab.eng.pek2.redhat.com/system --live --verbose --p2p
   Migration: [100.00 %]error: operation failed: job 'migration out' unexpectedly failed
   

3. check qemu log, the log is the same as above one.

Expected results

Migration should succeed after update the mac address or combined queue size

Actual results

Migration failed after modifying the mac or current combined queue size for vdpa interface. And after the migration failed, the VM network broke.