-
Bug
-
Resolution: Won't Do
-
Normal
-
None
-
rhel-9.0.0
-
None
-
Moderate
-
rhel-sst-virtualization-networking
-
ssg_virtualization
-
3
-
False
-
-
None
-
None
-
None
-
None
-
Known Issue
-
-
Done
-
-
x86_64
-
None
Description of problem:
During the live migration of vm with VF, I can not get ping reply from guest immediately after the VF is hot-unplugged.(the vm didn't reach downtime at this time).
Version-Release number of selected component (if applicable):
Host:
4.18.0-147.3.1.el8_1.x86_64
qemu-kvm-4.1.0-20.module+el8.1.1+5309+6d656f05.x86_64
Guest:
4.18.0-147.3.1.el8_1.x86_64
How reproducible:
10/10
Steps to Reproduce:
1.On source host,create 82599ES VF and set the mac address of the VF
ip link set enp6s0f0 vf 0 mac 22:2b:62:bb:a9:82
2.start a source guest with 82599ES VF which enables failover
/usr/libexec/qemu-kvm -name rhel811 -M q35 -enable-kvm \
-monitor stdio \
-nodefaults \
-m 4G \
-boot menu=on \
-cpu Haswell-noTSX-IBRS \
-device pcie-root-port,id=root.1,chassis=1,addr=0x2.0,multifunction=on \
-device pcie-root-port,id=root.2,chassis=2,addr=0x2.1 \
-device pcie-root-port,id=root.3,chassis=3,addr=0x2.2 \
-device pcie-root-port,id=root.4,chassis=4,addr=0x2.3 \
-device pcie-root-port,id=root.5,chassis=5,addr=0x2.4 \
-device pcie-root-port,id=root.6,chassis=6,addr=0x2.5 \
-device pcie-root-port,id=root.7,chassis=7,addr=0x2.6 \
-device pcie-root-port,id=root.8,chassis=8,addr=0x2.7 \
-smp 2,sockets=1,cores=2,threads=2,maxcpus=4 \
-qmp tcp:0:6666,server,nowait \
-blockdev node-name=back_image,driver=file,cache.direct=on,cache.no-flush=off,filename=/nfsmount/migra_test/rhel811_q35.qcow2,aio=threads \
-blockdev node-name=drive-virtio-disk0,driver=qcow2,cache.direct=on,cache.no-flush=off,file=back_image \
-device virtio-blk-pci,drive=drive-virtio-disk0,id=disk0,bus=root.1 \
-device VGA,id=video1,bus=root.2 \
-vnc :0 \
-netdev tap,id=hostnet0,vhost=on \
-device virtio-net-pci,netdev=hostnet0,id=net0,mac=22:2b:62:bb:a9:82,bus=root.3,failover=on \
-device vfio-pci,host=0000:06:10.0,id=hostdev0,bus=root.4,failover_pair_id=net0 \
3.check the network info in source guest
- ifconfig
enp3s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.73.33.236 netmask 255.255.254.0 broadcast 10.73.33.255
ether 22:2b:62:bb:a9:82 txqueuelen 1000 (Ethernet)
RX packets 28683 bytes 1961744 (1.8 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 93 bytes 13770 (13.4 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
enp3s0nsby: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether 22:2b:62:bb:a9:82 txqueuelen 1000 (Ethernet)
RX packets 28345 bytes 1924974 (1.8 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
enp4s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether 22:2b:62:bb:a9:82 txqueuelen 1000 (Ethernet)
RX packets 339 bytes 36836 (35.9 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 95 bytes 14406 (14.0 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
- ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether 22:2b:62:bb:a9:82 brd ff:ff:ff:ff:ff:ff
3: enp3s0nsby: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master enp3s0 state UP mode DEFAULT group default qlen 1000
link/ether 22:2b:62:bb:a9:82 brd ff:ff:ff:ff:ff:ff
4: enp4s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master enp3s0 state UP mode DEFAULT group default qlen 1000
link/ether 22:2b:62:bb:a9:82 brd ff:ff:ff:ff:ff:ff
4.On target host,create NetXtreme BCM57810 VF and set the mac address of the VF
ip link set enp131s0f0 vf 0 mac 22:2b:62:bb:a9:82
5.start a target guest in listening mode in order to wait for migrating from source guest
...
-incoming tcp:0:5800 \
6.keep pinging the vm during the migration
- ping 10.73.33.236
7.Migrate guest from source host to target host.
(qemu) migrate -d tcp:10.73.73.73:5800
migrate guest successfully.
8.check ping output
- ping 10.73.33.236
64 bytes from 10.73.33.236: icmp_seq=59 ttl=61 time=3.07 ms
64 bytes from 10.73.33.236: icmp_seq=60 ttl=61 time=4.35 ms
64 bytes from 10.73.33.236: icmp_seq=61 ttl=61 time=2.10 ms
64 bytes from 10.73.33.236: icmp_seq=62 ttl=61 time=4.53 ms[1]
64 bytes from 10.73.33.236: icmp_seq=88 ttl=61 time=7.39 ms[2]
64 bytes from 10.73.33.236: icmp_seq=89 ttl=61 time=4.35 ms
64 bytes from 10.73.33.236: icmp_seq=90 ttl=61 time=5.82 ms
64 bytes from 10.73.33.236: icmp_seq=91 ttl=61 time=4.39 ms
[1]
when "virtio_net virtio1 enp3s0: failover primary slave:enp4s0 unregistered" is outputed in source guest vm dmesg,ping will not work until the migration is completed.
[2]
when migration is completed,ping works again.
Actual results:
when "virtio_net virtio1 enp3s0: failover primary slave:enp4s0 unregistered" is outputed in source guest vm dmesg,ping will not work until the migration is completed.
Expected results:
ping should always work during migration, because hypervisor will fail over to the virtio netdev datapath when the VF is unplugged.
Additional info:
(1)
- lspci | grep -i 82599
06:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
06:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
(2)
This problem can be reproduced with NetXtreme II BCM57810
(3)
This problem can be reproduced in RHEL82-AV
The test env info is as following:
host:
qemu-kvm-4.2.0-4.module+el8.2.0+5220+e82621dc.x86_64
4.18.0-167.el8.x86_64
guest:
4.18.0-167.el8.x86_64
[RHEL-7336] ping can not always work during live migration of vm with failover VF
pm-rhel
added a comment - Issue migration from Bugzilla to Jira is in process at this time. This will be the last message in Jira copied from the Bugzilla bug.
pm-rhel
added a comment - Issue migration from Bugzilla to Jira is in process at this time. This will be the last message in Jira copied from the Bugzilla bug. pm-rhel
added a comment - After evaluating this issue, there are no plans to address it further or fix it in an upcoming release. Therefore, it is being closed. If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened.
pm-rhel
added a comment - After evaluating this issue, there are no plans to address it further or fix it in an upcoming release. Therefore, it is being closed. If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened. pm-rhel
added a comment - After evaluating this issue, there are no plans to address it further or fix it in an upcoming release. Therefore, it is being closed. If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened.
pm-rhel
added a comment - After evaluating this issue, there are no plans to address it further or fix it in an upcoming release. Therefore, it is being closed. If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened. pm-rhel
added a comment - After evaluating this issue, there are no plans to address it further or fix it in an upcoming release. Therefore, it is being closed. If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened.
pm-rhel
added a comment - After evaluating this issue, there are no plans to address it further or fix it in an upcoming release. Therefore, it is being closed. If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened. 
Juan Quintela (Inactive)
added a comment - Laurent Vivier is the maintainer of VF, so I am letting this to him.
I still think that the Network Configuration is wrong, but I don't have hardware to check anymore.
Could you post the configuration (xml and qemu command line when running) of the guest?
Thanks, Juan.
Juan Quintela (Inactive)
added a comment - Laurent Vivier is the maintainer of VF, so I am letting this to him.
I still think that the Network Configuration is wrong, but I don't have hardware to check anymore.
Could you post the configuration (xml and qemu command line when running) of the guest?
Thanks, Juan. (In reply to Juan Quintela from comment #37)
> Can you show your network configuration:
>
> ifconfig -a inside the guest
Finally I can reproduce the issue with live migration.
[root@localhost ~]# ping 192.168.200.79
PING 192.168.200.79 (192.168.200.79) 56(84) bytes of data.
64 bytes from 192.168.200.79: icmp_seq=1 ttl=64 time=0.105 ms
64 bytes from 192.168.200.79: icmp_seq=2 ttl=64 time=0.112 ms
64 bytes from 192.168.200.79: icmp_seq=3 ttl=64 time=0.091 ms
64 bytes from 192.168.200.79: icmp_seq=4 ttl=64 time=0.092 ms
[ 396.306453] virtio_net virtio2 enp4s0: failover primary slave:enp5s0 unregistered
From 192.168.200.132 icmp_seq=150 Destination Host Unreachable
From 192.168.200.132 icmp_seq=151 Destination Host Unreachable
From 192.168.200.132 icmp_seq=152 Destination Host Unreachable
Here are ifconfig -a in the guest:
[root@localhost ~]# ifconfig -a
enp4s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.200.132 netmask 255.255.255.0 broadcast 192.168.200.255
inet6 fe80::7629:599b:a503:e9df prefixlen 64 scopeid 0x20<link>
inet6 2001::ca9a:3558:8328:e3e0 prefixlen 64 scopeid 0x0<global>
ether 52:54:00:aa:1c:ef txqueuelen 1000 (Ethernet)
RX packets 1162 bytes 115157 (112.4 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 535 bytes 45966 (44.8 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
enp4s0nsby: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether 52:54:00:aa:1c:ef txqueuelen 1000 (Ethernet)
RX packets 1239 bytes 126915 (123.9 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 236 bytes 26624 (26.0 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
enp5s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.200.132 netmask 255.255.255.0 broadcast 192.168.200.255
inet6 fe80::6564:75b3:1b28:8516 prefixlen 64 scopeid 0x20<link>
ether 52:54:00:aa:1c:ef txqueuelen 1000 (Ethernet)
RX packets 210 bytes 23262 (22.7 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 195 bytes 14986 (14.6 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 186 bytes 20648 (20.1 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 186 bytes 20648 (20.1 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
>
> and ip link
>
Here are ip link on host
[root@dell-per440-25 ~]# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master switch state UP mode DEFAULT group default qlen 1000
link/ether f4:ee:08:0d:6e:bf brd ff:ff:ff:ff:ff:ff
altname enp4s0f0
3: eno2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
link/ether f4:ee:08:0d:6e:c0 brd ff:ff:ff:ff:ff:ff
altname enp4s0f1
4: enp59s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether 90:e2:ba:05:63:5e brd ff:ff:ff:ff:ff:ff
5: enp59s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br0 state UP mode DEFAULT group default qlen 1000
link/ether 90:e2:ba:05:63:5f brd ff:ff:ff:ff:ff:ff
vf 0 link/ether e2:94:f9:9e:1e:f8 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
6: switch: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether f4:ee:08:0d:6e:bf brd ff:ff:ff:ff:ff:ff
7: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether 90:e2:ba:05:63:5f brd ff:ff:ff:ff:ff:ff
8: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
link/ether 52:54:00:4f:51:aa brd ff:ff:ff:ff:ff:ff
49: enp59s0f1v0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
link/ether e2:94:f9:9e:1e:f8 brd ff:ff:ff:ff:ff:ff
> on the host. On #comment 21 I addressed that configuration shown on comment
> 18 is wrong. It can't be that you have both vfio and virtio devices with ip
> address. Only the "failover(bonding)" device should have configured one IP.
>
In our test, it seems we always have both vfio and virtio devices with the same ip address.
> Later, Juan.
I tried the bug on following network cards and machines with rhel9, but I didn't reproduce the bug.
[root@dell-per440-25 home]# lspci -s 0000:3b:00.1
3b:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
[root@dell-per730-28 tests]# lspci -s 0000:06:00.1
06:00.1 Ethernet controller: Intel Corporation Ethernet Controller XL710 for 40GbE QSFP+ (rev 02)
[root@dell-per440-22 ~]# lspci -s 0000:3b:00.1
3b:00.1 Ethernet controller: Intel Corporation Ethernet Controller XL710 for 40GbE QSFP+ (rev 02)
[root@dell-per440-25 home]# rpm -q qemu-kvm
qemu-kvm-6.2.0-8.el9.x86_64
[root@dell-per440-25 home]# uname -r
5.14.0-57.kpq0.el9.x86_64
[root@dell-per440-25 home]# rpm -q libvirt
libvirt-8.0.0-3.el9.x86_64
hotunplug vf and keep ping vm from before hotunplug
- virsh qemu-monitor-command rhel90 '{"execute":"device_del","arguments":{"id":"hostdev0"}}'
[root@vm-74-194 ~]# ifconfig
enp1s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.73.74.194 netmask 255.255.252.0 broadcast 10.73.75.255
inet6 2620:52:0:4948:f68e:38ff:fec3:9090 prefixlen 64 scopeid 0x0<global>
inet6 fe80::f68e:38ff:fec3:9090 prefixlen 64 scopeid 0x20<link>
ether f4:8e:38:c3:90:90 txqueuelen 1000 (Ethernet)
RX packets 174477 bytes 15284807 (14.5 MiB)
RX errors 0 dropped 4922 overruns 0 frame 0
TX packets 1299 bytes 109928 (107.3 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
enp4s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.200.132 netmask 255.255.255.0 broadcast 192.168.200.255
inet6 fe80::7629:599b:a503:e9df prefixlen 64 scopeid 0x20<link>
inet6 2001::ca9a:3558:8328:e3e0 prefixlen 64 scopeid 0x0<global>
ether 52:54:00:aa:1c:ef txqueuelen 1000 (Ethernet)
RX packets 22650 bytes 2265604 (2.1 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 3374 bytes 328154 (320.4 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
enp4s0nsby: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.200.132 netmask 255.255.255.0 broadcast 192.168.200.255
inet6 fe80::17be:e17c:345e:a239 prefixlen 64 scopeid 0x20<link>
ether 52:54:00:aa:1c:ef txqueuelen 1000 (Ethernet)
RX packets 22754 bytes 2276133 (2.1 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 3325 bytes 325312 (317.6 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
ping works well before hotunplug and after hotunplug.
- ping 192.168.200.132
PING 192.168.200.132 (192.168.200.132) 56(84) bytes of data.
64 bytes from 192.168.200.132: icmp_seq=1 ttl=64 time=0.229 ms
64 bytes from 192.168.200.132: icmp_seq=2 ttl=64 time=0.215 ms
64 bytes from 192.168.200.132: icmp_seq=3 ttl=64 time=0.136 ms
64 bytes from 192.168.200.132: icmp_seq=4 ttl=64 time=0.205 ms
64 bytes from 192.168.200.132: icmp_seq=5 ttl=64 time=0.203 ms
64 bytes from 192.168.200.132: icmp_seq=6 ttl=64 time=0.203 ms
64 bytes from 192.168.200.132: icmp_seq=7 ttl=64 time=0.199 ms
64 bytes from 192.168.200.132: icmp_seq=8 ttl=64 time=0.202 ms
64 bytes from 192.168.200.132: icmp_seq=9 ttl=64 time=0.204 ms
Juan Quintela (Inactive)
added a comment - Can you show your network configuration:
ifconfig -a inside the guest
and ip link
on the host. On #comment 21 I addressed that configuration shown on comment 18 is wrong. It can't be that you have both vfio and virtio devices with ip address. Only the "failover(bonding)" device should have configured one IP.
Later, Juan.
Juan Quintela (Inactive)
added a comment - Can you show your network configuration:
ifconfig -a inside the guest
and ip link
on the host. On #comment 21 I addressed that configuration shown on comment 18 is wrong. It can't be that you have both vfio and virtio devices with ip address. Only the "failover(bonding)" device should have configured one IP.
Later, Juan. pm-rhel
added a comment - After evaluating this issue, there are no plans to address it further or fix it in an upcoming release. Therefore, it is being closed. If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened.
pm-rhel
added a comment - After evaluating this issue, there are no plans to address it further or fix it in an upcoming release. Therefore, it is being closed. If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened. 
This is a legit bug but it will take a lot of work to fix upstream and we simply don't have the capacity to address it. In addition, this is not a supported scenario that would be used by a layered product. Feel free to reopen if you would like to revisit this decision.