-
Bug
-
Resolution: Done
-
Normal
-
None
-
rhel-9.4
-
libvirt-10.5.0-1.el10
-
None
-
Moderate
-
sst_virtualization
-
ssg_virtualization
-
5
-
False
-
-
None
-
Red Hat Enterprise Linux
-
None
-
None
-
Automated
-
-
x86_64
-
None
What were you trying to do that didn't work?
The mlx vfio migration fails after enabling switchover-ack capabilities
Please provide the package NVR for which bug is seen:
qemu-kvm-8.1.0-3.el9.x86_64
5.14.0-378.el9.x86_64
libvirt-9.5.0-7.el9_3.x86_64
How reproducible:
100%
Steps to reproduce
1. create a MT2910 VF and setup the VF for migration
2. start a Q35 + SEABIOS VM with a mlx5_vfio_pci VF
3. enable the switchover-ack capabilities
/bin/virsh qemu-monitor-command --hmp rhel94 "migrate_set_capability return-path on"
/bin/virsh qemu-monitor-command --hmp rhel94 "migrate_set_capability switchover-ack on"
/bin/virsh qemu-monitor-command --hmp rhel94 "info migrate_capabilities" xbzrle: off rdma-pin-all: off auto-converge: off zero-blocks: off compress: off events: on postcopy-ram: off x-colo: off release-ram: off return-path: on pause-before-switchover: off multifd: off dirty-bitmaps: off postcopy-blocktime: off late-block-activate: off x-ignore-shared: off validate-uuid: off background-snapshot: off zero-copy-send: off postcopy-preempt: off switchover-ack: on dirty-limit: off
4. migrate the VM
# /bin/virsh migrate --live --domain rhel94 --desturi qemu+ssh://10.73.212.96/system error: internal error: QEMU unexpectedly closed the monitor (vm='rhel94'): 2023-10-30T11:06:38.935143Z qemu-kvm: 0000:22:00.1: Received INIT_DATA_SENT but switchover ack is not used 2023-10-30T11:06:38.935199Z qemu-kvm: error while loading state section id 84(0000:00:02.3:00.0/vfio) 2023-10-30T11:06:38.935491Z qemu-kvm: load of migration failed: Invalid argument
5. check the qemu-kvm log
# cat /var/log/libvirt/qemu/rhel94.log 2023-10-30 10:15:36.710+0000: initiating migration 2023-10-30T10:15:42.915950Z qemu-kvm: failed to save SaveStateEntry with id(name): 2(ram): -5 2023-10-30T10:15:42.933625Z qemu-kvm: Unable to write to socket: Bad file descriptor
Expected results
The mlx vfio migration finishes well
Actual results
The mlx vfio migration failed
Additional info
(1) The mellanox CX-7 device I used:
# flint -d 0000:22:00.0 query full
Image type: FS4
FW Version: 28.38.1002
FW Release Date: 3.8.2023
Part Number: MCX75310AAS-HEA_Ax
Description: NVIDIA ConnectX-7 HHHL Adapter Card; 200GbE / NDR200 IB (default mode); Single-port OSFP; PCIe 5.0 x16; Crypto Disabled; Secure Boot Enabled;
Product Version: 28.38.1002
Rom Info: type=UEFI version=14.31.20 cpu=AMD64,AARCH64
type=PXE version=3.7.201 cpu=AMD64
Description: UID GuidsNumber
Base GUID: 946dae03001db182 2
Base MAC: 946dae1db182 2
Image VSD: N/A
Device VSD: N/A
PSID: MT_0000000844
Security Attributes: secure-fw
Default Update Method: fw_ctrl
Life cycle: GA SECURED
Secure Boot Capable: Enabled
EFUSE Security Ver: 0
Image Security Ver: 0
Security Ver Program: Manually ; Disabled
Encryption: Enabled
(2) How to create a MT2910 VF and setup the VF for migration
1.1 load the mlx5_vfio_pci module
# modprobe mlx5_vfio_pci
1.2 create VF
# sudo sh -c "echo 0 > /sys/bus/pci/devices/0000:b1:00.0/sriov_numvfs"
# sudo sh -c "echo 1 > /sys/bus/pci/devices/0000:b1:00.0/sriov_numvfs"
1.3 set VF mac
# sudo sh -c "ip link set ens2f0np0 vf 0 mac 52:54:00:01:01:01"
1.4 unbind created VF from driver
# sudo sh -c "echo 0000:b1:00.2 > /sys/bus/pci/drivers/mlx5_core/unbind"
1.5 set switchdev mode on PF
# sudo sh -c "devlink dev eswitch set pci/0000:b1:00.0 mode switchdev"
# sudo sh -c "devlink dev eswitch show pci/0000:b1:00.0"
pci/0000:b1:00.0: mode switchdev inline-mode none encap-mode basic
1.6 enable VF's migration feature
# sudo sh -c "devlink port function set pci/0000:b1:00.0/1 migratable enable"
# sudo sh -c "devlink port show pci/0000:b1:00.0/1"
…
function:
hw_addr 52:54:00:01:01:01 roce enable migratable enable
1.7 bind VF to mlx5_vfio_pci driver
# sudo sh -c "echo '15b3 101e' > /sys/bus/pci/drivers/mlx5_vfio_pci/new_id"
# sudo sh -c "echo '15b3 101e' > /sys/bus/pci/drivers/mlx5_vfio_pci/remove_id"
# readlink -f /sys/bus/pci/devices/0000\:b1\:00.2/driver
/sys/bus/pci/drivers/mlx5_vfio_pci
(3) The mlx vfio migration finishes well without enabling the switchover-ack capability
(4) libvirt will enable the return-path capability in default since libvirt-8.0 when migration.
(5) The qmp generated when running the "/bin/virsh migrate --live --domain rhel93 --desturi qemu+ssh://10.73.212.96/system"
1089.987 > {"execute":"query-block","id":"libvirt-462"}
...
1089.988 > {"execute":"query-migrate","id":"libvirt-463"}
1089.988 < {"return": {}, "id": "libvirt-463"}
1091.093 > {"execute":"query-migrate-parameters","id":"libvirt-464"}
...
1091.094 > {"execute":"migrate-set-capabilities","arguments":{"capabilities":[{"capability":"xbzrle","state":false},{"capability":"auto-converge","state":false},{"capability":"rdma-pin-all","state":false},{"capability":"postcopy-ram","state":false},{"capability":"compress","state":false},{"capability":"pause-before-switchover","state":true},{"capability":"late-block-activate","state":false},{"capability":"multifd","state":false},{"capability":"dirty-bitmaps","state":false},{"capability":"return-path","state":true}
1091.096 < {"return": {}, "id": "libvirt-465"}
1091.096 > {"execute":"migrate-set-parameters","arguments":{"tls-creds":"","tls-hostname":"","max-bandwidth":9223372036853727232},"id":"libvirt-466"}
1091.097 < {"return": {}, "id": "libvirt-466"}
1091.099 > {"execute":"getfd","arguments":{"fdname":"migrate"},"id":"libvirt-467"} (fd=21)
1091.099 < {"return": {}, "id": "libvirt-467"}
1091.099 > {"execute":"migrate","arguments":{"detach":true,"resume":false,"uri":"fd:migrate"},"id":"libvirt-468"}
1091.100 ! {"timestamp": {"seconds": 1698660936, "microseconds": 712265}, "event": "MIGRATION", "data": {"status": "setup"}}
1091.100 < {"return": {}, "id": "libvirt-468"}
1091.111 ! {"timestamp": {"seconds": 1698660936, "microseconds": 723837}, "event": "MIGRATION_PASS", "data": {"pass": 1}}
1091.144 ! {"timestamp": {"seconds": 1698660936, "microseconds": 755979}, "event": "MIGRATION", "data": {"status": "active"}}
1097.119 ! {"timestamp": {"seconds": 1698660942, "microseconds": 731124}, "event": "MIGRATION_PASS", "data": {"pass": 2}}
1097.179 ! {"timestamp": {"seconds": 1698660942, "microseconds": 791015}, "event": "MIGRATION_PASS", "data": {"pass": 3}}
1097.233 ! {"timestamp": {"seconds": 1698660942, "microseconds": 845040}, "event": "MIGRATION_PASS", "data": {"pass": 4}}
1097.235 ! {"timestamp": {"seconds": 1698660942, "microseconds": 847730}, "event": "MIGRATION_PASS", "data": {"pass": 5}}
1097.237 ! {"timestamp": {"seconds": 1698660942, "microseconds": 849697}, "event": "MIGRATION_PASS", "data": {"pass": 6}}
1097.239 ! {"timestamp": {"seconds": 1698660942, "microseconds": 851670}, "event": "MIGRATION_PASS", "data": {"pass": 7}}
1097.241 ! {"timestamp": {"seconds": 1698660942, "microseconds": 853632}, "event": "MIGRATION_PASS", "data": {"pass": 8}}
1097.243 ! {"timestamp": {"seconds": 1698660942, "microseconds": 855602}, "event": "MIGRATION_PASS", "data": {"pass": 9}}
1097.245 ! {"timestamp": {"seconds": 1698660942, "microseconds": 857572}, "event": "MIGRATION_PASS", "data": {"pass": 10}}
1097.247 ! {"timestamp": {"seconds": 1698660942, "microseconds": 859799}, "event": "MIGRATION_PASS", "data": {"pass": 11}}
1097.250 ! {"timestamp": {"seconds": 1698660942, "microseconds": 862019}, "event": "MIGRATION_PASS", "data": {"pass": 12}}
1097.252 ! {"timestamp": {"seconds": 1698660942, "microseconds": 863989}, "event": "MIGRATION_PASS", "data": {"pass": 13}}
1097.253 ! {"timestamp": {"seconds": 1698660942, "microseconds": 865954}, "event": "MIGRATION_PASS", "data": {"pass": 14}}
1097.255 ! {"timestamp": {"seconds": 1698660942, "microseconds": 867921}, "event": "MIGRATION_PASS", "data": {"pass": 15}}
1097.257 ! {"timestamp": {"seconds": 1698660942, "microseconds": 869889}, "event": "MIGRATION_PASS", "data": {"pass": 16}}
1097.259 ! {"timestamp": {"seconds": 1698660942, "microseconds": 871863}, "event": "MIGRATION_PASS", "data": {"pass": 17}}
1097.262 ! {"timestamp": {"seconds": 1698660942, "microseconds": 874086}, "event": "MIGRATION_PASS", "data": {"pass": 18}}
1097.264 ! {"timestamp": {"seconds": 1698660942, "microseconds": 876251}, "event": "MIGRATION_PASS", "data": {"pass": 19}}
1097.266 ! {"timestamp": {"seconds": 1698660942, "microseconds": 878216}, "event": "MIGRATION_PASS", "data": {"pass": 20}}
1097.268 ! {"timestamp": {"seconds": 1698660942, "microseconds": 880187}, "event": "MIGRATION_PASS", "data": {"pass": 21}}
1097.270 ! {"timestamp": {"seconds": 1698660942, "microseconds": 882165}, "event": "MIGRATION_PASS", "data": {"pass": 22}}
1097.272 ! {"timestamp": {"seconds": 1698660942, "microseconds": 884377}, "event": "MIGRATION_PASS", "data": {"pass": 23}}
1097.274 ! {"timestamp": {"seconds": 1698660942, "microseconds": 886657}, "event": "MIGRATION_PASS", "data": {"pass": 24}}
1097.276 ! {"timestamp": {"seconds": 1698660942, "microseconds": 888870}, "event": "MIGRATION_PASS", "data": {"pass": 25}}
1097.279 ! {"timestamp": {"seconds": 1698660942, "microseconds": 891860}, "event": "MIGRATION_PASS", "data": {"pass": 26}}
1097.281 ! {"timestamp": {"seconds": 1698660942, "microseconds": 893825}, "event": "MIGRATION_PASS", "data": {"pass": 27}}
1097.283 ! {"timestamp": {"seconds": 1698660942, "microseconds": 895792}, "event": "MIGRATION_PASS", "data": {"pass": 28}}
1097.285 ! {"timestamp": {"seconds": 1698660942, "microseconds": 897764}, "event": "MIGRATION_PASS", "data": {"pass": 29}}
1097.287 ! {"timestamp": {"seconds": 1698660942, "microseconds": 899733}, "event": "MIGRATION_PASS", "data": {"pass": 30}}
1097.289 ! {"timestamp": {"seconds": 1698660942, "microseconds": 901703}, "event": "MIGRATION_PASS", "data": {"pass": 31}}
1097.291 ! {"timestamp": {"seconds": 1698660942, "microseconds": 903670}, "event": "MIGRATION_PASS", "data": {"pass": 32}}
1097.293 ! {"timestamp": {"seconds": 1698660942, "microseconds": 905640}, "event": "MIGRATION_PASS", "data": {"pass": 33}}
1097.295 ! {"timestamp": {"seconds": 1698660942, "microseconds": 907604}, "event": "MIGRATION_PASS", "data": {"pass": 34}}
1097.297 ! {"timestamp": {"seconds": 1698660942, "microseconds": 909575}, "event": "MIGRATION_PASS", "data": {"pass": 35}}
1097.299 ! {"timestamp": {"seconds": 1698660942, "microseconds": 911542}, "event": "MIGRATION_PASS", "data": {"pass": 36}}
1097.301 ! {"timestamp": {"seconds": 1698660942, "microseconds": 913517}, "event": "MIGRATION_PASS", "data": {"pass": 37}}
1097.303 ! {"timestamp": {"seconds": 1698660942, "microseconds": 915812}, "event": "MIGRATION_PASS", "data": {"pass": 38}}
1097.304 ! {"timestamp": {"seconds": 1698660942, "microseconds": 916051}, "event": "MIGRATION", "data": {"status": "failed"}}
1097.304 > {"execute":"query-migrate","id":"libvirt-469"}
1097.322 < {"return": {"status": "failed", "error-desc": "Unable to write to socket: Bad file descriptor"}, "id": "libvirt-469"}
1097.322 > {"execute":"migrate-set-capabilities","arguments":{"capabilities":[{"capability":"xbzrle","state":false},{"capability":"auto-converge","state":false},{"capability":"rdma-pin-all","state":false},{"capability":"postcopy-ram","state":false},{"capability":"compress","state":false},{"capability":"pause-before-switchover","state":false},{"capability":"late-block-activate","state":false},{"capability":"multifd","state":false},{"capability":"dirty-bitmaps","state":false}......}
1097.325 < {"id": "libvirt-470", "error": {"class": "GenericError", "desc": "Capability 'switchover-ack' requires capability 'return-path'"}}
Additional info:
(1) The auto cmd to reproduce this issue :
# python3 /home/private_autocase/vfio/vfio_sriov_test.py --feature=vf --domain=$VM --device_name=MT2910-01 --machine_type=q35 --test_list="set_switchover_ack_in_mlx_device_migration"
- is cloned by
-
-
- Closed
-
