Loading...

XML

Word

Printable

Type: Bug
Resolution: Done
Priority: Normal
Fix Version/s: None
Affects Version/s: rhel-9.4
Component/s: libvirt / Live Migration
Labels:

Fixed in Build:
libvirt-10.5.0-1.el10
Regression:
None
Severity:
Moderate

Pool Team:

rhel-sst-virtualization
Sub-System Group:

ssg_virtualization

Story Points:
5
Blocked:
False
Blocked Reason:

Hide

None

Show
None
Product Documentation Required:
None
Products:

Red Hat Enterprise Linux
Sprint:
None

Preliminary Testing:
None
Test Link:
https://polarion.engineering.redhat.com/polarion/redirect/project/RHELVIRT/workitem?id=VIRT-300056
Test Coverage:

Automated

Experience:
Architecture:

x86_64

SFDC Cases Counter:
SFDC Cases Open:
SFDC Cases Links:

Planning:
None

What were you trying to do that didn't work?

The mlx vfio migration fails after enabling switchover-ack capabilities

Please provide the package NVR for which bug is seen:

qemu-kvm-8.1.0-3.el9.x86_64
5.14.0-378.el9.x86_64
libvirt-9.5.0-7.el9_3.x86_64

How reproducible:

100%

Steps to reproduce

1. create a MT2910 VF and setup the VF for migration

2. start a Q35 + SEABIOS VM with a mlx5_vfio_pci VF

3. enable the switchover-ack capabilities

/bin/virsh qemu-monitor-command --hmp rhel94 "migrate_set_capability return-path on"

/bin/virsh qemu-monitor-command --hmp rhel94 "migrate_set_capability switchover-ack on"

/bin/virsh qemu-monitor-command --hmp rhel94 "info migrate_capabilities"
xbzrle: off
rdma-pin-all: off
auto-converge: off
zero-blocks: off
compress: off
events: on
postcopy-ram: off
x-colo: off
release-ram: off
return-path: on
pause-before-switchover: off
multifd: off
dirty-bitmaps: off
postcopy-blocktime: off
late-block-activate: off
x-ignore-shared: off
validate-uuid: off
background-snapshot: off
zero-copy-send: off
postcopy-preempt: off
switchover-ack: on
dirty-limit: off

4. migrate the VM

# /bin/virsh migrate --live --domain rhel94 --desturi qemu+ssh://10.73.212.96/system
error: internal error: QEMU unexpectedly closed the monitor (vm='rhel94'): 2023-10-30T11:06:38.935143Z qemu-kvm: 0000:22:00.1: Received INIT_DATA_SENT but switchover ack is not used
2023-10-30T11:06:38.935199Z qemu-kvm: error while loading state section id 84(0000:00:02.3:00.0/vfio)
2023-10-30T11:06:38.935491Z qemu-kvm: load of migration failed: Invalid argument

5. check the qemu-kvm log

# cat /var/log/libvirt/qemu/rhel94.log
2023-10-30 10:15:36.710+0000: initiating migration
2023-10-30T10:15:42.915950Z qemu-kvm: failed to save SaveStateEntry with id(name): 2(ram): -5
2023-10-30T10:15:42.933625Z qemu-kvm: Unable to write to socket: Bad file descriptor

Expected results

The mlx vfio migration finishes well

Actual results

The mlx vfio migration failed

Additional info

(1) The mellanox CX-7 device I used:

# flint -d 0000:22:00.0 query full
Image type:            FS4
FW Version:            28.38.1002
FW Release Date:       3.8.2023
Part Number:           MCX75310AAS-HEA_Ax
Description:           NVIDIA ConnectX-7 HHHL Adapter Card; 200GbE / NDR200 IB (default mode); Single-port OSFP; PCIe 5.0 x16; Crypto Disabled; Secure Boot Enabled;
Product Version:       28.38.1002
Rom Info:              type=UEFI version=14.31.20 cpu=AMD64,AARCH64
                       type=PXE version=3.7.201 cpu=AMD64
Description:           UID                GuidsNumber
Base GUID:             946dae03001db182        2
Base MAC:              946dae1db182            2
Image VSD:             N/A
Device VSD:            N/A
PSID:                  MT_0000000844
Security Attributes:   secure-fw
Default Update Method: fw_ctrl
Life cycle:            GA SECURED
Secure Boot Capable:   Enabled
EFUSE Security Ver:    0
Image Security Ver:    0
Security Ver Program:  Manually ; Disabled
Encryption:            Enabled

(2) How to create a MT2910 VF and setup the VF for migration

1.1 load the mlx5_vfio_pci module

# modprobe mlx5_vfio_pci

1.2 create VF

# sudo sh -c "echo 0 > /sys/bus/pci/devices/0000:b1:00.0/sriov_numvfs"
# sudo sh -c "echo 1 > /sys/bus/pci/devices/0000:b1:00.0/sriov_numvfs"

1.3 set VF mac

# sudo sh -c  "ip link set ens2f0np0 vf 0 mac 52:54:00:01:01:01"

1.4 unbind created VF from driver

# sudo sh -c  "echo 0000:b1:00.2 > /sys/bus/pci/drivers/mlx5_core/unbind"

1.5 set switchdev mode on PF

# sudo sh -c "devlink dev eswitch set pci/0000:b1:00.0 mode switchdev"
# sudo sh -c "devlink dev eswitch show pci/0000:b1:00.0"
    pci/0000:b1:00.0: mode switchdev inline-mode none encap-mode basic

1.6 enable VF's migration feature

# sudo sh -c "devlink port function set pci/0000:b1:00.0/1 migratable enable"
# sudo sh -c  "devlink port show pci/0000:b1:00.0/1"
	  …
  function:
    hw_addr 52:54:00:01:01:01 roce enable migratable enable

1.7 bind VF to mlx5_vfio_pci driver

# sudo sh -c "echo '15b3 101e' > /sys/bus/pci/drivers/mlx5_vfio_pci/new_id"
# sudo sh -c "echo '15b3 101e' > /sys/bus/pci/drivers/mlx5_vfio_pci/remove_id"
# readlink -f /sys/bus/pci/devices/0000\:b1\:00.2/driver
  /sys/bus/pci/drivers/mlx5_vfio_pci

(3) The mlx vfio migration finishes well without enabling the switchover-ack capability

(4) libvirt will enable the return-path capability in default since libvirt-8.0 when migration.

(5) The qmp generated when running the "/bin/virsh migrate --live --domain rhel93 --desturi qemu+ssh://10.73.212.96/system"

1089.987 > {"execute":"query-block","id":"libvirt-462"}
...
1089.988 > {"execute":"query-migrate","id":"libvirt-463"}
1089.988 <  {"return": {}, "id": "libvirt-463"}
1091.093 > {"execute":"query-migrate-parameters","id":"libvirt-464"}
...
1091.094 > {"execute":"migrate-set-capabilities","arguments":{"capabilities":[{"capability":"xbzrle","state":false},{"capability":"auto-converge","state":false},{"capability":"rdma-pin-all","state":false},{"capability":"postcopy-ram","state":false},{"capability":"compress","state":false},{"capability":"pause-before-switchover","state":true},{"capability":"late-block-activate","state":false},{"capability":"multifd","state":false},{"capability":"dirty-bitmaps","state":false},{"capability":"return-path","state":true}
1091.096 <  {"return": {}, "id": "libvirt-465"}
1091.096 > {"execute":"migrate-set-parameters","arguments":{"tls-creds":"","tls-hostname":"","max-bandwidth":9223372036853727232},"id":"libvirt-466"}
1091.097 <  {"return": {}, "id": "libvirt-466"}
1091.099 > {"execute":"getfd","arguments":{"fdname":"migrate"},"id":"libvirt-467"} (fd=21)
1091.099 <  {"return": {}, "id": "libvirt-467"}
1091.099 > {"execute":"migrate","arguments":{"detach":true,"resume":false,"uri":"fd:migrate"},"id":"libvirt-468"}
1091.100 !  {"timestamp": {"seconds": 1698660936, "microseconds": 712265}, "event": "MIGRATION", "data": {"status": "setup"}}
1091.100 <  {"return": {}, "id": "libvirt-468"}
1091.111 !  {"timestamp": {"seconds": 1698660936, "microseconds": 723837}, "event": "MIGRATION_PASS", "data": {"pass": 1}}
1091.144 !  {"timestamp": {"seconds": 1698660936, "microseconds": 755979}, "event": "MIGRATION", "data": {"status": "active"}}
1097.119 !  {"timestamp": {"seconds": 1698660942, "microseconds": 731124}, "event": "MIGRATION_PASS", "data": {"pass": 2}}
1097.179 !  {"timestamp": {"seconds": 1698660942, "microseconds": 791015}, "event": "MIGRATION_PASS", "data": {"pass": 3}}
1097.233 !  {"timestamp": {"seconds": 1698660942, "microseconds": 845040}, "event": "MIGRATION_PASS", "data": {"pass": 4}}
1097.235 !  {"timestamp": {"seconds": 1698660942, "microseconds": 847730}, "event": "MIGRATION_PASS", "data": {"pass": 5}}
1097.237 !  {"timestamp": {"seconds": 1698660942, "microseconds": 849697}, "event": "MIGRATION_PASS", "data": {"pass": 6}}
1097.239 !  {"timestamp": {"seconds": 1698660942, "microseconds": 851670}, "event": "MIGRATION_PASS", "data": {"pass": 7}}
1097.241 !  {"timestamp": {"seconds": 1698660942, "microseconds": 853632}, "event": "MIGRATION_PASS", "data": {"pass": 8}}
1097.243 !  {"timestamp": {"seconds": 1698660942, "microseconds": 855602}, "event": "MIGRATION_PASS", "data": {"pass": 9}}
1097.245 !  {"timestamp": {"seconds": 1698660942, "microseconds": 857572}, "event": "MIGRATION_PASS", "data": {"pass": 10}}
1097.247 !  {"timestamp": {"seconds": 1698660942, "microseconds": 859799}, "event": "MIGRATION_PASS", "data": {"pass": 11}}
1097.250 !  {"timestamp": {"seconds": 1698660942, "microseconds": 862019}, "event": "MIGRATION_PASS", "data": {"pass": 12}}
1097.252 !  {"timestamp": {"seconds": 1698660942, "microseconds": 863989}, "event": "MIGRATION_PASS", "data": {"pass": 13}}
1097.253 !  {"timestamp": {"seconds": 1698660942, "microseconds": 865954}, "event": "MIGRATION_PASS", "data": {"pass": 14}}
1097.255 !  {"timestamp": {"seconds": 1698660942, "microseconds": 867921}, "event": "MIGRATION_PASS", "data": {"pass": 15}}
1097.257 !  {"timestamp": {"seconds": 1698660942, "microseconds": 869889}, "event": "MIGRATION_PASS", "data": {"pass": 16}}
1097.259 !  {"timestamp": {"seconds": 1698660942, "microseconds": 871863}, "event": "MIGRATION_PASS", "data": {"pass": 17}}
1097.262 !  {"timestamp": {"seconds": 1698660942, "microseconds": 874086}, "event": "MIGRATION_PASS", "data": {"pass": 18}}
1097.264 !  {"timestamp": {"seconds": 1698660942, "microseconds": 876251}, "event": "MIGRATION_PASS", "data": {"pass": 19}}
1097.266 !  {"timestamp": {"seconds": 1698660942, "microseconds": 878216}, "event": "MIGRATION_PASS", "data": {"pass": 20}}
1097.268 !  {"timestamp": {"seconds": 1698660942, "microseconds": 880187}, "event": "MIGRATION_PASS", "data": {"pass": 21}}
1097.270 !  {"timestamp": {"seconds": 1698660942, "microseconds": 882165}, "event": "MIGRATION_PASS", "data": {"pass": 22}}
1097.272 !  {"timestamp": {"seconds": 1698660942, "microseconds": 884377}, "event": "MIGRATION_PASS", "data": {"pass": 23}}
1097.274 !  {"timestamp": {"seconds": 1698660942, "microseconds": 886657}, "event": "MIGRATION_PASS", "data": {"pass": 24}}
1097.276 !  {"timestamp": {"seconds": 1698660942, "microseconds": 888870}, "event": "MIGRATION_PASS", "data": {"pass": 25}}
1097.279 !  {"timestamp": {"seconds": 1698660942, "microseconds": 891860}, "event": "MIGRATION_PASS", "data": {"pass": 26}}
1097.281 !  {"timestamp": {"seconds": 1698660942, "microseconds": 893825}, "event": "MIGRATION_PASS", "data": {"pass": 27}}
1097.283 !  {"timestamp": {"seconds": 1698660942, "microseconds": 895792}, "event": "MIGRATION_PASS", "data": {"pass": 28}}
1097.285 !  {"timestamp": {"seconds": 1698660942, "microseconds": 897764}, "event": "MIGRATION_PASS", "data": {"pass": 29}}
1097.287 !  {"timestamp": {"seconds": 1698660942, "microseconds": 899733}, "event": "MIGRATION_PASS", "data": {"pass": 30}}
1097.289 !  {"timestamp": {"seconds": 1698660942, "microseconds": 901703}, "event": "MIGRATION_PASS", "data": {"pass": 31}}
1097.291 !  {"timestamp": {"seconds": 1698660942, "microseconds": 903670}, "event": "MIGRATION_PASS", "data": {"pass": 32}}
1097.293 !  {"timestamp": {"seconds": 1698660942, "microseconds": 905640}, "event": "MIGRATION_PASS", "data": {"pass": 33}}
1097.295 !  {"timestamp": {"seconds": 1698660942, "microseconds": 907604}, "event": "MIGRATION_PASS", "data": {"pass": 34}}
1097.297 !  {"timestamp": {"seconds": 1698660942, "microseconds": 909575}, "event": "MIGRATION_PASS", "data": {"pass": 35}}
1097.299 !  {"timestamp": {"seconds": 1698660942, "microseconds": 911542}, "event": "MIGRATION_PASS", "data": {"pass": 36}}
1097.301 !  {"timestamp": {"seconds": 1698660942, "microseconds": 913517}, "event": "MIGRATION_PASS", "data": {"pass": 37}}
1097.303 !  {"timestamp": {"seconds": 1698660942, "microseconds": 915812}, "event": "MIGRATION_PASS", "data": {"pass": 38}}
1097.304 !  {"timestamp": {"seconds": 1698660942, "microseconds": 916051}, "event": "MIGRATION", "data": {"status": "failed"}}
1097.304 > {"execute":"query-migrate","id":"libvirt-469"}
1097.322 <  {"return": {"status": "failed", "error-desc": "Unable to write to socket: Bad file descriptor"}, "id": "libvirt-469"}
1097.322 > {"execute":"migrate-set-capabilities","arguments":{"capabilities":[{"capability":"xbzrle","state":false},{"capability":"auto-converge","state":false},{"capability":"rdma-pin-all","state":false},{"capability":"postcopy-ram","state":false},{"capability":"compress","state":false},{"capability":"pause-before-switchover","state":false},{"capability":"late-block-activate","state":false},{"capability":"multifd","state":false},{"capability":"dirty-bitmaps","state":false}......}
1097.325 <  {"id": "libvirt-470", "error": {"class": "GenericError", "desc": "Capability 'switchover-ack' requires capability 'return-path'"}}

Additional info:
(1) The auto cmd to reproduce this issue :

# python3 /home/private_autocase/vfio/vfio_sriov_test.py --feature=vf --domain=$VM --device_name=MT2910-01 --machine_type=q35 --test_list="set_switchover_ack_in_mlx_device_migration"

is cloned by

RHEL-45460 [rhel10-beta][mlx vfio migration] The migration fails with "Received INIT_DATA_SENT but switchover ack is not used" error

Closed

Assignee:: virt-maint

Reporter:: YangHang Liu

Developer:: virt-maint

QA Contact:: virt-bugs

Votes:: 0 Vote for this issue

Watchers:: 15 Start watching this issue

Created:: 2023/10/30 11:10 AM

Updated:: 2024/09/23 6:43 PM

Resolved:: 2024/07/10 8:53 AM

Details

Description

What were you trying to do that didn't work?

Please provide the package NVR for which bug is seen:

How reproducible:

Steps to reproduce

Expected results

Actual results

Additional info

Attachments

Issue Links

Easy Agile Planning Poker

Activity

People

Dates