Uploaded image for project: 'RHEL'
  1. RHEL
  2. RHEL-68302

Migration fails after hotplug vcpu

    • No
    • Important
    • CustomerScenariosInitiative
    • rhel-sst-virtualization
    • ssg_virtualization
    • None
    • False
    • Hide

      None

      Show
      None
    • None
    • Red Hat Enterprise Linux
    • None
    • None
    • None
    • Unspecified
    • None

      What were you trying to do that didn't work?
      Hotplug vcpu, then do migration, migration will fail with error:

      (qemu) 2024-11-20T03:20:07.385243Z qemu-kvm: get_pci_config_device: Bad config data: i=0xde read: 4 device: 5 cmask: ff wmask: 0 w1cmask:0
      2024-11-20T03:20:07.385284Z qemu-kvm: Failed to load PCIDevice:config
      2024-11-20T03:20:07.385302Z qemu-kvm: Failed to load virtio-scsi:virtio
      2024-11-20T03:20:07.385317Z qemu-kvm: error while loading state for instance 0x0 of device '0000:00:02.1:00.0/virtio-scsi'
      (qemu) info migrate
      globals:
      store-global-state: on
      only-migratable: off
      send-configuration: on
      send-section-footer: on
      clear-bitmap-shift: 18
      Migration status: failed (load of migration failed: Invalid argument)
      total time: 0 ms
      

      Please provide the package NVR for which bug is seen:
      hosts info: kernel-6.11.0-25.el10.x86_64 && qemu-kvm-9.1.0-5.el10.x86_64
      guest info: kernel-6.11.0-25.el10.x86_64

      How reproducible:
      100%

      Steps to reproduce
      1. Boot a VM with virtio-scsi disk and 2 smp specified:

      -device '{"driver":"virtio-scsi-pci","id":"virtio_scsi_pci0","bus":"root1","addr":"0x0"}' \
      -device '{"driver":"scsi-hd","id":"image1","device_id":"drive-image1","drive":"drive_image1","bus":"virtio_scsi_pci0.0","channel":0,"scsi-id":0,"lun":0,"bootindex":0,"write-cache":"on"}' \
      -blockdev '{"driver":"file","auto-read-only":true,"discard":"unmap","aio":"threads","cache":{"direct":true,"no-flush":false},"filename":"/mnt/xiaohli/rhel100-64-virtio-scsi-ovmf.qcow2","node-name":"drive_sys1"}' \
      -blockdev '{"driver":"qcow2","node-name":"drive_image1","read-only":false,"cache":{"direct":true,"no-flush":false},"file":"drive_sys1"}' \
      -smp 2,threads=1,sockets=1,maxcpus=12 \
      

      2. Boot a VM with same cli as 1 but append with '-incoming defer', and switch smp to 3:

      -device '{"driver":"virtio-scsi-pci","id":"virtio_scsi_pci0","bus":"root1","addr":"0x0"}' \
      -device '{"driver":"scsi-hd","id":"image1","device_id":"drive-image1","drive":"drive_image1","bus":"virtio_scsi_pci0.0","channel":0,"scsi-id":0,"lun":0,"bootindex":0,"write-cache":"on"}' \
      -blockdev '{"driver":"file","auto-read-only":true,"discard":"unmap","aio":"threads","cache":{"direct":true,"no-flush":false},"filename":"/mnt/xiaohli/rhel100-64-virtio-scsi-ovmf.qcow2","node-name":"drive_sys1"}' \
      -blockdev '{"driver":"qcow2","node-name":"drive_image1","read-only":false,"cache":{"direct":true,"no-flush":false},"file":"drive_sys1"}' \
      -smp 3,threads=1,sockets=1,maxcpus=12 \
      -incoming defer \
      

      3. Hotplug one vcpu in VM

      {"execute":"device_add","arguments":{"node-id":0,"socket-id":0,"core-id":2,"thread-id":0,"id":{},"driver":"Haswell-noTSX-IBRS-x86_64-cpu"}}
      

      And check VM, found hotplugging vcpu successfully, see the attachment.

      4. Migrate VM from the src to the dst host

      Expected results
      Migration succeeds, and VM works well

      Actual results
      Migration fails:

      (qemu) 2024-11-20T03:20:07.385243Z qemu-kvm: get_pci_config_device: Bad config data: i=0xde read: 4 device: 5 cmask: ff wmask: 0 w1cmask:0
      2024-11-20T03:20:07.385284Z qemu-kvm: Failed to load PCIDevice:config
      2024-11-20T03:20:07.385302Z qemu-kvm: Failed to load virtio-scsi:virtio
      2024-11-20T03:20:07.385317Z qemu-kvm: error while loading state for instance 0x0 of device '0000:00:02.1:00.0/virtio-scsi'
      (qemu) info migrate
      globals:
      store-global-state: on
      only-migratable: off
      send-configuration: on
      send-section-footer: on
      clear-bitmap-shift: 18
      Migration status: failed (load of migration failed: Invalid argument)
      total time: 0 ms
      

      Additional info:
      The full qemu command:

      /usr/libexec/qemu-kvm  \
      -name "mouse-vm" \
      -sandbox on \
      -machine q35,pflash0=drive_ovmf_code,pflash1=drive_ovmf_vars \
      -cpu Haswell-noTSX-IBRS,vmx=on,pdcm=on,f16c=on,rdrand=on,hypervisor=on,vme=on,ss=on,arat=on,tsc-adjust=on,umip=on,md-clear=on,stibp=on,flush-l1d=on,arch-capabilities=on,ssbd=on,xsaveopt=on,abm=on,pdpe1gb=on,invtsc=on,ibpb=on,ibrs=on,amd-stibp=on,amd-ssbd=on,skip-l1dfl-vmentry=on,pschange-mc-no=on,gds-no=on,vmx-ins-outs=on,vmx-true-ctls=on,vmx-store-lma=on,vmx-activity-hlt=on,vmx-activity-wait-sipi=on,vmx-vmwrite-vmexit-fields=on,vmx-apicv-xapic=on,vmx-ept=on,vmx-desc-exit=on,vmx-rdtscp-exit=on,vmx-apicv-x2apic=on,vmx-vpid=on,vmx-wbinvd-exit=on,vmx-unrestricted-guest=on,vmx-apicv-register=on,vmx-apicv-vid=on,vmx-rdrand-exit=on,vmx-invpcid-exit=on,vmx-vmfunc=on,vmx-shadow-vmcs=on,vmx-pml=on,vmx-ept-execonly=on,vmx-page-walk-4=on,vmx-ept-2mb=on,vmx-ept-1gb=on,vmx-invept=on,vmx-eptad=on,vmx-invept-single-context=on,vmx-invept-all-context=on,vmx-invvpid=on,vmx-invvpid-single-addr=on,vmx-invvpid-all-context=on,vmx-intr-exit=on,vmx-nmi-exit=on,vmx-vnmi=on,vmx-preemption-timer=on,vmx-posted-intr=on,vmx-vintr-pending=on,vmx-tsc-offset=on,vmx-hlt-exit=on,vmx-invlpg-exit=on,vmx-mwait-exit=on,vmx-rdpmc-exit=on,vmx-rdtsc-exit=on,vmx-cr3-load-noexit=on,vmx-cr3-store-noexit=on,vmx-cr8-load-exit=on,vmx-cr8-store-exit=on,vmx-flexpriority=on,vmx-vnmi-pending=on,vmx-movdr-exit=on,vmx-io-exit=on,vmx-io-bitmap=on,vmx-mtf=on,vmx-msr-bitmap=on,vmx-monitor-exit=on,vmx-pause-exit=on,vmx-secondary-ctls=on,vmx-exit-nosave-debugctl=on,vmx-exit-load-perf-global-ctrl=on,vmx-exit-ack-intr=on,vmx-exit-save-pat=on,vmx-exit-load-pat=on,vmx-exit-save-efer=on,vmx-exit-load-efer=on,vmx-exit-save-preemption-timer=on,vmx-entry-noload-debugctl=on,vmx-entry-ia32e-mode=on,vmx-entry-load-perf-global-ctrl=on,vmx-entry-load-pat=on,vmx-entry-load-efer=on,vmx-eptp-switching=on,tsc-frequency=1899996000,host-phys-bits=on,host-phys-bits-limit=46 \
      -nodefaults  \
      -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpmonitor1,server=on,wait=off \
      -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/monitor-catch_monitor,server=on,wait=off \
      -mon chardev=qmp_id_qmpmonitor1,mode=control \
      -mon chardev=qmp_id_catch_monitor,mode=control \
      -device '{"driver":"pcie-root-port","id":"root0","multifunction":true,"bus":"pcie.0","addr":"0x2","chassis":1}' \
      -device '{"driver":"pcie-root-port","id":"root1","port":11,"addr":"0x2.0x1","bus":"pcie.0","chassis":2}' \
      -device '{"driver":"pcie-root-port","id":"root2","port":12,"addr":"0x2.0x2","bus":"pcie.0","chassis":3}' \
      -device '{"driver":"pcie-root-port","id":"root3","port":13,"addr":"0x2.0x3","bus":"pcie.0","chassis":4}' \
      -device '{"driver":"pcie-root-port","id":"root4","port":14,"addr":"0x2.0x4","bus":"pcie.0","chassis":5}' \
      -device '{"driver":"pcie-root-port","id":"root5","port":15,"addr":"0x2.0x5","bus":"pcie.0","chassis":6}' \
      -device '{"driver":"pcie-root-port","id":"root6","port":16,"addr":"0x2.0x6","bus":"pcie.0","chassis":7}' \
      -device '{"driver":"pcie-root-port","id":"root7","port":17,"addr":"0x2.0x7","bus":"pcie.0","chassis":8}' \
      -device '{"driver":"pcie-root-port","id":"extra_root0","multifunction":true,"bus":"pcie.0","addr":"0x3","chassis":21}' \
      -device '{"driver":"pcie-root-port","id":"extra_root1","port":21,"addr":"0x3.0x1","bus":"pcie.0","chassis":22}' \
      -device '{"driver":"pcie-root-port","id":"extra_root2","port":22,"addr":"0x3.0x2","bus":"pcie.0","chassis":23}' \
      -device '{"driver":"nec-usb-xhci","id":"usb1","bus":"root0","addr":"0x0"}' \
      -device '{"driver":"virtio-scsi-pci","id":"virtio_scsi_pci0","bus":"root1","addr":"0x0"}' \
      -device '{"driver":"scsi-hd","id":"image1","device_id":"drive-image1","drive":"drive_image1","bus":"virtio_scsi_pci0.0","channel":0,"scsi-id":0,"lun":0,"bootindex":0,"write-cache":"on"}' \
      -device '{"driver":"virtio-net-pci","mac":"9a:8a:8b:8c:8d:8e","id":"net0","netdev":"tap0","bus":"root2","addr":"0x0"}' \
      -device '{"driver":"usb-tablet","id":"usb-tablet1","bus":"usb1.0","port":"1"}' \
      -device '{"driver":"virtio-balloon-pci","id":"balloon0","bus":"root3","addr":"0x0"}' \
      -device '{"driver":"VGA","id":"video0","vgamem_mb":16,"bus":"pcie.0","addr":"0x1"}' \
      -blockdev '{"driver":"file","auto-read-only":true,"discard":"unmap","aio":"threads","cache":{"direct":true,"no-flush":false},"filename":"/mnt/xiaohli/rhel100-64-virtio-scsi-ovmf.qcow2","node-name":"drive_sys1"}' \
      -blockdev '{"driver":"qcow2","node-name":"drive_image1","read-only":false,"cache":{"direct":true,"no-flush":false},"file":"drive_sys1"}' \
      -blockdev '{"node-name":"file_ovmf_code","driver":"file","filename":"/usr/share/OVMF/OVMF_CODE.secboot.fd","auto-read-only":true,"discard":"unmap"}' \
      -blockdev '{"node-name":"drive_ovmf_code","driver":"raw","read-only":true,"file":"file_ovmf_code"}' \
      -blockdev '{"node-name":"file_ovmf_vars","driver":"file","filename":"/mnt/xiaohli/rhel100-64-virtio-scsi-ovmf.qcow2_VARS.fd","auto-read-only":true,"discard":"unmap"}' \
      -blockdev '{"node-name":"drive_ovmf_vars","driver":"raw","read-only":false,"file":"file_ovmf_vars"}' \
      -netdev tap,id=tap0,vhost=on \
      -m 4096 \
      -smp 2,threads=1,sockets=1,maxcpus=12 \
      -vnc :10 \
      -rtc base=utc,clock=host \
      -boot menu=off,strict=off,order=cdn,once=c \
      -enable-kvm  \
      -qmp tcp:0:3333,server=on,wait=off \
      -serial tcp:0:4444,server=on,wait=off \
      -monitor stdio \
      -msg timestamp=on \
      -object '{"qom-type":"memory-backend-ram","id":"mem0","size":4294967296}' \
      -numa node,memdev=mem0 \
      

              virt-maint virt-maint
              rhn-support-xiaohli Xiaohui Li
              virt-maint virt-maint
              Xiaohui Li Xiaohui Li
              Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

                Created:
                Updated: