What were you trying to do that didn't work?

What is the impact of this issue to you?

1. Please provide the package NVR for which the bug is seen:

RHEL 9.8.0

kernel-5.14.0-644.el9.x86_64
qemu-kvm-10.1.0-7.el9.x86_64
edk2-ovmf-20241117-8.el9.noarch
seabios-bin-1.16.3-4.el9.noarch

RHEL 8.10

kernel-4.18.0-553.87.1.el8_10.x86_64
qemu-kvm-6.2.0-53.module+el8.10.0+23760+c0926e61.6.x86_64
edk2-ovmf-20220126gitbb1bba3d77-13.el8_10.8.noarch
seabios-bin-1.16.0-4.module+el8.9.0+19570+14a90618.noarch

How reproducible is this bug?:

Steps to reproduce

Step 1

1. Boot up a vm with seabios on rhel 9.8.0 host

/usr/libexec/qemu-kvm -name "mouse-vm" -sandbox off -machine pc-q35-rhel8.6.0 -nodefaults  -cpu Cascadelake-Server,hv_stimer,hv_synic,hv_vpindex,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_frequencies,hv_runtime,hv_tlbflush,hv_reenlightenment,hv_stimer_direct,hv_ipi,+kvm_pv_unhalt -chardev socket,id=charmonitor,path=/home/tmp1,server,nowait -chardev pty,id=charserial0 -chardev socket,id=charchannel1,path=/home/tmp2,server,nowait -chardev socket,id=seabioslog_id,path=/home/seabios,server,nowait -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -device ahci,id=ahci0,bus=pcie.0,addr=0x3 -device ide-cd,bus=ahci0.0,unit=0,drive=drive-ide0-1-1,id=ide0-1-1 -device '{"driver":"pcie-root-port","port":1,"chassis":1,"id":"pcie-root-port0","multifunction":true,"bus":"pcie.0","addr":"0x4"}' -device '{"driver":"virtio-scsi-pci","id":"scsi0","bus":"pcie-root-port0"}' -device '{"driver":"pcie-root-port","port":2,"chassis":2,"id":"pcie-root-port1","bus":"pcie.0","addr":"0x4.0x1"}' -device '{"driver":"scsi-hd","bus":"scsi0.0","lun":0,"drive":"drive-virtio-disk0","id":"virtio-disk0","bootindex":1}' -device '{"driver":"scsi-hd","drive":"drive-scsi-disk","bus":"scsi0.0","lun":1,"id":"data-disk1"}' -device '{"driver":"pcie-root-port","port":3,"chassis":3,"id":"pcie-root-port2","bus":"pcie.0","addr":"0x4.0x2"}' -device '{"driver":"virtio-serial-pci","id":"virtio-serial0","bus":"pcie-root-port2"}' -device '{"driver":"isa-serial","chardev":"charserial0","id":"serial0"}' -device '{"driver":"virtserialport","bus":"virtio-serial0.0","nr":2,"chardev":"charchannel1","id":"channel1","name":"org.qemu.guest_agent.0"}' -device intel-hda,id=sound0,bus=pcie.0,addr=0x7 -device intel-hda,id=sound1,bus=pcie.0,addr=0x8 -device intel-hda,id=sound2,bus=pcie.0,addr=0x9 -device ich9-intel-hda,id=sound3,bus=pcie.0,addr=0xa -device '{"driver":"pvpanic","ioport":1285}' -device '{"driver":"pcie-root-port","port":4,"chassis":4,"id":"pcie-root-port3","bus":"pcie.0","addr":"0x4.0x3"}' -device '{"driver":"e1000e","netdev":"hostnet1","id":"virtio-net-pci1","mac":"52:54:cd:b5:b2:e6","bus":"pcie-root-port3"}' -device '{"driver":"pcie-root-port","port":5,"chassis":5,"id":"pcie-root-port4","bus":"pcie.0","addr":"0x4.0x4"}' -device '{"driver":"virtio-net-pci","netdev":"hostnet2","id":"virtio-net-pci2","mac":"52:54:25:15:8d:5f","bus":"pcie-root-port4"}' -device ide-hd,drive=drive-data-disk,id=system-disk,logical_block_size=512,physical_block_size=512,min_io_size=512,opt_io_size=512,discard_granularity=512,ver=fuxc-ver,bus=ide.0,unit=0 -device '{"driver":"pcie-root-port","port":6,"chassis":6,"id":"pcie-root-port5","bus":"pcie.0","addr":"0x4.0x5"}' -device '{"driver":"ich9-usb-uhci6","id":"uhci6","bus":"pcie-root-port5"}' -device '{"driver":"usb-kbd","id":"kdb0","bus":"uhci6.0"}' -device '{"driver":"pcie-root-port","port":7,"chassis":7,"id":"pcie-root-port6","bus":"pcie.0","addr":"0x4.0x6"}' -device '{"driver":"ich9-usb-uhci5","id":"uhci5","bus":"pcie-root-port6"}' -device '{"driver":"usb-mouse","id":"mouse0","bus":"uhci5.0"}' -device '{"driver":"pcie-root-port","port":8,"chassis":8,"id":"pcie-root-port7","bus":"pcie.0","addr":"0x4.0x7"}' -device '{"driver":"qemu-xhci","id":"xhci","bus":"pcie-root-port7"}' -device '{"driver":"pcie-root-port","port":6,"chassis":9,"id":"pcie-root-port8","bus":"pcie.0","multifunction":true,"addr":"0x10"}' -device '{"driver":"usb-ehci","id":"ehci","bus":"pcie-root-port8"}' -device '{"driver":"pcie-root-port","chassis":10,"port":10,"id":"pcie-root-port9","bus":"pcie.0","addr":"0x10.0x1"}' -device '{"driver":"piix3-usb-uhci","id":"usb","bus":"pcie-root-port9"}' -device '{"driver":"pcie-root-port","port":11,"chassis":11,"id":"pcie-root-port10","bus":"pcie.0","addr":"0x10.0x2"}' -device '{"driver":"ich9-usb-uhci3","id":"uhci","bus":"pcie-root-port10"}' -device '{"driver":"usb-storage","drive":"drive-usb-0","id":"usb-0","removable":true,"bus":"uhci.0","port":"1"}' -device '{"driver":"pcie-root-port","port":12,"chassis":12,"id":"pcie-root-port11","bus":"pcie.0","addr":"0x10.0x3"}' -device '{"driver":"pcie-root-port","port":13,"chassis":13,"id":"pcie-root-port12","bus":"pcie.0","addr":"0x10.0x4"}' -device '{"driver":"ich9-usb-ehci1","id":"ehci1","bus":"pcie-root-port11"}' -device '{"driver":"usb-storage","drive":"drive-usb-1","id":"usb-1","removable":true,"bus":"ehci.0","port":"1"}' -device '{"driver":"qemu-xhci","id":"xhci1","bus":"pcie-root-port12"}' -device '{"driver":"usb-storage","drive":"drive-usb-2","id":"usb-2","removable":true,"bus":"xhci1.0","port":"1"}' -device '{"driver":"pcie-root-port","port":14,"chassis":14,"id":"pcie-root-port13","bus":"pcie.0","addr":"0x10.0x5"}' -device '{"driver":"virtio-rng-pci","id":"rng0","bus":"pcie-root-port13"}' -device '{"driver":"pcie-root-port","port":15,"chassis":15,"id":"pcie-root-port14","bus":"pcie.0","addr":"0x10.0x6"}' -device '{"driver":"virtio-balloon-pci","id":"balloon0","bus":"pcie-root-port14"}' -device isa-debugcon,chardev=seabioslog_id,iobase=0x402 -device virtio-vga -device '{"driver":"pcie-root-port","port":16,"chassis":16,"id":"pcie-root-port15","bus":"pcie.0","addr":"0x10.0x7"}' -device '{"driver":"pcie-pci-bridge","id":"pci.1","bus":"pcie-root-port15","addr":"0x0"}' -device '{"driver":"i6300esb","id":"wdt0","bus":"pci.1","addr":"0x1"}' -blockdev '{"driver":"file","cache":

{"direct":true,"no-flush":false}

,"filename":"/home/rhel810-81-nfs/a1.iso","node-name":"drive_sys1"}' -blockdev '{"driver":"raw","node-name":"drive-ide0-1-0","file":"drive_sys1"}' -blockdev '{"driver":"file","cache":

{"direct":true,"no-flush":false}

,"filename":"/home/rhel810-81-nfs/a2.iso","node-name":"drive_sys2"}' -blockdev '{"driver":"raw","node-name":"drive-ide0-1-1","file":"drive_sys2"}' -blockdev '{"driver":"file","cache":

{"direct":true,"no-flush":false}

,"filename":"/home/rhel810-81-nfs/rhel8100-64-virtio-scsi.qcow2","node-name":"drive_sys3"}' -blockdev '{"driver":"qcow2","node-name":"drive-virtio-disk0","file":"drive_sys3"}' -blockdev '{"driver":"file","cache":

{"direct":true,"no-flush":false}

,"filename":"/home/rhel810-81-nfs/d2.qcow2","node-name":"drive_sys4"}' -blockdev '{"driver":"qcow2","node-name":"drive-scsi-disk","file":"drive_sys4"}' -blockdev '{"driver":"file","cache":

{"direct":true,"no-flush":false}

,"filename":"/home/rhel810-81-nfs/d3.raw","node-name":"drive_sys5"}' -blockdev '{"driver":"raw","node-name":"drive-data-disk","file":"drive_sys5"}' -blockdev '{"driver":"file","cache":

{"direct":true,"no-flush":false}

,"filename":"/home/rhel810-81-nfs/d4.qcow2","node-name":"drive_sys6"}' -blockdev '{"driver":"qcow2","node-name":"usb-disk1","file":"drive_sys6"}' -blockdev '{"driver":"file","cache":

{"direct":true,"no-flush":false}

,"filename":"/home/rhel810-81-nfs/d5.qcow2","node-name":"drive_sys7"}' -blockdev '{"driver":"qcow2","node-name":"usb-disk2","file":"drive_sys7"}' -blockdev '{"driver":"file","cache":

{"direct":true,"no-flush":false}

,"filename":"/home/rhel810-81-nfs/d6.qcow2","node-name":"drive_sys8"}' -blockdev '{"driver":"qcow2","node-name":"usb-disk3","file":"drive_sys8"}' -blockdev '{"driver":"file","cache":

{"direct":true,"no-flush":false}

,"filename":"/home/rhel810-81-nfs/d7.qcow2","node-name":"drive_sys9"}' -blockdev '{"driver":"qcow2","node-name":"drive-usb-0","file":"drive_sys9"}' -blockdev '{"driver":"file","cache":

{"direct":true,"no-flush":false}

,"filename":"/home/rhel810-81-nfs/d8.qcow2","node-name":"drive_sys10"}' -blockdev '{"driver":"qcow2","node-name":"drive-usb-1","file":"drive_sys10"}' -blockdev '{"driver":"file","cache":

{"direct":true,"no-flush":false}

,"filename":"/home/rhel810-81-nfs/d9.qcow2","node-name":"drive_sys11"}' -blockdev '{"driver":"qcow2","node-name":"drive-usb-2","file":"drive_sys11"}' -object '

{"qom-type":"memory-backend-ram","id":"mem-1","prealloc":true,"size":2147483648,"host-nodes":[0],"policy":"bind"}

' -object '

{"qom-type":"memory-backend-ram","id":"mem-2","prealloc":true,"size":2147483648,"host-nodes":[0],"policy":"bind"}

' -numa node,memdev=mem-1 -numa node,memdev=mem-2 -netdev tap,id=hostnet1,vhost=on,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown -netdev tap,id=hostnet2,vhost=on,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown -m 4096,slots=256,maxmem=32G -smp 8,cores=1,threads=1,sockets=8 -vnc :10 -rtc base=utc,clock=host -boot order=cdn,once=c,menu=on,strict=on -enable-kvm  -qmp tcp:0:3333,server,nowait -qmp tcp:0:9999,server=on,wait=off -qmp tcp:0:9888,server=on,wait=off -serial tcp:0:4444,server,nowait -monitor stdio -watchdog-action reset

Step 2 

a.did migration from 9.8.0 to 8.10 host - successfully 

b.boot up vm with "-incoming defer" on rhel 9.8.0 host

c.migrate the vm back to 9.8.0 host  from rhel 8.10 host

(migrate -d tcp:10.73.212.88:4000)

Actual results

pages-per-second: 34313

downtime ram: 714838 kbytes

(qemu) qemu-kvm: failed to save SaveStateEntry with id(name): 1(ram)

qemu-kvm: Unable to write to socket: Connection reset by peer

Expected results

Migration can finish without any errors

Actual results

(qemu) qemu-kvm: ../block.c:6570: bdrv_inactivate_recurse: Assertion `!(bs->open_flags & BDRV_O_INACTIVE)' failed. 

Above doesn't look like the same issue raised in this bug.  Is this a separate bug (bug2)?

Others

Per developer, reported a new bug (2)

However..

(qemu) qemu-kvm: ../block.c:6570: bdrv_inactivate_recurse: Assertion `!(bs->open_flags & BDRV_O_INACTIVE)' failed. 

Above doesn't look like the same issue raised in this bug.  Is this a separate bug (bug2)?

The original error in the description is (bug1):

(qemu) qemu-kvm: failed to save SaveStateEntry with id(name): 1(ram)qemu-kvm: Unable to write to socket: Connection reset by peer 

For bug1, we still need to reproduce, there the important thing is to capture error dumped or crash backtrace of the destination QEMU.

For bug2, I'm not sure how you triggered this path to inactivate some already inactivated disks, I wonder if you have done some disk operations before or during the migration.  We are aware of relevant issues indeed and upstream should likely have it fixed with some recent series [1,2].  Those are still some invasive changes when backporting to RHEL8.10 (the src QEMU version here), so we'll need to evaluate there.  It would be good if we can know more about steps to reproduce bug2. Copy Kevin Wolf too.

Min Deng, some possible next steps here for considerations:

We may want to open another ticket for bug2.  The src QEMU you were using is qemu-kvm-6.2.0-53.el8_10.6.  Please evaluate.
Please keep using this ticket to track bug1, and we'll still need either error reports or crash dump / backtraces from destination side to know why the link was shut.
Do you know the reproduceability for both of the bugs?

[1] https://lore.kernel.org/all/20241206230838.1111496-7-peterx@redhat.com
[2] https://lore.kernel.org/all/20250204211407.381505-1-kwolf@redhat.com