Uploaded image for project: 'RHEL'
  1. RHEL
  2. RHEL-90160

virtqemud crashed on target host during resume postcopy migration

Linking RHIVOS CVEs to...Migration: Automation ...Sync from "Extern...XMLWordPrintable

    • libvirt-11.4.0-1.el10
    • Yes
    • Low
    • rhel-virt-core-libvirt-1
    • ssg_virtualization
    • 23
    • None
    • False
    • False
    • Hide

      None

      Show
      None
    • None
    • None
    • Unspecified
    • Unspecified
    • Unspecified
    • All
    • 11.4.0
    • None

      What were you trying to do that didn't work?
      virtqemud crashed on target host during resume postcopy migration

      What is the impact of this issue to you?

      Please provide the package NVR for which the bug is seen:
      libvirt-11.2.0-1.el10.x86_64
      qemu-kvm-10.0.0-1.el10.x86_64

      How reproducible is this bug?:
      100%

      Steps to reproduce:

      
      1. Start a guest.
      
      2. Do live migration.
      # virsh migrate test qemu+ssh://XXXX/system --live --postcopy --verbose --bandwidth 5 --postcopy-bandwidth 5
      
      3. Change to postcopy and abort migration in another terminal.
      # virsh migrate-postcopy test
      
      # virsh domjobabort test --postcopy
      
      4. In the first terminal will get following error.
      # virsh migrate test qemu+ssh://XXXX/system --live --postcopy --verbose --bandwidth 5 --postcopy-bandwidth 5
      Migration: [ 8.50 %]error: operation failed: job 'migration in' failed in post-copy phase 
      
      
      Then resume postcopy migration.
      # virsh migrate test qemu+ssh://XXXX/system --live --postcopy --verbose --bandwidth 5 --postcopy-bandwidth 5 --postcopy-resume
      error: End of file while reading data: Warning: Permanently added '10.72.132.46' (ED25519) to the list of known hosts.
      virt-ssh-helper: could not proxy traffic: End of file while reading data: Input/output error: Input/output error
      
      # virsh list --all
       Id   Name   State
      ---------------------
       1    test   paused
       
      
      5. Check the virtqemud coredump file on target host:
      # virsh list --all
       Id   Name   State
      ----------------------
       1    test   running
      
      # coredumpctl list
      TIME                         PID UID GID SIG     COREFILE EXE                 SIZE
      Thu 2025-05-08 01:48:21 EDT 5305   0   0 SIGSEGV present  /usr/sbin/virtqemud 1.1M
      
      # coredumpctl dump 5305
                 PID: 5305 (virtqemud)
                 UID: 0 (root)
                 GID: 0 (root)
              Signal: 11 (SEGV)
           Timestamp: Thu 2025-05-08 01:48:20 EDT (32min ago)
        Command Line: /usr/sbin/virtqemud --timeout 120
          Executable: /usr/sbin/virtqemud
       Control Group: /system.slice/virtqemud.service
                Unit: virtqemud.service
               Slice: system.slice
             Boot ID: 6dcce9f49d78425eaedd3626a6cc3881
          Machine ID: a5e174db5c894405a5747d2e800515e6
            Hostname: dell-per750-63.lab.eng.pek2.redhat.com
             Storage: /var/lib/systemd/coredump/core.virtqemud.0.6dcce9f49d78425eaedd3626a6cc3881.5305.1746683300000000.zst (present)
        Size on Disk: 1.1M
             Message: Process 5305 (virtqemud) of user 0 dumped core.
                      
                      Module libcap.so.2 from rpm libcap-2.69-7.el10.x86_64
                      Module libnss_systemd.so.2 from rpm systemd-257-11.el10.x86_64
                      Module libnbd.so.0 from rpm libnbd-1.22.2-1.el10.x86_64
                      Module libvirt_driver_qemu.so from rpm libvirt-11.2.0-1.el10.x86_64
                      Module libbrotlicommon.so.1 from rpm brotli-1.1.0-6.el10.x86_64
                      Module libevent-2.1.so.7 from rpm libevent-2.1.12-16.el10.x86_64
                      Module libkeyutils.so.1 from rpm keyutils-1.6.3-5.el10.x86_64
                      Module libkrb5support.so.0 from rpm krb5-1.21.3-7.el10.x86_64
                      Module libblkid.so.1 from rpm util-linux-2.40.2-10.el10.x86_64
                      Module libbrotlidec.so.1 from rpm brotli-1.1.0-6.el10.x86_64
                      Module libssl.so.3 from rpm openssl-3.5.0-2.el10.x86_64
                      Module libpsl.so.5 from rpm libpsl-0.21.5-6.el10.x86_64
                      Module libnghttp2.so.14 from rpm nghttp2-1.64.0-2.el10.x86_64
                      Module libcrypt.so.2 from rpm libxcrypt-4.4.36-10.el10.x86_64
                      Module libcrypto.so.3 from rpm openssl-3.5.0-2.el10.x86_64
                      Module libtasn1.so.6 from rpm libtasn1-4.20.0-1.el10.x86_64
                      Module libunistring.so.5 from rpm libunistring-1.1-10.el10.x86_64
                      Module libidn2.so.0 from rpm libidn2-2.3.7-3.el10.x86_64
                      Module libp11-kit.so.0 from rpm p11-kit-0.25.5-7.el10.x86_64
                      Module libattr.so.1 from rpm attr-2.5.2-5.el10.x86_64
                      Module liblzma.so.5 from rpm xz-5.6.2-3.el10.x86_64
                      Module libcom_err.so.2 from rpm e2fsprogs-1.47.1-3.el10.x86_64
                      Module libk5crypto.so.3 from rpm krb5-1.21.3-7.el10.x86_64
                      Module libkrb5.so.3 from rpm krb5-1.21.3-7.el10.x86_64
                      Module libgssapi_krb5.so.2 from rpm krb5-1.21.3-7.el10.x86_64
                      Module libmount.so.1 from rpm util-linux-2.40.2-10.el10.x86_64
                      Module libz.so.1 from rpm zlib-ng-2.2.3-2.el10.x86_64
                      Module libgmodule-2.0.so.0 from rpm glib2-2.80.4-4.el10.x86_64
                      Module libffi.so.8 from rpm libffi-3.4.4-9.el10.x86_64
                      Module libpcre2-8.so.0 from rpm pcre2-10.44-1.el10.3.x86_64
                      Module libcurl.so.4 from rpm curl-8.12.1-2.el10.x86_64
                      Module libsasl2.so.3 from rpm cyrus-sasl-2.1.28-27.el10.x86_64
                      Module libssh.so.4 from rpm libssh-0.11.1-1.el10.x86_64
                      Module libselinux.so.1 from rpm libselinux-3.8-1.el10.x86_64
                      Module libnuma.so.1 from rpm numactl-2.0.19-1.el10.x86_64
                      Module libnl-3.so.200 from rpm libnl3-3.11.0-1.el10.x86_64
                      Module libjson-c.so.5 from rpm json-c-0.18-3.el10.x86_64
                      Module libgnutls.so.30 from rpm gnutls-3.8.9-14.el10.x86_64
                      Module libcap-ng.so.0 from rpm libcap-ng-0.8.4-6.el10.x86_64
                      Module libaudit.so.1 from rpm audit-4.0.3-4.el10.x86_64
                      Module libacl.so.1 from rpm acl-2.3.2-4.el10.x86_64
                      Module libxml2.so.2 from rpm libxml2-2.12.5-5.el10_0.x86_64
                      Module libtirpc.so.3 from rpm libtirpc-1.3.5-1.el10.x86_64
                      Module libgio-2.0.so.0 from rpm glib2-2.80.4-4.el10.x86_64
                      Module libgobject-2.0.so.0 from rpm glib2-2.80.4-4.el10.x86_64
                      Module libglib-2.0.so.0 from rpm glib2-2.80.4-4.el10.x86_64
                      Module libvirt-qemu.so.0 from rpm libvirt-11.2.0-1.el10.x86_64
                      Module libvirt-lxc.so.0 from rpm libvirt-11.2.0-1.el10.x86_64
                      Module libvirt.so.0 from rpm libvirt-11.2.0-1.el10.x86_64
                      Stack trace of thread 5306:
                      #0  0x00007f6320552fed qemuProcessIncomingDefNew (libvirt_driver_qemu.so + 0x152fed)
                      #1  0x00007f632050d662 qemuMigrationDstPrepare (libvirt_driver_qemu.so + 0x10d662)
                      #2  0x00007f632050edf5 qemuMigrationDstPrepareAny (libvirt_driver_qemu.so + 0x10edf5)
                      #3  0x00007f63205104b3 qemuMigrationDstPrepareDirect (libvirt_driver_qemu.so + 0x1104b3)
                      #4  0x00007f63204e1d85 qemuDomainMigratePrepare3Params.lto_priv.0 (libvirt_driver_qemu.so + 0xe1d85)
                      #5  0x00007f632611cfa4 virDomainMigratePrepare3Params (libvirt.so.0 + 0x31cfa4)
                      #6  0x000055bed59e8a91 n/a (n/a + 0x0)
                      ELF object binary architecture: AMD x86-64
      
      

      Expected results
      Migration can complete successfully.

      Actual results
      virtqemud crashed on target host when doing postcopy resume migration

      Additional info:

        1. test.log-dst
          8 kB
        2. virtqemud.log-dst
          2.61 MB
        3. virtqemud.log-src
          1.29 MB

              jdenemar@redhat.com Jiri Denemark
              rhn-support-lcheng Liping Cheng
              Jiri Denemark Jiri Denemark
              Liping Cheng Liping Cheng
              Votes:
              0 Vote for this issue
              Watchers:
              15 Start watching this issue

                Created:
                Updated: