Uploaded image for project: 'RHEL'
  1. RHEL
  2. RHEL-52493

Migration never finished after cancel migration nearly the end of the migration

    • No
    • Moderate
    • rhel-sst-virtualization
    • ssg_virtualization
    • 5
    • False
    • Hide

      None

      Show
      None
    • None
    • None
    • None
    • None
    • x86_64
    • None

      What were you trying to do that didn't work?

      Migration never finished after cancel migration nearly the end of the migration

      Please provide the package NVR for which bug is seen:

      libvirt-10.5.0-4.el9.x86_64

      How reproducible:

      30% (looks like a race condition issue)

      Steps to reproduce

      1. prepare a vm and do post-copy + NBD migration
      
      # virsh migrate vm1 qemu+tcp://XXX/system --live --p2p --tls --postcopy --postcopy-after-precopy --copy-storage-all --migrate-disks vda --verbose 
      
      2. cancel migration by Ctrl + C at the end of postcopy
      # virsh migrate vm1 qemu+tcp://XXX/system --live --p2p --tls --postcopy --postcopy-after-precopy --copy-storage-all --migrate-disks vda --verbose 
      Migration: [100.00 %]^C
      Migration: [100.00 %]
      
      Migration: [100.00 %]^C
      
      3. migration cannot be finished and from gdb you can see that is block in the virtqemud not in the virsh command:
      
      Thread 2 (Thread 0x7f2907bff640 (LWP 77856) "rpc-virtqemud"):
      #0  0x00007f291668679a in __futex_abstimed_wait_common () from /lib64/libc.so.6
      #1  0x00007f2916688fa0 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libc.so.6
      #2  0x00007f2916d4b8fb in virCondWait (c=<optimized out>, m=<optimized out>) at ../src/util/virthread.c:171
      #3  0x00007f2916d76c1b in virDomainObjWait (vm=0x7f28c0093840) at ../src/conf/domain_conf.c:4244
      #4  0x00007f28ee4be481 in qemuDomainObjWait (vm=<optimized out>) at ../src/qemu/qemu_domain.c:12389
      #5  0x00007f28ee5169b7 in qemuMigrationSrcRun (driver=driver@entry=0x7f28c00225a0, vm=vm@entry=0x7f28c0093840, xmlin=xmlin@entry=0x0, persist_xml=persist_xml@entry=0x0, cookiein=cookiein@entry=0x7f290003cb20 "<qemu-migration>\n  <name>vm1</name>\n  <uuid>2e2f2556-5500-47d3-87c6-278d00793c3c</uuid>\n  <hostname>TARGET_HOST</hostname>\n  <hostuuid>4c4c4544-0032-5010-8036-c2c04f344733</"..., cookieinlen=cookieinlen@entry=888, cookieout=0x7f2907bfe4b0, cookieoutlen=0x7f2907bfe48c, flags=98371, resource=0, spec=0x7f2907bfe2e0, dconn=0x7f28f800b610, graphicsuri=0x0, nmigrate_disks=1, migrate_disks=0x7f29000377c0, migParams=0x7f2900038e20, nbdURI=0x0) at ../src/qemu/qemu_migration.c:4987
      #6  0x00007f28ee517eb7 in qemuMigrationSrcPerformNative (driver=driver@entry=0x7f28c00225a0, vm=vm@entry=0x7f28c0093840, xmlin=xmlin@entry=0x0, persist_xml=persist_xml@entry=0x0, uri=uri@entry=0x7f2900055af0 "tcp:TARGET_HOST:49152", cookiein=cookiein@entry=0x7f290003cb20 "<qemu-migration>\n  <name>vm1</name>\n  <uuid>2e2f2556-5500-47d3-87c6-278d00793c3c</uuid>\n  <hostname>TARGET_HOST</hostname>\n  <hostuuid>4c4c4544-0032-5010-8036-c2c04f344733</"..., cookieinlen=888, cookieout=0x7f2907bfe4b0, cookieoutlen=0x7f2907bfe48c, flags=98371, resource=0, dconn=0x7f28f800b610, graphicsuri=0x0, nmigrate_disks=1, migrate_disks=0x7f29000377c0, migParams=0x7f2900038e20, nbdURI=0x0) at ../src/qemu/qemu_migration.c:5247
      #7  0x00007f28ee519fca in qemuMigrationSrcPerformPeer2Peer3 (flags=<optimized out>, useParams=<optimized out>, bandwidth=<optimized out>, migParams=<optimized--Type <RET> for more, q to quit, c to continue without paging--
       out>, nbdURI=<optimized out>, nbdPort=<optimized out>, migrate_disks=<optimized out>, nmigrate_disks=<optimized out>, listenAddress=<optimized out>, graphicsuri=<optimized out>, uri=<optimized out>, dname=<optimized out>, persist_xml=<optimized out>, xmlin=<optimized out>, vm=<optimized out>, dconnuri=<optimized out>, dconn=0x7f28f800b610, sconn=<optimized out>, driver=0x7f28c00225a0) at ../src/qemu/qemu_migration.c:5721
      #8  qemuMigrationSrcPerformPeer2Peer (v3proto=<synthetic pointer>, resource=<optimized out>, dname=<optimized out>, flags=<optimized out>, migParams=<optimized out>, nbdURI=<optimized out>, nbdPort=<optimized out>, migrate_disks=<optimized out>, nmigrate_disks=<optimized out>, listenAddress=<optimized out>, graphicsuri=<optimized out>, uri=<optimized out>, dconnuri=<optimized out>, persist_xml=<optimized out>, xmlin=<optimized out>, vm=<optimized out>, sconn=<optimized out>, driver=0x7f28c00225a0) at ../src/qemu/qemu_migration.c:6022
      #9  qemuMigrationSrcPerformJob (driver=driver@entry=0x7f28c00225a0, conn=conn@entry=0x7f28f800b510, vm=vm@entry=0x7f28c0093840, xmlin=xmlin@entry=0x0, persist_xml=persist_xml@entry=0x0, dconnuri=<optimized out>, uri=<optimized out>, graphicsuri=<optimized out>, listenAddress=<optimized out>, nmigrate_disks=<optimized out>, migrate_disks=<optimized out>, nbdPort=<optimized out>, nbdURI=<optimized out>, migParams=<optimized out>, cookiein=<optimized out>, cookieinlen=<optimized out>, cookieout=<optimized out>, cookieoutlen=<optimized out>, flags=<optimized out>, dname=<optimized out>, resource=<optimized out>, v3proto=<optimized out>) at ../src/qemu/qemu_migration.c:6111
      #10 0x00007f28ee51a5ab in qemuMigrationSrcPerform (driver=0x7f28c00225a0, conn=0x7f28f800b510, vm=0x7f28c0093840, xmlin=0x0, persist_xml=0x0, dconnuri=0x7f2900005f80 "qemu+tcp://TARGET_HOST/system", uri=0x0, graphicsuri=0x0, listenAddress=0x0, nmigrate_disks=1, migrate_disks=0x7f29000377c0, nbdPort=0, nbdURI=0x0, migParams=0x7f2900038e20, cookiein=0x0, cookieinlen=0, cookieout=0x7f2907bfe8b8, cookieoutlen=0x7f2907bfe8ac, flags=98371, dname=0x0, resource=0, v3proto=true) at ../src/qemu/qemu_migration.c:6336
      #11 0x00007f28ee4e0dd2 in qemuDomainMigratePerform3Params (dom=0x7f29000036e0, dconnuri=0x7f2900005f80 "qemu+tcp://TARGET_HOST/system", params=0x7f29000388c0, nparams=1, cookiein=0x0, cookieinlen=0, cookieout=0x7f2907bfe8b8, cookieoutlen=0x7f2907bfe8ac, flags=98371) at ../src/qemu/qemu_driver.c:11287
      #12 0x00007f2916f1ff39 in virDomainMigratePerform3Params (domain=domain@entry=0x7f29000036e0, dconnuri=0x7f2900005f80 "qemu+tcp://TARGET_HOST/system", params=<optimized out>, nparams=1, cookiein=0x0, cookieinlen=0, cookieout=0x7f2907bfe8b8, cookieoutlen=0x7f2907bfe8ac, flags=98371) at ../src/libvirt-domain.c:5433
      #13 0x0000564f0b935a37 in remoteDispatchDomainMigratePerform3Params (server=<optimized out>, msg=0x564f0d8dd080, ret=0x7f2900005fd0, args=0x7f290000ae60, rerr=0x7f2907bfe9a0, client=<optimized out>) at ../src/remote/remote_daemon_dispatch.c:5754
      #14 remoteDispatchDomainMigratePerform3ParamsHelper (server=<optimized out>, client=<optimized out>, msg=0x564f0d8dd080, rerr=0x7f2907bfe9a0, args=0x7f290000ae60, ret=0x7f2900005fd0) at src/remote/remote_daemon_dispatch_stubs.h:8943
      #15 0x00007f2916e1444c in virNetServerProgramDispatchCall (msg=0x564f0d8dd080, client=0x564f0d8dc560, server=0x564f0d8cd080, prog=0x564f0d8cf810) at ../src/rpc/virnetserverprogram.c:423
      #16 virNetServerProgramDispatch (prog=0x564f0d8cf810, server=server@entry=0x564f0d8cd080, client=0x564f0d8dc560, msg=0x564f0d8dd080) at ../src/rpc/virnetserverprogram.c:299
      #17 0x00007f2916e14758 in virNetServerProcessMsg (msg=<optimized out>, prog=<optimized out>, client=<optimized out>, srv=0x564f0d8cd080) at ../src/rpc/virnetserver.c:135
      #18 virNetServerHandleJob (jobOpaque=0x564f0d8c8890, opaque=0x564f0d8cd080) at ../src/rpc/virnetserver.c:155
      #19 0x00007f2916d50ef3 in virThreadPoolWorker (opaque=<optimized out>) at ../src/util/virthreadpool.c:164
      #20 0x00007f2916d50149 in virThreadHelper (data=<optimized out>) at ../src/util/virthread.c:256
      #21 0x00007f2916689c02 in start_thread () from /lib64/libc.so.6
      --Type <RET> for more, q to quit, c to continue without paging--
      #22 0x00007f291670ec40 in clone3 () from /lib64/libc.so.6
      

      Expected results

      Migration can be finished or aborted when user cancel the migration at the end of the migration

      Actual results

      Migration not finished both on the virsh and virtqemud

        1. bt.txt
          19 kB
          Luyao Huang
        2. virtqemud.log
          6.53 MB
          Luyao Huang

              jdenemar@redhat.com Jiri Denemark
              rhn-support-lhuang Luyao Huang
              virt-maint virt-maint
              Luyao Huang Luyao Huang
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

                Created:
                Updated: