Loading...

Type: Bug
Resolution: Duplicate
Priority: Minor
Fix Version/s: None
Affects Version/s: rhel-9.5
Component/s: qemu-kvm / Live Migration
Labels:

Regression:
No
Severity:
Low

Pool Team:

rhel-sst-virtualization
Sub-System Group:

ssg_virtualization

Story Points:
None
ACKs Check:

QE ack
Blocked:
False
Blocked Reason:

Hide

None

Show
None
Product Documentation Required:
None
Products:

Red Hat Enterprise Linux
Sprint:
None

Preliminary Testing:
None
Test Coverage:
None

Experience:
Architecture:

x86_64
OS:
Linux

SFDC Cases Counter:
SFDC Cases Open:
SFDC Cases Links:

Planning:
None

What were you trying to do that didn't work?

The RT VM's post-copy migration will keep running without finish

Please provide the package NVR for which bug is seen:

host:
5.14.0-426.el9.x86_64
qemu-kvm-8.2.0-6.el9.x86_64
libvirt-10.0.0-4.el9.x86_64

How reproducible:

The reproducer ratio < 10%

Steps to reproduce

1. create a ovs-dpdk

2. start a rt-VM with three 1Q vhost-user interfaces
3. post-copy migrate the VM

# /bin/virsh migrate --verbose --persistent --postcopy --live rhel9.4 qemu+ssh://192.168.1.2/system
# /bin/virsh migrate-postcopy rhel9.4

or

# /bin/virsh migrate --verbose --persistent --postcopy  --timeout 3 --timeout-postcopy --live rhel9.4 qemu+ssh://192.168.1.2/system

4. repeat the post-copy migration multiple times

Expected results

The RT VM's post-copy migration always works

Actual results

[1] The QEMU_MONITOR_RECV_REPLY keeps returning "postcopy-active"
[2] the migration will keep running without finish
[3] the source VM is in RUNNING (post-copy) status and target VM is in PAUSED (post-copy)status,

Additional info:

[source machine] gdb -p `pidof qemu-kvm`
gdb) info threads 
  Id   Target Id                                            Frame 
* 1    Thread 0x7f131ab7af80 (LWP 472099) "qemu-kvm"        0x00007f131bcfbafe in ppoll () from target:/lib64/libc.so.6
  2    Thread 0x7f1319b75640 (LWP 472105) "qemu-kvm"        0x00007f131bd0113d in syscall () from target:/lib64/libc.so.6
  3    Thread 0x7f13017fa640 (LWP 472117) "IO mon_iothread" 0x00007f131bcfb9ff in poll () from target:/lib64/libc.so.6
  4    Thread 0x7f13027fc640 (LWP 472118) "CPU 0/KVM"       0x00007f131bc8079a in __futex_abstimed_wait_common () from target:/lib64/libc.so.6
  5    Thread 0x7f13037fe640 (LWP 472119) "CPU 1/KVM"       0x00007f131bc8079a in __futex_abstimed_wait_common () from target:/lib64/libc.so.6
  6    Thread 0x7f1310db9640 (LWP 472120) "CPU 2/KVM"       0x00007f131bc8079a in __futex_abstimed_wait_common () from target:/lib64/libc.so.6
  7    Thread 0x7f1303fff640 (LWP 472121) "CPU 3/KVM"       0x00007f131bc8079a in __futex_abstimed_wait_common () from target:/lib64/libc.so.6
  8    Thread 0x7f1300ff9640 (LWP 472122) "CPU 4/KVM"       0x00007f131bc8079a in __futex_abstimed_wait_common () from target:/lib64/libc.so.6
  9    Thread 0x7f10ef7fe640 (LWP 472123) "CPU 5/KVM"       0x00007f131bc8079a in __futex_abstimed_wait_common () from target:/lib64/libc.so.6
  10   Thread 0x7f10ed3fe640 (LWP 472124) "vnc_worker"      0x00007f131bc8079a in __futex_abstimed_wait_common () from target:/lib64/libc.so.6
  11   Thread 0x7f105e7fc640 (LWP 472726) "return path"     0x00007f131bd097ef in recvmsg () from target:/lib64/libc.so.6
  12   Thread 0x7f10bdffb640 (LWP 472727) "live_migration"  0x00007f131bc8079a in __futex_abstimed_wait_common () from target:/lib64/libc.so.6
(gdb) 
(gdb) 
(gdb) thread apply all bt
Thread 12 (Thread 0x7f10bdffb640 (LWP 472727) "live_migration"):
#0  0x00007f131bc8079a in __futex_abstimed_wait_common () from target:/lib64/libc.so.6
#1  0x00007f131bc856d3 in __pthread_clockjoin_ex () from target:/lib64/libc.so.6
#2  0x00005569399e0f3c in close_return_path_on_source ()
#3  0x00005569399dfeb2 in migration_thread ()
#4  0x0000556939e9974a in qemu_thread_start.llvm ()
#5  0x00007f131bc83c02 in start_thread () from target:/lib64/libc.so.6
#6  0x00007f131bd07f34 in clone () from target:/lib64/libc.so.6

Thread 11 (Thread 0x7f105e7fc640 (LWP 472726) "return path"):
#0  0x00007f131bd097ef in recvmsg () from target:/lib64/libc.so.6
#1  0x0000556939c3c85e in qio_channel_socket_readv ()
#2  0x0000556939f2c7ed in qemu_fill_buffer.llvm ()
#3  0x0000556939f2cf12 in qemu_get_be16 ()
#4  0x00005569399e1218 in source_return_path_thread ()
#5  0x0000556939e9974a in qemu_thread_start.llvm ()
#6  0x00007f131bc83c02 in start_thread () from target:/lib64/libc.so.6
#7  0x00007f131bd07f34 in clone () from target:/lib64/libc.so.6

Thread 10 (Thread 0x7f10ed3fe640 (LWP 472124) "vnc_worker"):
#0  0x00007f131bc8079a in __futex_abstimed_wait_common () from target:/lib64/libc.so.6
#1  0x00007f131bc82fa0 in pthread_cond_wait@@GLIBC_2.3.2 () from target:/lib64/libc.so.6
#2  0x0000556939e98bb6 in qemu_cond_wait_impl ()
#3  0x000055693981b6ea in vnc_worker_thread.llvm ()
#4  0x0000556939e9974a in qemu_thread_start.llvm ()
#5  0x00007f131bc83c02 in start_thread () from target:/lib64/libc.so.6
#6  0x00007f131bd07f34 in clone () from target:/lib64/libc.so.6

Thread 9 (Thread 0x7f10ef7fe640 (LWP 472123) "CPU 5/KVM"):
#0  0x00007f131bc8079a in __futex_abstimed_wait_common () from target:/lib64/libc.so.6
#1  0x00007f131bc82fa0 in pthread_cond_wait@@GLIBC_2.3.2 () from target:/lib64/libc.so.6
#2  0x0000556939e98bb6 in qemu_cond_wait_impl ()
#3  0x000055693999c66b in qemu_wait_io_event ()
#4  0x0000556939c0f351 in kvm_vcpu_thread_fn ()
#5  0x0000556939e9974a in qemu_thread_start.llvm ()
#6  0x00007f131bc83c02 in start_thread () from target:/lib64/libc.so.6
#7  0x00007f131bd07f34 in clone () from target:/lib64/libc.so.6

Thread 8 (Thread 0x7f1300ff9640 (LWP 472122) "CPU 4/KVM"):
#0  0x00007f131bc8079a in __futex_abstimed_wait_common () from target:/lib64/libc.so.6
#1  0x00007f131bc82fa0 in pthread_cond_wait@@GLIBC_2.3.2 () from target:/lib64/libc.so.6
#2  0x0000556939e98bb6 in qemu_cond_wait_impl ()
#3  0x000055693999c66b in qemu_wait_io_event ()
#4  0x0000556939c0f351 in kvm_vcpu_thread_fn ()
#5  0x0000556939e9974a in qemu_thread_start.llvm ()
#6  0x00007f131bc83c02 in start_thread () from target:/lib64/libc.so.6
#7  0x00007f131bd07f34 in clone () from target:/lib64/libc.so.6

Thread 7 (Thread 0x7f1303fff640 (LWP 472121) "CPU 3/KVM"):
#0  0x00007f131bc8079a in __futex_abstimed_wait_common () from target:/lib64/libc.so.6
#1  0x00007f131bc82fa0 in pthread_cond_wait@@GLIBC_2.3.2 () from target:/lib64/libc.so.6
#2  0x0000556939e98bb6 in qemu_cond_wait_impl ()
#3  0x000055693999c66b in qemu_wait_io_event ()
#4  0x0000556939c0f351 in kvm_vcpu_thread_fn ()
#5  0x0000556939e9974a in qemu_thread_start.llvm ()
#6  0x00007f131bc83c02 in start_thread () from target:/lib64/libc.so.6
#7  0x00007f131bd07f34 in clone () from target:/lib64/libc.so.6

Thread 6 (Thread 0x7f1310db9640 (LWP 472120) "CPU 2/KVM"):
#0  0x00007f131bc8079a in __futex_abstimed_wait_common () from target:/lib64/libc.so.6
#1  0x00007f131bc82fa0 in pthread_cond_wait@@GLIBC_2.3.2 () from target:/lib64/libc.so.6
#2  0x0000556939e98bb6 in qemu_cond_wait_impl ()
#3  0x000055693999c66b in qemu_wait_io_event ()
#4  0x0000556939c0f351 in kvm_vcpu_thread_fn ()
#5  0x0000556939e9974a in qemu_thread_start.llvm ()
#6  0x00007f131bc83c02 in start_thread () from target:/lib64/libc.so.6
#7  0x00007f131bd07f34 in clone () from target:/lib64/libc.so.6

Thread 5 (Thread 0x7f13037fe640 (LWP 472119) "CPU 1/KVM"):
#0  0x00007f131bc8079a in __futex_abstimed_wait_common () from target:/lib64/libc.so.6
#1  0x00007f131bc82fa0 in pthread_cond_wait@@GLIBC_2.3.2 () from target:/lib64/libc.so.6
#2  0x0000556939e98bb6 in qemu_cond_wait_impl ()
#3  0x000055693999c66b in qemu_wait_io_event ()
#4  0x0000556939c0f351 in kvm_vcpu_thread_fn ()
#5  0x0000556939e9974a in qemu_thread_start.llvm ()
#6  0x00007f131bc83c02 in start_thread () from target:/lib64/libc.so.6
#7  0x00007f131bd07f34 in clone () from target:/lib64/libc.so.6

Thread 4 (Thread 0x7f13027fc640 (LWP 472118) "CPU 0/KVM"):
#0  0x00007f131bc8079a in __futex_abstimed_wait_common () from target:/lib64/libc.so.6
#1  0x00007f131bc82fa0 in pthread_cond_wait@@GLIBC_2.3.2 () from target:/lib64/libc.so.6
#2  0x0000556939e98bb6 in qemu_cond_wait_impl ()
#3  0x000055693999c66b in qemu_wait_io_event ()
#4  0x0000556939c0f351 in kvm_vcpu_thread_fn ()
#5  0x0000556939e9974a in qemu_thread_start.llvm ()
#6  0x00007f131bc83c02 in start_thread () from target:/lib64/libc.so.6
#7  0x00007f131bd07f34 in clone () from target:/lib64/libc.so.6

Thread 3 (Thread 0x7f13017fa640 (LWP 472117) "IO mon_iothread"):
#0  0x00007f131bcfb9ff in poll () from target:/lib64/libc.so.6
#1  0x00007f131c1031fc in g_main_context_iterate.constprop () from target:/lib64/libglib-2.0.so.0
#2  0x00007f131c0ad5a3 in g_main_loop_run () from target:/lib64/libglib-2.0.so.0
#3  0x0000556939c7012f in iothread_run ()
#4  0x0000556939e9974a in qemu_thread_start.llvm ()
#5  0x00007f131bc83c02 in start_thread () from target:/lib64/libc.so.6
#6  0x00007f131bd07f34 in clone () from target:/lib64/libc.so.6

Thread 2 (Thread 0x7f1319b75640 (LWP 472105) "qemu-kvm"):
#0  0x00007f131bd0113d in syscall () from target:/lib64/libc.so.6
#1  0x0000556939e993ef in qemu_event_wait ()
#2  0x0000556939ea8579 in call_rcu_thread ()
#3  0x0000556939e9974a in qemu_thread_start.llvm ()
#4  0x00007f131bc83c02 in start_thread () from target:/lib64/libc.so.6
#5  0x00007f131bd08c40 in clone3 () from target:/lib64/libc.so.6

Thread 1 (Thread 0x7f131ab7af80 (LWP 472099) "qemu-kvm"):
#0  0x00007f131bcfbafe in ppoll () from target:/lib64/libc.so.6
#1  0x0000556939eb4806 in main_loop_wait ()
#2  0x00005569399aba04 in qemu_main_loop ()
#3  0x00005569397dce1a in qemu_default_main ()
#4  0x00007f131bc23590 in __libc_start_call_main () from target:/lib64/libc.so.6
#5  0x00007f131bc23640 in __libc_start_main_impl () from target:/lib64/libc.so.6
#6  0x00005569397dc535 in _start ()

Details

Description

What were you trying to do that didn't work?

Please provide the package NVR for which bug is seen:

How reproducible:

Steps to reproduce

Expected results

Actual results

Attachments

Easy Agile Planning Poker

Activity

People

Dates