-
Bug
-
Resolution: Cannot Reproduce
-
Normal
-
None
-
rhel-9.4
-
None
-
Moderate
-
rhel-virt-core
-
ssg_virtualization
-
None
-
QE ack
-
False
-
False
-
-
None
-
Red Hat Enterprise Linux
-
None
-
None
-
None
-
-
x86_64
-
Linux
-
None
What were you trying to do that didn't work?
The qemu-kvm crashes when post-copy migrating the rt-VM with vhost-user interfaces for multi times
Please provide the package NVR for which bug is seen:
host:
qemu-kvm-8.1.0-4.el9.x86_64
5.14.0-389.el9.x86_64+rt
guest:
5.14.0-389.el9.x86_64+rt
How reproducible:
100%
Steps to reproduce
1. create a ovs-dpdk
2. start a rt-VM with three 1Q vhost-user interfaces
3. post-copy migrate the VM
4. repeat the migration multiple times
In my case, the migration fails at the 3th times
5. check the migration job/qemu-kvm log
On the source host:
initiating migration qemu-kvm: ../softmmu/runstate.c:199: void runstate_set(RunState): Assertion `new_state < RUN_STATE__MAX' failed. shutting down, reason=crashed
Expected results
The qemu-kvm will not crash
Actual results
The qemu-kvm crashes
Additional info:
(1) the VM's qemu-kvm coredump info on the source host:
# coredumpctl info
PID: 6701 (qemu-kvm)
UID: 0 (root)
GID: 107 (qemu)
Signal: 6 (ABRT)
Timestamp: Tue 2023-11-28 04:07:24 EST (1h 8min ago)
Command Line: /usr/libexec/qemu-kvm -name guest=rhel9.4,debug-threads=on -S -object $'{"qom-type":"secret","id":"masterKey0","format> Executable: /usr/libexec/qemu-kvm
Control Group: /machine.slice/machine-qemu\x2d2\x2drhel9.4.scope/libvirt/emulator
Unit: machine-qemu\x2d2\x2drhel9.4.scope
Slice: machine.slice
Boot ID: 2a7bcb0dc2b2436abbacb3aa393dfe7f
Machine ID: 09f2cab20f914272bf3d34215f7fe2ab
Hostname: dell-per740-03.lab.eng.pek2.redhat.com
Storage: /var/lib/systemd/coredump/core.qemu-kvm.0.2a7bcb0dc2b2436abbacb3aa393dfe7f.6701.1701162444000000.zst (present)
Size on Disk: 1.5M
Message: Process 6701 (qemu-kvm) of user 0 dumped core.
Stack trace of thread 6831:
#0 0x00007f246e4e265c __pthread_kill_implementation (libc.so.6 + 0xa365c)
#1 0x00007f246e493d06 raise (libc.so.6 + 0x54d06)
#2 0x00007f246e4677f3 abort (libc.so.6 + 0x287f3)
#3 0x00007f246e46771b __assert_fail_base.cold (libc.so.6 + 0x2871b)
#4 0x00007f246e48cca6 __assert_fail (libc.so.6 + 0x4dca6)
#5 0x00005637a610d8db runstate_set (qemu-kvm + 0x53b8db)
#6 0x00005637a613d60c migration_thread (qemu-kvm + 0x56b60c)
#7 0x00005637a65bcfea qemu_thread_start.llvm.17279737596791441160 (qemu-kvm + 0x9eafea)
#8 0x00007f246e4e0912 start_thread (libc.so.6 + 0xa1912)
#9 0x00007f246e47e314 __clone (libc.so.6 + 0x3f314)
Stack trace of thread 6720:
#0 0x00007f246e4dd4aa __futex_abstimed_wait_common (libc.so.6 + 0x9e4aa)
#1 0x00007f246e4dfcb0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0xa0cb0)
#2 0x00005637a65bc5d6 qemu_cond_wait_impl (qemu-kvm + 0x9ea5d6)
#3 0x00005637a60ffb9b qemu_wait_io_event (qemu-kvm + 0x52db9b)
#4 0x00005637a634f161 kvm_vcpu_thread_fn (qemu-kvm + 0x77d161)
#5 0x00005637a65bcfea qemu_thread_start.llvm.17279737596791441160 (qemu-kvm + 0x9eafea)
#6 0x00007f246e4e0912 start_thread (libc.so.6 + 0xa1912)
#7 0x00007f246e47e314 __clone (libc.so.6 + 0x3f314)
Stack trace of thread 6721:
#0 0x00007f246e4dd4aa __futex_abstimed_wait_common (libc.so.6 + 0x9e4aa)
#1 0x00007f246e4dfcb0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0xa0cb0)
#2 0x00005637a65bc5d6 qemu_cond_wait_impl (qemu-kvm + 0x9ea5d6)
#3 0x00005637a60ffb9b qemu_wait_io_event (qemu-kvm + 0x52db9b)
#4 0x00005637a634f161 kvm_vcpu_thread_fn (qemu-kvm + 0x77d161)
#5 0x00005637a65bcfea qemu_thread_start.llvm.17279737596791441160 (qemu-kvm + 0x9eafea)
#6 0x00007f246e4e0912 start_thread (libc.so.6 + 0xa1912)
#7 0x00007f246e47e314 __clone (libc.so.6 + 0x3f314)
Stack trace of thread 6722:
#0 0x00007f246e4dd4aa __futex_abstimed_wait_common (libc.so.6 + 0x9e4aa)
#1 0x00007f246e4dfcb0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0xa0cb0)
#2 0x00005637a65bc5d6 qemu_cond_wait_impl (qemu-kvm + 0x9ea5d6)
#3 0x00005637a60ffb9b qemu_wait_io_event (qemu-kvm + 0x52db9b)
#4 0x00005637a634f161 kvm_vcpu_thread_fn (qemu-kvm + 0x77d161)
#5 0x00005637a65bcfea qemu_thread_start.llvm.17279737596791441160 (qemu-kvm + 0x9eafea)
#6 0x00007f246e4e0912 start_thread (libc.so.6 + 0xa1912)
#7 0x00007f246e47e314 __clone (libc.so.6 + 0x3f314)
Stack trace of thread 6723:
#0 0x00007f246e4dd4aa __futex_abstimed_wait_common (libc.so.6 + 0x9e4aa)
#1 0x00007f246e4dfcb0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0xa0cb0)
#2 0x00005637a65bc5d6 qemu_cond_wait_impl (qemu-kvm + 0x9ea5d6)
#3 0x00005637a60ffb9b qemu_wait_io_event (qemu-kvm + 0x52db9b)
#4 0x00005637a634f161 kvm_vcpu_thread_fn (qemu-kvm + 0x77d161)
#5 0x00005637a65bcfea qemu_thread_start.llvm.17279737596791441160 (qemu-kvm + 0x9eafea)
#6 0x00007f246e4e0912 start_thread (libc.so.6 + 0xa1912)
#7 0x00007f246e47e314 __clone (libc.so.6 + 0x3f314)
Stack trace of thread 6724:
#0 0x00007f246e4dd4aa __futex_abstimed_wait_common (libc.so.6 + 0x9e4aa)
#1 0x00007f246e4dfcb0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0xa0cb0)
#2 0x00005637a65bc5d6 qemu_cond_wait_impl (qemu-kvm + 0x9ea5d6)
#3 0x00005637a60ffb9b qemu_wait_io_event (qemu-kvm + 0x52db9b)
#4 0x00005637a634f161 kvm_vcpu_thread_fn (qemu-kvm + 0x77d161)
#5 0x00005637a65bcfea qemu_thread_start.llvm.17279737596791441160 (qemu-kvm + 0x9eafea)
#6 0x00007f246e4e0912 start_thread (libc.so.6 + 0xa1912)
#7 0x00007f246e47e314 __clone (libc.so.6 + 0x3f314)
Stack trace of thread 6725:
#0 0x00007f246e4dd4aa __futex_abstimed_wait_common (libc.so.6 + 0x9e4aa)
#1 0x00007f246e4dfcb0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0xa0cb0)
#2 0x00005637a65bc5d6 qemu_cond_wait_impl (qemu-kvm + 0x9ea5d6)
#3 0x00005637a60ffb9b qemu_wait_io_event (qemu-kvm + 0x52db9b)
#4 0x00005637a634f161 kvm_vcpu_thread_fn (qemu-kvm + 0x77d161)
#5 0x00005637a65bcfea qemu_thread_start.llvm.17279737596791441160 (qemu-kvm + 0x9eafea)
#6 0x00007f246e4e0912 start_thread (libc.so.6 + 0xa1912)
#7 0x00007f246e47e314 __clone (libc.so.6 + 0x3f314)
Stack trace of thread 6701:
#0 0x00007f246e5819be ppoll (libc.so.6 + 0x1429be)
#1 0x00005637a65d64f6 main_loop_wait (qemu-kvm + 0xa044f6)
#2 0x00005637a610e637 qemu_main_loop (qemu-kvm + 0x53c637)
#3 0x00005637a5f59cba qemu_default_main (qemu-kvm + 0x387cba)
#4 0x00007f246e47eeb0 __libc_start_call_main (libc.so.6 + 0x3feb0)
#5 0x00007f246e47ef60 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x3ff60)
#6 0x00005637a5f593d5 _start (qemu-kvm + 0x3873d5)
Stack trace of thread 6706:
#0 0x00007f246e47de5d syscall (libc.so.6 + 0x3ee5d)
#1 0x00005637a65bcd4f qemu_event_wait (qemu-kvm + 0x9ead4f)
#2 0x00005637a65cae49 call_rcu_thread (qemu-kvm + 0x9f8e49)
#3 0x00005637a65bcfea qemu_thread_start.llvm.17279737596791441160 (qemu-kvm + 0x9eafea)
#4 0x00007f246e4e0912 start_thread (libc.so.6 + 0xa1912)
#5 0x00007f246e47e450 __clone3 (libc.so.6 + 0x3f450)
Stack trace of thread 6719:
#0 0x00007f246e5818bf __poll (libc.so.6 + 0x1428bf)
#1 0x00007f246e8751fc g_main_context_iterate.constprop.0 (libglib-2.0.so.0 + 0xaa1fc)
#2 0x00007f246e81f5a3 g_main_loop_run (libglib-2.0.so.0 + 0x545a3)
#3 0x00005637a63a9dbf iothread_run (qemu-kvm + 0x7d7dbf)
#4 0x00005637a65bcfea qemu_thread_start.llvm.17279737596791441160 (qemu-kvm + 0x9eafea)
#5 0x00007f246e4e0912 start_thread (libc.so.6 + 0xa1912)
#6 0x00007f246e47e314 __clone (libc.so.6 + 0x3f314)
Stack trace of thread 6726:
#0 0x00007f246e4dd4aa __futex_abstimed_wait_common (libc.so.6 + 0x9e4aa)
#1 0x00007f246e4dfcb0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0xa0cb0)
#2 0x00005637a65bc5d6 qemu_cond_wait_impl (qemu-kvm + 0x9ea5d6)
#3 0x00005637a5f96416 vnc_worker_thread.llvm.11853996392063472260 (qemu-kvm + 0x3c4416)
#4 0x00005637a65bcfea qemu_thread_start.llvm.17279737596791441160 (qemu-kvm + 0x9eafea)
#5 0x00007f246e4e0912 start_thread (libc.so.6 + 0xa1912)
#6 0x00007f246e47e314 __clone (libc.so.6 + 0x3f314)
ELF object binary architecture: AMD x86-64
(2) The VM xml can be as following:
[1] http://10.73.72.41/log/bug/JIRA-17369/iommu/VM.xml
[2] http://10.73.72.41/log/bug/JIRA-17369/non-iommu/VM.xml
