-
Bug
-
Resolution: Unresolved
-
Major
-
rhel-10.0
-
Yes
-
Important
-
CustomerScenariosInitiative
-
rhel-sst-virtualization
-
ssg_virtualization
-
None
-
False
-
-
None
-
None
-
None
-
Automated
-
-
Unspecified
-
None
What were you trying to do that didn't work?
Hit guest call trace under two cases:
1. First boot two VM on the source host, and then boot the third VM3 on the destination host. the VM3 hit call trace on the destination
2. Migrate the VM from the source to the destination during the early stage of boot
What is the impact of this issue to you?
About case 1: the VM3 automatically quit after call trace;
About case 2: migration completes, only hit the guest call trace info when check dmesg on the destination host after migration.
Please provide the package NVR for which the bug is seen:
hosts: kernel-6.11.0-25.el10.x86_64 && qemu-kvm-9.1.0-1.el10.x86_64
guest: kernel-6.11.0-25.el10.x86_64
How reproducible is this bug?:
100%
Steps to reproduce
- Boot VMs with below qemu command lines
For case1:
Boot two VMs on the source host, then boot VM3 on the destination host,
the VM3 on the dst host hit call trace:
2024-09-25-06:48:24: [ 101.780517] watchdog: BUG: soft lockup - CPU#2 stuck for 24s! [systemd:1] 2024-09-25-06:48:24: [ 101.786263] Modules linked in: fuse loop nfnetlink xfs nvme_tcp nvme_keyring sd_mod nvme_fabrics ahci nvme_core bochs libahci drm_vram_helper crct10dif_pclmul crc32_pclmul crc32c_intel nvme_auth virtio_net drm_ttm_helper libata net_failover ttm ghash_clmulni_intel virtio_scsi failover serio_raw dm_mirror dm_region_hash dm_log be2iscsi iscsi_boot_sysfs cxgb4i cxgb4 tls libcxgbi libcxgb iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi dm_multipath dm_mod 2024-09-25-06:48:24: [ 101.788048] CPU: 2 UID: 0 PID: 1 Comm: systemd Not tainted 6.11.0-25.el10.x86_64 #1 2024-09-25-06:48:24: g [0;1;39msyste[ 101.788555] Hardware name: Red Hat KVM/RHEL, BIOS edk2-20240524-6.el10 05/24/2024 2024-09-25-06:48:24: md-modules-load.[ 101.789073] RIP: 0010:_raw_spin_unlock_irqrestore+0x1f/0x30 2024-09-25-06:48:24: service[0m - Lo[ 101.789582] Code: 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 0f 1f 44 00 00 e8 82 07 00 00 90 f7 c6 00 02 00 00 74 01 fb 65 ff 0d 89 d0 c3 73 <74> 05 c3 cc cc cc cc 0f 1f 44 00 00 c3 cc cc cc cc 90 90 90 90 90 2024-09-25-06:48:24: ad Kernel Module[ 101.790609] RSP: 0018:ffffa5a24001b8f8 EFLAGS: 00000286 2024-09-25-06:48:24: s... 2024-09-25-06:48:24: 2024-09-25-06:48:24: 2024-09-25-06:48:24: [0;21;3[ 101.791170] RAX: 0000000000000001 RBX: ffffffff8e5bd060 RCX: 0000000000000000 2024-09-25-06:48:24: 9msystemd[1]: St[ 101.791697] RDX: 0000000000000002 RSI: 0000000000000246 RDI: ffffffff8e5bd060 2024-09-25-06:48:24: arting systemd-n[ 101.792214] RBP: ffff8efbaf943100 R08: 0000000000000002 R09: 0000000000000001 2024-09-25-06:48:24: etwork-generator[ 101.792737] R10: ffffffff8d721be8 R11: 0000000000000003 R12: ffff8efb80294d40 2024-09-25-06:48:24: .service - Gener[ 101.793219] R13: 0000000000001002 R14: ffff8efb811b0000 R15: ffff8efb81593750 2024-09-25-06:48:24: ate network unit[ 101.793722] FS: 00007f94dc448440(0000) GS:ffff8efbfbd00000(0000) knlGS:0000000000000000 2024-09-25-06:48:24: s from Kernel co[ 101.794227] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 2024-09-25-06:48:24: mmand line...[0[ 101.794715] CR2: 00007f0a1bc51080 CR3: 00000001026c6003 CR4: 0000000000170ef0 2024-09-25-06:48:24: m 2024-09-25-06:48:24: [0;21;39mAdd[ 101.795225] Call Trace: 2024-09-25-06:48:24: ing 2097148k swa[ 101.795723] <IRQ> 2024-09-25-06:48:24: p on /dev/mapper[ 101.796234] ? show_trace_log_lvl+0x1b0/0x2f0 2024-09-25-06:48:24: /rhel-swap. Pri[ 101.796722] ? show_trace_log_lvl+0x1b0/0x2f0 2024-09-25-06:48:24: ority:-2 extents[ 101.797208] ? uart_dtr_rts+0x86/0x90 2024-09-25-06:48:24: :1 across:209714[ 101.797677] ? watchdog_timer_fn.cold+0x3d/0xa0 2024-09-25-06:48:24: 8k [0m 2024-09-25-06:48:24: [ 101.798154] ? __pfx_watchdog_timer_fn+0x10/0x10 2024-09-25-06:48:24: Starting [0;1[ 101.798611] ? __hrtimer_run_queues+0x13c/0x2a0 2024-09-25-06:48:24: ;39msystemd-netw[ 101.799066] ? hrtimer_interrupt+0xfa/0x210 2024-09-25-06:48:24: ork-generator[ 101.799500] ? __sysvec_apic_timer_interrupt+0x55/0x100 2024-09-25-06:48:24: [0mk units from[ 101.799936] ? sysvec_apic_timer_interrupt+0x6c/0x90 2024-09-25-06:48:24: Kernel command [ 101.800358] </IRQ> 2024-09-25-06:48:24: line... 2024-09-25-06:48:24: 2024-09-25-06:48:24: 2024-09-25-06:48:24: [0;2[ 101.800780] <TASK> 2024-09-25-06:48:24: 1;39msystemd[1]:[ 101.801186] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 2024-09-25-06:48:24: systemd-pcrmach[ 101.801605] ? _raw_spin_unlock_irqrestore+0x1f/0x30 2024-09-25-06:48:24: ine.service - TP[ 101.802023] ? _raw_spin_unlock_irqrestore+0xe/0x30 2024-09-25-06:48:24: M PCR Machine ID[ 101.802429] uart_dtr_rts+0x86/0x90 2024-09-25-06:48:24: Measurement was[ 101.802849] tty_port_block_til_ready+0x27d/0x330 2024-09-25-06:48:24: skipped because[ 101.803253] ? __pfx_autoremove_wake_function+0x10/0x10 2024-09-25-06:48:24: of an unmet con[ 101.803650] uart_open+0x1e/0x30 2024-09-25-06:48:24: dition check (Co[ 101.804048] tty_open+0x161/0x6a0 2024-09-25-06:48:24: nditionSecurity=[ 101.804427] chrdev_open+0xcb/0x240 2024-09-25-06:48:24: measured-uki).[[ 101.804809] ? __pfx_chrdev_open+0x10/0x10 2024-09-25-06:48:24: 0m[ 101.805164] do_dentry_open+0x25a/0x490 2024-09-25-06:48:24: [ 101.805514] vfs_open+0x34/0xf0 2024-09-25-06:48:24: [ 101.805867] do_open+0x165/0x3d0 2024-09-25-06:48:24: 2024-09-25-06:48:24: [ 101.806206] path_openat+0x124/0x2d0 2024-09-25-06:48:24: [ 101.806539] ? pick_next_task+0x50/0xb10 2024-09-25-06:48:24: [ 101.806875] do_filp_open+0xc4/0x170 2024-09-25-06:48:24: Startin[ 101.807191] do_sys_openat2+0xae/0xe0 2024-09-25-06:48:24: g [0;1;39msyste[ 101.807492] __x64_sys_openat+0x55/0xa0 2024-09-25-06:48:24: md-remount-fs.se[ 101.807795] do_syscall_64+0x7d/0x160 2024-09-25-06:48:24: rvic…[0munt R[ 101.808089] ? __do_sys_newfstat+0x68/0x70 2024-09-25-06:48:24: oot and Kernel F[ 101.808366] ? __memcg_slab_free_hook+0x100/0x150 2024-09-25-06:48:24: ile Systems... 2024-09-25-06:48:24: 2024-09-25-06:48:24: [ 101.808633] ? __x64_sys_close+0x3c/0x80 2024-09-25-06:48:24: [ 101.808895] ? kmem_cache_free+0x3d5/0x400 2024-09-25-06:48:24: 2024-09-25-06:48:24: [ 101.809139] ? syscall_exit_to_user_mode+0x10/0x1f0 2024-09-25-06:48:24: Startin[ 101.809381] ? do_syscall_64+0x89/0x160 2024-09-25-06:48:24: g [0;1;39msyste[ 101.809618] ? filp_flush+0x56/0x70 2024-09-25-06:48:24: md-udev-load-cre[ 101.809851] ? syscall_exit_to_user_mode+0x10/0x1f0 2024-09-25-06:48:24: dent[0mLoad [ 101.810064] ? do_syscall_64+0x89/0x160 2024-09-25-06:48:24: udev Rules from [ 101.810287] ? syscall_exit_to_user_mode+0x10/0x1f0 2024-09-25-06:48:24: Credentials... 2024-09-25-06:48:24: 2024-09-25-06:48:24: [ 101.810495] ? do_syscall_64+0x89/0x160 2024-09-25-06:48:24: [ 101.810700] ? do_syscall_64+0x89/0x160 2024-09-25-06:48:24: 2024-09-25-06:48:24: [ 101.810907] ? exc_page_fault+0x73/0x160 2024-09-25-06:48:24: [ 101.811106] entry_SYSCALL_64_after_hwframe+0x76/0x7e 2024-09-25-06:48:24: Startin[ 101.811310] RIP: 0033:0x7f94dbb20210 2024-09-25-06:48:24: g [0;1;39msyste[ 101.811648] Code: 83 e2 40 75 55 89 f0 f7 d0 a9 00 00 41 00 74 4a 80 3d 43 de 0d 00 00 74 6e 89 da 48 89 ee bf 9c ff ff ff b8 01 01 00 00 0f 05 <48> 3d 00 f0 ff ff 0f 87 94 00 00 00 48 8b 54 24 28 64 48 2b 14 25 2024-09-25-06:48:24: [ 101.812084] RSP: 002b:00007ffc30704f80 EFLAGS: 00000202 ORIG_RAX: 0000000000000101 2024-09-25-06:48:24: md-udev-trigger.[ 101.812304] RAX: ffffffffffffffda RBX: 0000000000080101 RCX: 00007f94dbb20210 2024-09-25-06:48:24: service[0m - Co[ 101.812527] RDX: 0000000000080101 RSI: 00007f94dc37a5d6 RDI: 00000000ffffff9c 2024-09-25-06:48:24: ldplug All udev [ 101.812772] RBP: 00007f94dc37a5d6 R08: 0000000000000073 R09: 00000000ffffffff 2024-09-25-06:48:24: Devices... 2024-09-25-06:48:24: 2024-09-25-06:48:24: [ 101.813058] R10: 0000000000000000 R11: 0000000000000202 R12: 0000000000080101 2024-09-25-06:48:24: 2024-09-25-06:48:24: [ 101.813289] R13: 00007f94dc37a5d6 R14: 00007ffc30705000 R15: 000055f8c591def0 2024-09-25-06:48:24: [ 101.813522] </TASK> 2024-09-25-06:48:24: [[0;32m OK [[ 101.813761] watchdog: BUG: soft lockup - CPU#0 stuck for 24s! [plymouthd:618] 2024-09-25-06:48:24: 0m] Activated sw[ 101.814059] Modules linked in: fuse loop nfnetlink xfs nvme_tcp nvme_keyring sd_mod nvme_fabrics ahci nvme_core bochs libahci drm_vram_helper crct10dif_pclmul crc32_pclmul crc32c_intel nvme_auth virtio_net drm_ttm_helper libata net_failover ttm ghash_clmulni_intel virtio_scsi failover serio_raw dm_mirror dm_region_hash dm_log be2iscsi iscsi_boot_sysfs cxgb4i cxgb4 tls libcxgbi libcxgb iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi dm_multipath dm_mod 2024-09-25-06:48:24: [ 101.815154] CPU: 0 UID: 0 PID: 618 Comm: plymouthd Tainted: G L ------- --- 6.11.0-25.el10.x86_64 #1 2024-09-25-06:48:24: ap [0;1;39mdev-[ 101.815550] Tainted: [L]=SOFTLOCKUP 2024-09-25-06:48:24: disk-by\x2duuid[ 101.815947] Hardware name: Red Hat KVM/RHEL, BIOS edk2-20240524-6.el10 05/24/2024 2024-09-25-06:48:24: [0m4cb060-21a[ 101.816331] RIP: 0010:_raw_spin_unlock_irqrestore+0x1f/0x30 2024-09-25-06:48:24: 8-434a-91e6-83e0[ 101.816736] Code: 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 0f 1f 44 00 00 e8 82 07 00 00 90 f7 c6 00 02 00 00 74 01 fb 65 ff 0d 89 d0 c3 73 <74> 05 c3 cc cc cc cc 0f 1f 44 00 00 c3 cc cc cc cc 90 90 90 90 90 2024-09-25-06:48:24: 369e1296. 2024-09-25-06:48:24: 2024-09-25-06:48:24: [ 101.817604] RSP: 0018:ffffa5a2403a3be0 EFLAGS: 00000286 2024-09-25-06:48:24: 2024-09-25-06:48:24: [ 101.818040] RAX: 0000000000000001 RBX: 0000000000000e75 RCX: 0000000000000000 2024-09-25-06:48:24: [ 101.818454] RDX: 0000000000000000 RSI: 0000000000000286 RDI: ffffffff8e5bd060 2024-09-25-06:48:24: [[0;32m OK [[ 101.818849] RBP: ffffffff8e5bd060 R08: 0000000000000000 R09: 0000000000000001 2024-09-25-06:48:24: 0m] Started [0;[ 101.819181] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000286 2024-09-25-06:48:24: 1;39msystemd-jou[ 101.819506] R13: ffffa5a2403d1000 R14: ffff8efb815ec000 R15: 00000000000000a7 2024-09-25-06:48:24: rnald.service[0[ 101.819838] FS: 00007fd043521280(0000) GS:ffff8efbfbc00000(0000) knlGS:0000000000000000 2024-09-25-06:48:24: m - Journal Serv[ 101.820174] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 2024-09-25-06:48:24: ice. 2024-09-25-06:48:24: 2024-09-25-06:48:24: [ 101.820510] CR2: 00007f32e1aee080 CR3: 00000001137b8004 CR4: 0000000000170ef0 2024-09-25-06:48:24: 2024-09-25-06:48:24: [ 101.820863] Call Trace: 2024-09-25-06:48:24: [ 101.821211] <IRQ> 2024-09-25-06:48:24: [[0;32m OK [[ 101.821561] ? show_trace_log_lvl+0x1b0/0x2f0 2024-09-25-06:48:24: 0m] Mounted [0;[ 101.821924] ? show_trace_log_lvl+0x1b0/0x2f0 2024-09-25-06:48:24: 1;39mdev-hugepag[ 101.822279] ? uart_write_room+0x74/0x190 2024-09-25-06:48:24: es.mount[0m - H[ 101.822634] ? watchdog_timer_fn.cold+0x3d/0xa0 2024-09-25-06:48:24: uge Pages File S[ 101.822996] ? __pfx_watchdog_timer_fn+0x10/0x10 2024-09-25-06:48:24: ystem. 2024-09-25-06:48:24: 2024-09-25-06:48:24: 2024-09-25-06:48:24: [[0;3[ 101.823353] ? __hrtimer_run_queues+0x13c/0x2a0 2024-09-25-06:48:24: 2m OK [0m] Mo[ 101.823710] ? hrtimer_interrupt+0xfa/0x210 2024-09-25-06:48:24: unted [0;1;39md[ 101.824070] ? __sysvec_apic_timer_interrupt+0x55/0x100 2024-09-25-06:48:24: ev-mqueue.mount[ 101.824429] ? sysvec_apic_timer_interrupt+0x6c/0x90 2024-09-25-06:48:24: [0m - POSIX Mess[ 101.824792] </IRQ> 2024-09-25-06:48:24: age Queue File S[ 101.825144] <TASK> 2024-09-25-06:48:24: ystem. 2024-09-25-06:48:24: 2024-09-25-06:48:24: 2024-09-25-06:48:24: [[0;3[ 101.825497] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 2024-09-25-06:48:24: 2m OK [0m] Mo[ 101.825861] ? _raw_spin_unlock_irqrestore+0x1f/0x30 2024-09-25-06:48:24: unted [0;1;39ms[ 101.826218] uart_write_room+0x74/0x190 2024-09-25-06:48:24: ys-kernel-debug.[ 101.826577] process_output_block+0x3a/0x1c0 2024-09-25-06:48:24: mount[0m - Kern[ 101.826954] ? add_wait_queue+0x7e/0xb0 2024-09-25-06:48:24: el Debug File Sy[ 101.827312] n_tty_write+0x1a4/0x3c0 2024-09-25-06:48:24: stem. 2024-09-25-06:48:24: 2024-09-25-06:48:24: 2024-09-25-06:48:24: [[0;32[ 101.827668] ? __pfx_woken_wake_function+0x10/0x10 2024-09-25-06:48:24: m OK [0m] Mou[ 101.828033] iterate_tty_write+0x114/0x240 2024-09-25-06:48:24: nted [0;1;39msy[ 101.828392] file_tty_write.isra.0+0x86/0xb0 2024-09-25-06:48:24: s-kernel-tracing[ 101.828746] vfs_write+0x294/0x460 2024-09-25-06:48:24: .mount[0m - Ker[ 101.829087] ksys_write+0x6d/0xf0 2024-09-25-06:48:24: nel Trace File S[ 101.829426] do_syscall_64+0x7d/0x160 2024-09-25-06:48:24: ystem. 2024-09-25-06:48:24: 2024-09-25-06:48:24: 2024-09-25-06:48:24: [ 101.829769] ? syscall_exit_to_user_mode+0x10/0x1f0 2024-09-25-06:48:24: [ 101.830107] ? do_syscall_64+0x89/0x160 2024-09-25-06:48:24: [ 101.830448] ? ksys_read+0x6d/0xf0 2024-09-25-06:48:24: [ 101.830787] ? syscall_exit_to_user_mode+0x10/0x1f0 2024-09-25-06:48:24: [ 101.831121] ? do_syscall_64+0x89/0x160 2024-09-25-06:48:24: [ 101.831434] entry_SYSCALL_64_after_hwframe+0x76/0x7e 2024-09-25-06:48:24: [ 101.831738] RIP: 0033:0x7fd0437ac6a4 2024-09-25-06:48:24: [ 101.832048] Code: c7 00 16 00 00 00 b8 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 80 3d a5 c9 0d 00 00 74 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 48 83 ec 28 48 89 54 24 18 48 2024-09-25-06:48:24: [ 101.832667] RSP: 002b:00007ffeb2d85198 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 2024-09-25-06:48:24: [0;21;39mfuse: [ 101.832997] RAX: ffffffffffffffda RBX: 0000563fd8c56110 RCX: 00007fd0437ac6a4 2024-09-25-06:48:24: init (API versio[ 101.833299] RDX: 00000000000000a7 RSI: 0000563fd8c81f60 RDI: 000000000000000a 2024-09-25-06:48:24: n 7.40)[0m[ 101.833584] RBP: 00000000000000a7 R08: 0000563fd8c844c0 R09: 00000000000000d0 2024-09-25-06:48:24: [ 101.833871] R10: 0000000000000009 R11: 0000000000000202 R12: 0000563fd8c890c0 2024-09-25-06:48:24: [ 101.834152] R13: 0000563fd8c80640 R14: 0000563fcc03e4c2 R15: 0000563fd8c89180 2024-09-25-06:48:24: 2024-09-25-06:48:24: [ 101.834442] </TASK> 2024-09-25-06:48:24: [ 101.834705] watchdog: BUG: soft lockup - CPU#1 stuck for 24s! [kworker/1:2:301] 2024-09-25-06:48:24: [ 101.835127] Modules linked in: fuse loop nfnetlink xfs nvme_tcp nvme_keyring sd_mod nvme_fabrics ahci nvme_core bochs libahci drm_vram_helper crct10dif_pclmul crc32_pclmul crc32c_intel nvme_auth virtio_net drm_ttm_helper libata net_failover ttm ghash_clmulni_intel virtio_scsi failover serio_raw dm_mirror dm_region_hash dm_log be2iscsi iscsi_boot_sysfs cxgb4i cxgb4 tls libcxgbi libcxgb iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi dm_multipath dm_mod 2024-09-25-06:48:24: [ 101.836467] CPU: 1 UID: 0 PID: 301 Comm: kworker/1:2 Tainted: G L ------- --- 6.11.0-25.el10.x86_64 #1 2024-09-25-06:48:24: [ 101.836946] Tainted: [L]=SOFTLOCKUP 2024-09-25-06:48:24: [[0;32m OK [[ 101.837422] Hardware name: Red Hat KVM/RHEL, BIOS edk2-20240524-6.el10 05/24/2024 2024-09-25-06:48:24: 0m] Finished [0[ 101.837846] Workqueue: pm pm_runtime_work 2024-09-25-06:48:24: ;1;39mkmod-stati[ 101.838238] RIP: 0010:_raw_spin_unlock_irqrestore+0x1f/0x30 2024-09-25-06:48:24: c-nodes.service[ 101.838662] Code: 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 0f 1f 44 00 00 e8 82 07 00 00 90 f7 c6 00 02 00 00 74 01 fb 65 ff 0d 89 d0 c3 73 <74> 05 c3 cc cc cc cc 0f 1f 44 00 00 c3 cc cc cc cc 90 90 90 90 90 2024-09-25-06:48:24: [ 101.839604] RSP: 0018:ffffa5a2405c3d50 EFLAGS: 00000286 2024-09-25-06:48:24: [ 101.840102] RAX: 0000000000000001 RBX: ffffffff8e5bd060 RCX: 0000000000000000 2024-09-25-06:48:24: [0mCreate Lis[ 101.840595] RDX: 0000000000000001 RSI: 0000000000000246 RDI: ffffffff8e5bd060 2024-09-25-06:48:24: t of Static Devi[ 101.841099] RBP: ffff8efb838ecc00 R08: 0000000000000001 R09: 0000000000000001 2024-09-25-06:48:24: ce Nodes. 2024-09-25-06:48:24: 2024-09-25-06:48:24: 2024-09-25-06:48:24: [[[ 101.841623] R10: 0000000000000003 R11: 0000000000000000 R12: 0000000000000246 2024-09-25-06:48:24: 0;32m OK [0m][ 101.842128] R13: 0000000000000001 R14: 0000000000000000 R15: ffff8efb901499c0 2024-09-25-06:48:24: Finished [0;1;[ 101.842628] FS: 0000000000000000(0000) GS:ffff8efbfbc80000(0000) knlGS:0000000000000000 2024-09-25-06:48:24: 39mlvm2-monitor.[ 101.843144] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 2024-09-25-06:48:24: service[0m - Mo[ 101.843652] CR2: 00005618866fa9d0 CR3: 00000001113c4005 CR4: 0000000000170ef0 2024-09-25-06:48:24: …using dmevent[ 101.844174] Call Trace: 2024-09-25-06:48:24: d or progress po[ 101.844682] <IRQ> 2024-09-25-06:48:24: lling. 2024-09-25-06:48:24: 2024-09-25-06:48:24: 2024-09-25-06:48:24: [[0;3[ 101.845201] ? show_trace_log_lvl+0x1b0/0x2f0 2024-09-25-06:48:24: 2m OK [0m] Fi[ 101.845715] ? show_trace_log_lvl+0x1b0/0x2f0 2024-09-25-06:48:24: nished [0;1;39m[ 101.846224] ? serial_port_runtime_suspend+0x12b/0x230 2024-09-25-06:48:24: modprobe@configf[ 101.846743] ? watchdog_timer_fn.cold+0x3d/0xa0 2024-09-25-06:48:24: s.service[0m - [ 101.847265] ? __pfx_watchdog_timer_fn+0x10/0x10 2024-09-25-06:48:24: Load Kernel Modu[ 101.847777] ? __hrtimer_run_queues+0x13c/0x2a0 2024-09-25-06:48:24: le configfs. 2024-09-25-06:48:24: 2024-09-25-06:48:24: 2024-09-25-06:48:24: [ 101.848282] ? hrtimer_interrupt+0xfa/0x210 2024-09-25-06:48:24: [[0;32m OK [[ 101.848796] ? __sysvec_apic_timer_interrupt+0x55/0x100 2024-09-25-06:48:24: 0m] Finished [0[ 101.849300] ? sysvec_apic_timer_interrupt+0x6c/0x90 2024-09-25-06:48:24: ;1;39mmodprobe@d[ 101.849816] </IRQ> 2024-09-25-06:48:24: m_mod.service[0[ 101.850319] <TASK> 2024-09-25-06:48:24: m - Load Kernel [ 101.850832] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 2024-09-25-06:48:24: Module dm_mod. 2024-09-25-06:48:24: 2024-09-25-06:48:24: [ 101.851344] ? _raw_spin_unlock_irqrestore+0x1f/0x30 2024-09-25-06:48:24: [ 101.851866] serial_port_runtime_suspend+0x12b/0x230 2024-09-25-06:48:24: 2024-09-25-06:48:24: [[0;32m OK [0m] Finished [ 101.852372] ? __pfx_serial_port_runtime_suspend+0x10/0x10 2024-09-25-06:48:24: [0;1;39mmodprobe[ 101.852895] __rpm_callback+0x44/0x170 2024-09-25-06:48:24: @dm_multipath.se[ 101.853417] ? __pfx_serial_port_runtime_suspend+0x10/0x10 2024-09-25-06:48:24: r[0m - Load [ 101.853941] rpm_callback+0x55/0x60 2024-09-25-06:48:24: Kernel Module dm[ 101.854449] ? __pfx_serial_port_runtime_suspend+0x10/0x10 2024-09-25-06:48:24: _multipath. 2024-09-25-06:48:24: 2024-09-25-06:48:24: 2024-09-25-06:48:24: [[ 101.854962] rpm_suspend+0xe6/0x5f0 2024-09-25-06:48:24: [0;32m OK [0[ 101.855449] ? _raw_spin_unlock+0xe/0x30 2024-09-25-06:48:24: m] Finished [0;[ 101.855982] ? finish_task_switch.isra.0+0x99/0x2b0 2024-09-25-06:48:24: 1;39mmodprobe@dr[ 101.856461] pm_runtime_work+0x98/0xb0 2024-09-25-06:48:24: m.service[0m - [ 101.856949] process_one_work+0x177/0x330 2024-09-25-06:48:24: Load Kernel Modu[ 101.857453] worker_thread+0x252/0x390 2024-09-25-06:48:24: le drm. 2024-09-25-06:48:24: 2024-09-25-06:48:24: 2024-09-25-06:48:24: [[0;[ 101.857954] ? __pfx_worker_thread+0x10/0x10 2024-09-25-06:48:24: 32m OK [0m] F[ 101.858424] kthread+0xd2/0x100 2024-09-25-06:48:24: inished [0;1;39[ 101.858798] ? __pfx_kthread+0x10/0x10 2024-09-25-06:48:24: mmodprobe@efi_ps[ 101.859209] ret_from_fork+0x34/0x50 2024-09-25-06:48:24: tore.service[0m[ 101.859649] ? __pfx_kthread+0x10/0x10 2024-09-25-06:48:24: - Load Kernel M[ 101.860088] ret_from_fork_asm+0x1a/0x30 2024-09-25-06:48:24: odule efi_pstore[ 101.860485] </TASK> 2024-09-25-06:48:24: . 2024-09-25-06:48:24: 2024-09-25-06:48:24: 2024-09-25-06:48:24: [[0;32m O[ 101.996834] systemd-journald[916]: Collecting audit messages is disabled. 2024-09-25-06:48:24: K [0m] Finishe[ 104.147963] systemd[1]: Starting systemd-udev-trigger.service - Coldplug All udev Devices... 2024-09-25-06:48:24: [ 104.158162] systemd[1]: Activated swap dev-disk-by\x2duuid-794cb060\x2d21a8\x2d434a\x2d91e6\x2d83e0369e1296.swap - /dev/disk/by-uuid/794cb060-21a8-434a-91e6-83e0369e1296. 2024-09-25-06:48:24: [ 104.161685] systemd[1]: Started systemd-journald.service - Journal Service. 2024-09-25-06:48:24: d [0;1;39mmodprobe@fuse.service[0m - Load Kernel Module fuse. 2024-09-25-06:48:24: 2024-09-25-06:48:24: 2024-09-25-06:48:24: [[0;32m OK [0m] Finished [0;1;39mmodprobe@loop.service[0m - Load Kernel Module loop. 2024-09-25-06:48:24: 2024-09-25-06:48:24: 2024-09-25-06:48:24: [[0;32m OK [0m] Finished [0;1;39msystemd-modules-load.service[0m - Load Kernel Modules. 2024-09-25-06:48:24: 2024-09-25-06:48:24: 2024-09-25-06:48:24: [[0;32m OK [0m] Finished [0;1;39msystemd-network-generator…[0mork units from Kernel command line. 2024-09-25-06:48:24: 2024-09-25-06:48:24: 2024-09-25-06:48:24: [[0;32m OK [0m] Finished [0;1;39msystemd-remount-fs.servic[0mmount Root and Kernel File Systems. 2024-09-25-06:48:24: 2024-09-25-06:48:24: 2024-09-25-06:48:24: [[0;32m OK [0m] Reached target [0;1;39mswap.target[0m - Swaps. 2024-09-25-06:48:24: 2024-09-25-06:48:24: 2024-09-25-06:48:24: Mounting [0;1;39msys-fs-fuse-connections.mount[0m - FUSE Control File System... 2024-09-25-06:48:24: 2024-09-25-06:48:24: 2024-09-25-06:48:24: [0;21;39msystemd[1]: Starting systemd-remount-fs.service - Remount Root and Kernel File Systems...[0m 2024-09-25-06:48:24: [0;21;39msystemd[1]: systemd-tpm2-setup-early.service - Early TPM SRK Setup was skipped because of an unmet condition check (ConditionSecurity=measured-uki).[0m 2024-09-25-06:48:24: [0;21;39msystemd[1]: Starting systemd-udev-load-credentials.service - Load udev Rules from Credentials...[0m 2024-09-25-06:48:24: [0;21;33mwatchdog: BUG: soft lockup - CPU#2 stuck for 24s! [systemd:1][0m 2024-09-25-06:48:24: [0;21;33mModules linked in: fuse loop nfnetlink xfs nvme_tcp nvme_keyring sd_mod nvme_fabrics ahci nvme_core bochs libahci drm_vram_helper crct10dif_pclmul crc32_pclmul crc32c_intel nvme_auth virtio_net drm_ttm_helper libata net_failover ttm ghash_clmulni_intel virtio_scsi failover serio_raw dm_mirror dm_region_hash dm_log be2iscsi iscsi_boot_sysfs cxgb4i cxgb4 tls libcxgbi libcxgb iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi dm_multipath dm_mod[0m 2024-09-25-06:48:24: [0;21;33mCPU: 2 UID: 0 PID: 1 Comm: systemd Not tainted 6.11.0-25.el10.x86_64 #1[0m 2024-09-25-06:48:24: [0;21;33mHardware name: Red Hat KVM/RHEL, BIOS edk2-20240524-6.el10 05/24/2024[0m 2024-09-25-06:48:24: [0;21;33mRIP: 0010:_raw_spin_unlock_irqrestore+0x1f/0x30[0m 2024-09-25-06:48:24: [0;21;33mCode: 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 0f 1f 44 00 00 e8 82 07 00 00 90 f7 c6 00 02 00 00 74 01 fb 65 ff 0d 89 d0 c3 73 <74> 05 c3 cc cc cc cc 0f 1f 44 00 00 c3 cc cc cc cc 90 90 90 90 90[0m 2024-09-25-06:48:24: [0;21;33mRSP: 0018:ffffa5a24001b8f8 EFLAGS: 00000286[0m 2024-09-25-06:48:24: [0;21;33mRAX: 0000000000000001 RBX: ffffffff8e5bd060 RCX: 0000000000000000[0m 2024-09-25-06:48:24: [0;21;33mRDX: 0000000000000002 RSI: 0000000000000246 RDI: ffffffff8e5bd060[0m 2024-09-25-06:48:24: [0;21;33mRBP: ffff8efbaf943100 R08: 0000000000000002 R09: 0000000000000001[0m 2024-09-25-06:48:24: [0;21;33mR10: ffffffff8d721be8 R11: 0000000000000003 R12: ffff8efb80294d40[0m 2024-09-25-06:48:24: [0;21;33mR13: 0000000000001002 R14: ffff8efb811b0000 R15: ffff8efb81593750[0m 2024-09-25-06:48:24: [0;21;33mFS: 00007f94dc448440(0000) GS:ffff8efbfbd00000(0000) knlGS:0000000000000000[0m 2024-09-25-06:48:24: [0;21;33mCS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033[0m 2024-09-25-06:48:24: [0;21;33mCR2: 00007f0a1bc51080 CR3: 00000001026c6003 CR4: 0000000000170ef0[0m 2024-09-25-06:48:24: [0;21;33mCall Trace:[0m 2024-09-25-06:48:24: [0;21;33m <IRQ>[0m 2024-09-25-06:48:24: [0;21;33m ? show_trace_log_lvl+0x1b0/0x2f0[0m 2024-09-25-06:48:24: [0;21;33m ? show_trace_log_lvl+0x1b0/0x2f0[0m 2024-09-25-06:48:24: [0;21;33m ? uart_dtr_rts+0x86/0x90[0m 2024-09-25-06:48:24: [0;21;33m ? watchdog_timer_fn.cold+0x3d/0xa0[0m 2024-09-25-06:48:24: [0;21;33m ? __pfx_watchdog_timer_fn+0x10/0x10[0m 2024-09-25-06:48:24: [0;21;33m ? __hrtimer_run_queues+0x13c/0x2a0[0m 2024-09-25-06:48:24: [0;21;33m ? hrtimer_interrupt+0xfa/0x210[0m 2024-09-25-06:48:24: [0;21;33m ? __sysvec_apic_timer_interrupt+0x55/0x100[0m 2024-09-25-06:48:24: [0;21;33m ? sysvec_apic_timer_interrupt+0x6c/0x90[0m 2024-09-25-06:48:24: [0;21;33m </IRQ>[0m 2024-09-25-06:48:24: [0;21;33m <TASK>[0m 2024-09-25-06:48:24: [0;21;33m ? asm_sysvec_apic_timer_interrupt+0x1a/0x20[0m 2024-09-25-06:48:24: [0;21;33m ? _raw_spin_unlock_irqrestore+0x1f/0x30[0m 2024-09-25-06:48:24: [0;21;33m ? _raw_spin_unlock_irqrestore+0xe/0x30[0m 2024-09-25-06:48:24: [0;21;33m uart_dtr_rts+0x86/0x90[0m 2024-09-25-06:48:24: [0;21;33m tty_port_block_til_ready+0x27d/0x330[0m 2024-09-25-06:48:24: [0;21;33m ? __pfx_autoremove_wake_function+0x10/0x10[0m 2024-09-25-06:48:24: [0;21;33m uart_open+0x1e/0x30[0m 2024-09-25-06:48:24: [0;21;33m tty_open+0x161/0x6a0[0m 2024-09-25-06:48:24: [0;21;33m chrdev_open+0xcb/0x240[0m 2024-09-25-06:48:24: [0;21;33m ? __pfx_chrdev_open+0x10/0x10[0m 2024-09-25-06:48:24: [0;21;33m do_dentry_open+0x25a/0x490[0m 2024-09-25-06:48:24: [0;21;33m vfs_open+0x34/0xf0[0m 2024-09-25-06:48:24: [0;21;33m do_open+0x165/0x3d0[0m 2024-09-25-06:48:24: [0;21;33m path_openat+0x124/0x2d0[0m 2024-09-25-06:48:24: [0;21;33m ? pick_next_task+0x50/0xb10[0m 2024-09-25-06:48:24: [0;21;33m do_filp_open+0xc4/0x170[0m 2024-09-25-06:48:24: [0;21;33m do_sys_openat2+0xae/0xe0[0m 2024-09-25-06:48:24: [0;21;33m __x64_sys_openat+0x55/0xa0[0m 2024-09-25-06:48:24: [0;21;33m do_syscall_64+0x7d/0x160[0m 2024-09-25-06:48:24: [0;21;33m ? __do_sys_newfstat+0x68/0x70[0m 2024-09-25-06:48:24: [0;21;33m ? __memcg_slab_free_hook+0x100/0x150[0m 2024-09-25-06:48:24: [0;21;33m ? __x64_sys_close+0x3c/0x80[0m 2024-09-25-06:48:24: [0;21;33m ? kmem_cache_free+0x3d5/0x400[0m 2024-09-25-06:48:24: [0;21;33m ? syscall_exit_to_user_mode+0x10/0x1f0[0m 2024-09-25-06:48:24: [0;21;33m ? do_syscall_64+0x89/0x160[0m 2024-09-25-06:48:24: [0;21;33m ? filp_flush+0x56/0x70[0m 2024-09-25-06:48:24: [0;21;33m ? syscall_exit_to_user_mode+0x10/0x1f0[0m 2024-09-25-06:48:24: [0;21;33m ? do_syscall_64+0x89/0x160[0m 2024-09-25-06:48:24: [0;21;33m ? syscall_exit_to_user_mode+0x10/0x1f0[0m 2024-09-25-06:48:24: [0;21;33m ? do_syscall_64+0x89/0x160[0m 2024-09-25-06:48:24: [0;21;33m ? do_syscall_64+0x89/0x160[0m 2024-09-25-06:48:24: [0;21;33m ? exc_page_fault+0x73/0x160[0m 2024-09-25-06:48:24: [0;21;33m entry_SYSCALL_64_after_hwframe+0x76/0x7e[0m 2024-09-25-06:48:24: [0;21;33mRIP: 0033:0x7f94dbb20210[0m 2024-09-25-06:48:24: [0;21;33mCode: 83 e2 40 75 55 89 f0 f7 d0 a9 00 00 41 00 74 4a 80 3d 43 de 0d 00 00 74 6e 89 da 48 89 ee bf 9c ff ff ff b8 01 01 00 00 0f 05 <48> 3d 00 f0 ff ff 0f 87 94 00 00 00 48 8b 54 24 28 64 48 2b 14 25[0m 2024-09-25-06:48:24: [0;21;33mRSP: 002b:00007ffc30704f80 EFLAGS: 00000202 ORIG_RAX: 0000000000000101[0m 2024-09-25-06:48:24: [0;21;33mRAX: ffffffffffffffda RBX: 0000000000080101 RCX: 00007f94dbb20210[0m 2024-09-25-06:48:24: [0;21;33mRDX: 0000000000080101 RSI: 00007f94dc37a5d6 RDI: 00000000ffffff9c[0m 2024-09-25-06:48:24: [0;21;33mRBP: 00007f94dc37a5d6 R08: 0000000000000073 R09: 00000000ffffffff[0m 2024-09-25-06:48:24: [0;21;33mR10: 0000000000000000 R11: 0000000000000202 R12: 0000000000080101[0m 2024-09-25-06:48:24: [0;21;33mR13: 00007f94dc37a5d6 R14: 00007ffc30705000 R15: 000055f8c591def0[0m 2024-09-25-06:48:24: [0;21;33m </TASK>[0m 2024-09-25-06:48:24: [0;21;33mwatchdog: BUG: soft lockup - CPU#0 stuck for 24s! [plymouthd:618][0m 2024-09-25-06:48:24: [0;21;33mModules linked in: fuse loop nfnetlink xfs nvme_tcp nvme_keyring sd_mod nvme_fabrics ahci nvme_core bochs libahci drm_vram_helper crct10dif_pclmul crc32_pclmul crc32c_intel nvme_auth virtio_net drm_ttm_helper libata net_failover ttm ghash_clmulni_intel virtio_scsi failover serio_raw dm_mirror dm_region_hash dm_log be2iscsi iscsi_boot_sysfs cxgb4i cxgb4 tls libcxgbi libcxgb iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi dm_multipath dm_mod[0m 2024-09-25-06:48:24: [0;21;33mCPU: 0 UID: 0 PID: 618 Comm: plymouthd Tainted: G L ------- --- 6.11.0-25.el10.x86_64 #1[0m 2024-09-25-06:48:24: [0;21;33mTainted: [L]=SOFTLOCKUP[0m 2024-09-25-06:48:24: [0;21;33mHardware name: Red Hat KVM/RHEL, BIOS edk2-20240524-6.el10 05/24/2024[0m 2024-09-25-06:48:24: [0;21;33mRIP: 0010:_raw_spin_unlock_irqrestore+0x1f/0x30[0m 2024-09-25-06:48:24: [0;21;33mCode: 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 0f 1f 44 00 00 e8 82 07 00 00 90 f7 c6 00 02 00 00 74 01 fb 65 ff 0d 89 d0 c3 73 <74> 05 c3 cc cc cc cc 0f 1f 44 00 00 c3 cc cc cc cc 90 90 90 90 90[0m 2024-09-25-06:48:24: [0;21;33mRSP: 0018:ffffa5a2403a3be0 EFLAGS: 00000286[0m 2024-09-25-06:48:24: [0;21;33mRAX: 0000000000000001 RBX: 0000000000000e75 RCX: 0000000000000000[0m 2024-09-25-06:48:24: [0;21;33mRDX: 0000000000000000 RSI: 0000000000000286 RDI: ffffffff8e5bd060[0m 2024-09-25-06:48:24: [0;21;33mRBP: ffffffff8e5bd060 R08: 0000000000000000 R09: 0000000000000001[0m 2024-09-25-06:48:24: [0;21;33mR10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000286[0m 2024-09-25-06:48:24: [0;21;33mR13: ffffa5a2403d1000 R14: ffff8efb815ec000 R15: 00000000000000a7[0m 2024-09-25-06:48:24: [0;21;33mFS: 00007fd043521280(0000) GS:ffff8efbfbc00000(0000) knlGS:0000000000000000[0m 2024-09-25-06:48:24: [0;21;33mCS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033[0m 2024-09-25-06:48:24: [0;21;33mCR2: 00007f32e1aee080 CR3: 00000001137b8004 CR4: 0000000000170ef0[0m 2024-09-25-06:48:24: [0;21;33mCall Trace:[0m 2024-09-25-06:48:24: [0;21;33m <IRQ>[0m 2024-09-25-06:48:24: [0;21;33m ? show_trace_log_lvl+0x1b0/0x2f0[0m 2024-09-25-06:48:24: [0;21;33m ? show_trace_log_lvl+0x1b0/0x2f0[0m 2024-09-25-06:48:24: [0;21;33m ? uart_write_room+0x74/0x190[0m 2024-09-25-06:48:24: [0;21;33m ? watchdog_timer_fn.cold+0x3d/0xa0[0m 2024-09-25-06:48:24: [0;21;33m ? __pfx_watchdog_timer_fn+0x10/0x10[0m 2024-09-25-06:48:24: [0;21;33m ? __hrtimer_run_queues+0x13c/0x2a0[0m 2024-09-25-06:48:24: [0;21;33m ? hrtimer_interrupt+0xfa/0x210[0m 2024-09-25-06:48:24: [0;21;33m ? __sysvec_apic_timer_interrupt+0x55/0x100[0m 2024-09-25-06:48:24: [0;21;33m ? sysvec_apic_timer_interrupt+0x6c/0x90[0m 2024-09-25-06:48:24: [0;21;33m </IRQ>[0m 2024-09-25-06:48:24: [0;21;33m <TASK>[0m 2024-09-25-06:48:24: [0;21;33m ? asm_sysvec_apic_timer_interrupt+0x1a/0x20[0m 2024-09-25-06:48:24: [0;21;33m ? _raw_spin_unlock_irqrestore+0x1f/0x30[0m 2024-09-25-06:48:24: [0;21;33m uart_write_room+0x74/0x190[0m 2024-09-25-06:48:24: [0;21;33m process_output_block+0x3a/0x1c0[0m 2024-09-25-06:48:24: [0;21;33m ? add_wait_queue+0x7e/0xb0[0m 2024-09-25-06:48:24: [0;21;33m n_tty_write+0x1a4/0x3c0[0m 2024-09-25-06:48:24: [0;21;33m ? __pfx_woken_wake_function+0x10/0x10[0m 2024-09-25-06:48:24: [0;21;33m iterate_tty_write+0x114/0x240[0m 2024-09-25-06:48:24: [0;21;33m file_tty_write.isra.0+0x86/0xb0[0m 2024-09-25-06:48:24: [0;21;33m vfs_write+0x294/0x460[0m 2024-09-25-06:48:24: [0;21;33m ksys_write+0x6d/0xf0[0m 2024-09-25-06:48:24: [0;21;33m do_syscall_64+0x7d/0x160[0m 2024-09-25-06:48:24: [0;21;33m ? syscall_exit_to_user_mode+0x10/0x1f0[0m 2024-09-25-06:48:24: [0;21;33m ? do_syscall_64+0x89/0x160[0m 2024-09-25-06:48:24: [0;21;33m ? ksys_read+0x6d/0xf0[0m 2024-09-25-06:48:24: [0;21;33m ? syscall_exit_to_user_mode+0x10/0x1f0[0m 2024-09-25-06:48:24: [0;21;33m ? do_syscall_64+0x89/0x160[0m 2024-09-25-06:48:24: [0;21;33m entry_SYSCALL_64_after_hwframe+0x76/0x7e[0m 2024-09-25-06:48:24: [0;21;33mRIP: 0033:0x7fd0437ac6a4[0m 2024-09-25-06:48:24: [0;21;33mCode: c7 00 16 00 00 00 b8 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 80 3d a5 c9 0d 00 00 74 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 48 83 ec 28 48 89 54 24 18 48[0m 2024-09-25-06:48:24: [0;21;33mRSP: 002b:00007ffeb2d85198 EFLAGS: 00000202 ORIG_RAX: 0000000000000001[0m 2024-09-25-06:48:24: [0;21;33mRAX: ffffffffffffffda RBX: 0000563fd8c56110 RCX: 00007fd0437ac6a4[0m 2024-09-25-06:48:24: [0;21;33mRDX: 00000000000000a7 RSI: 0000563fd8c81f60 RDI: 000000000000000a[0m 2024-09-25-06:48:24: [0;21;33mRBP: 00000000000000a7 R08: 0000563fd8c844c0 R09: 00000000000000d0[0m 2024-09-25-06:48:24: [0;21;33mR10: 0000000000000009 R11: 0000000000000202 R12: 0000563fd8c890c0[0m 2024-09-25-06:48:24: [0;21;33mR13: 0000563fd8c80640 R14: 0000563fcc03e4c2 R15: 0000563fd8c89180[0m 2024-09-25-06:48:24: [0;21;33m </TASK>[0m 2024-09-25-06:48:24: [0;21;33mwatchdog: BUG: soft lockup - CPU#1 stuck for 24s! [kworker/1:2:301][0m 2024-09-25-06:48:24: [0;21;33mModules linked in: fuse loop nfnetlink xfs nvme_tcp nvme_keyring sd_mod nvme_fabrics ahci nvme_core bochs libahci drm_vram_helper crct10dif_pclmul crc32_pclmul crc32c_intel nvme_auth virtio_net drm_ttm_helper libata net_failover ttm ghash_clmulni_intel virtio_scsi failover serio_raw dm_mirror dm_region_hash dm_log be2iscsi iscsi_boot_sysfs cxgb4i cxgb4 tls libcxgbi libcxgb iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi dm_multipath dm_mod[0m 2024-09-25-06:48:24: [0;21;33mCPU: 1 UID: 0 PID: 301 Comm: kworker/1:2 Tainted: G L ------- --- 6.11.0-25.el10.x86_64 #1[0m 2024-09-25-06:48:24: [0;21;33mTainted: [L]=SOFTLOCKUP[0m 2024-09-25-06:48:24: [0;21;33mHardware name: Red Hat KVM/RHEL, BIOS edk2-20240524-6.el10 05/24/2024[0m 2024-09-25-06:48:24: [0;21;33mWorkqueue: pm pm_runtime_work[0m 2024-09-25-06:48:24: [0;21;33mRIP: 0010:_raw_spin_unlock_irqrestore+0x1f/0x30[0m 2024-09-25-06:48:24: [0;21;33mCode: 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 0f 1f 44 00 00 e8 82 07 00 00 90 f7 c6 00 02 00 00 74 01 fb 65 ff 0d 89 d0 c3 73 <74> 05 c3 cc cc cc cc 0f 1f 44 00 00 c3 cc cc cc cc 90 90 90 90 90[0m 2024-09-25-06:48:24: [0;21;33mRSP: 0018:ffffa5a2405c3d50 EFLAGS: 00000286[0m 2024-09-25-06:48:24: [0;21;33mRAX: 0000000000000001 RBX: ffffffff8e5bd060 RCX: 0000000000000000[0m 2024-09-25-06:48:24: [0;21;33mRDX: 0000000000000001 RSI: 0000000000000246 RDI: ffffffff8e5bd060[0m 2024-09-25-06:48:24: [0;21;33mRBP: ffff8efb838ecc00 R08: 0000000000000001 R09: 0000000000000001[0m 2024-09-25-06:48:24: [0;21;33mR10: 0000000000000003 R11: 0000000000000000 R12: 0000000000000246[0m 2024-09-25-06:48:24: [0;21;33mR13: 0000000000000001 R14: 0000000000000000 R15: ffff8efb901499c0[0m 2024-09-25-06:48:24: [0;21;33mFS: 0000000000000000(0000) GS:ffff8efbfbc80000(0000) knlGS:0000000000000000[0m 2024-09-25-06:48:24: [0;21;33mCS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033[0m 2024-09-25-06:48:24: [0;21;33mCR2: 00005618866fa9d0 CR3: 00000001113c4005 CR4: 0000000000170ef0[0m 2024-09-25-06:48:24: [0;21;33mCall Trace:[0m 2024-09-25-06:48:24: [0;21;33m <IRQ>[0m 2024-09-25-06:48:24: [0;21;33m ? show_trace_log_lvl+0x1b0/0x2f0[0m 2024-09-25-06:48:24: [0;21;33m ? show_trace_log_lvl+0x1b0/0x2f0[0m 2024-09-25-06:48:24: [0;21;33m ? serial_port_runtime_suspend+0x12b/0x230[0m 2024-09-25-06:48:24: [0;21;33m ? watchdog_timer_fn.cold+0x3d/0xa0[0m 2024-09-25-06:48:24: [0;21;33m ? __pfx_watchdog_timer_fn+0x10/0x10[0m 2024-09-25-06:48:24: [0;21;33m ? __hrtimer_run_queues+0x13c/0x2a0[0m 2024-09-25-06:48:24: [0;21;33m ? hrtimer_interrupt+0xfa/0x210[0m 2024-09-25-06:48:24: [0;21;33m ? __sysvec_apic_timer_interrupt+0x55/0x100[0m 2024-09-25-06:48:24: [0;21;33m ? sysvec_apic_timer_interrupt+0x6c/0x90[0m 2024-09-25-06:48:24: [0;21;33m </IRQ>[0m 2024-09-25-06:48:24: [0;21;33m <TASK>[0m 2024-09-25-06:48:24: [0;21;33m ? asm_sysvec_apic_timer_interrupt+0x1a/0x20[0m 2024-09-25-06:48:24: [0;21;33m ? _raw_spin_unlock_irqrestore+0x1f/0x30[0m 2024-09-25-06:48:24: [0;21;33m serial_port_runtime_suspend+0x12b/0x230[0m 2024-09-25-06:48:24: [0;21;33m ? __pfx_serial_port_runtime_suspend+0x10/0x10[0m 2024-09-25-06:48:24: [0;21;33m __rpm_callback+0x44/0x170[0m 2024-09-25-06:48:24: [0;21;33m ? __pfx_serial_port_runtime_suspend+0x10/0x10[0m 2024-09-25-06:48:24: [0;21;33m rpm_callback+0x55/0x60[0m 2024-09-25-06:48:24: [0;21;33m ? __pfx_serial_port_runtime_suspend+0x10/0x10[0m 2024-09-25-06:48:24: [0;21;33m rpm_suspend+0xe6/0x5f0[0m 2024-09-25-06:48:24: [0;21;33m ? _raw_spin_unlock+0xe/0x30[0m 2024-09-25-06:48:24: [0;21;33m ? finish_task_switch.isra.0+0x99/0x2b0[0m 2024-09-25-06:48:24: [0;21;33m pm_runtime_work+0x98/0xb0[0m 2024-09-25-06:48:24: [0;21;33m process_one_work+0x177/0x330[0m 2024-09-25-06:48:24: [0;21;33m worker_thread+0x252/0x390[0m 2024-09-25-06:48:24: [0;21;33m ? __pfx_worker_thread+0x10/0x10[0m 2024-09-25-06:48:24: [0;21;33m kthread+0xd2/0x100[0m 2024-09-25-06:48:24: [0;21;33m ? __pfx_kthread+0x10/0x10[0m 2024-09-25-06:48:24: [0;21;33m ret_from_fork+0x34/0x50[0m 2024-09-25-06:48:24: [0;21;33m ? __pfx_kthread+0x10/0x10[0m 2024-09-25-06:48:24: [0;21;33m ret_from_fork_asm+0x1a/0x30[0m 2024-09-25-06:48:24: [0;21;33m </TASK>[0m 2024-09-25-06:48:24: [0;21;39msystemd-journald[916]: Collecting audit messages is disabled.[0m 2024-09-25-06:48:24: [0;21;39msystemd[1]: Starting systemd-udev-trigger.service - Coldplug All udev Devices...[0m 2024-09-25-06:48:24: [0;21;39msystemd[1]: Activated swap dev-disk-by\x2duuid-794cb060\x2d21a8\x2d434a\x2d91e6\x2d83e0369e1296.swap - /dev/disk/by-uuid/794cb060-21a8-434a-91e6-83e0369e1296.[0m 2024-09-25-06:48:24: [0;21;39msystemd[1]: Started systemd-journald.service - Journal Service.[0m
For case2:
Boot the VM also with '-S', cont VM and then start migration:
Sending qmp command : {"execute": "migrate-incoming", "arguments": {"uri": "tcp:[::]:4000"}, "id": "R0NhJlXA"}Responding qmp command: {"return": {}, "id": "R0NhJlXA"} Sending qmp command : {"execute": "cont", "id": "ByzCEP23"} Responding qmp command: {"return": {}, "id": "ByzCEP23"} onds": 1727269941, "microseconds": 612960}, "event": "RTC_CHANGE", "data": {"offset": -1, "qom-path": "/machine/unattached/device[6]/rtc"}} Sending qmp command : {"execute": "query-status", "id": "n5TqS8U3"} Responding qmp command: {"return": {"status": "running", "running": true}, "id": "n5TqS8U3"} Sending qmp command : {"execute": "query-version", "id": "RWJZBqxm"} Responding qmp command: {"return": {"qemu": {"micro": 0, "minor": 1, "major": 9}, "package": "qemu-kvm-9.1.0-1.el10"}, "id": "RWJZBqxm"} Sending qmp command : {"execute": "migrate", "arguments": {"uri": "tcp:10.72.136.64:4000", "detach": true, "resume": false}, "id": "Dsk5HErl"}
After migration, found VM call trace:
2024-09-25-09:16:08: [ 163.777468] watchdog: BUG: soft lockup - CPU#3 stuck for 37s! [khugepaged:57] 2024-09-25-09:16:08: [ 163.777503] Modules linked in: nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 rfkill nf_tables qrtr vfat fat intel_rapl_msr intel_rapl_common kvm_intel kvm iTCO_wdt iTCO_vendor_support rapl pcspkr lpc_ich virtio_balloon i2c_i801 i2c_smbus joydev sg fuse loop nfnetlink xfs nvme_tcp ahci crct10dif_pclmul bochs drm_vram_helper libahci nvme_keyring crc32_pclmul sd_mod nvme_fabrics drm_ttm_helper crc32c_intel nvme_core virtio_scsi virtio_net ttm libata net_failover ghash_clmulni_intel failover nvme_auth serio_raw dm_mirror dm_region_hash dm_log be2iscsi iscsi_boot_sysfs cxgb4i cxgb4 tls libcxgbi libcxgb iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi dm_multipath dm_mod 2024-09-25-09:16:08: [ 163.777595] CPU: 3 UID: 0 PID: 57 Comm: khugepaged Kdump: loaded Not tainted 6.11.0-25.el10.x86_64 #1 2024-09-25-09:16:08: [ 163.777602] Hardware name: Red Hat KVM/RHEL, BIOS edk2-20240524-6.el10 05/24/2024 2024-09-25-09:16:08: [ 163.777607] RIP: 0010:smp_call_function_many_cond+0x112/0x4c0 2024-09-25-09:16:08: [ 163.777653] Code: d0 e8 e2 b1 4a 00 3b 05 4c 2e 1a 02 73 26 48 63 d0 49 8b 34 24 48 03 34 d5 60 7d b3 85 8b 56 08 83 e2 01 74 0a f3 90 8b 4e 08 <83> e1 01 75 f6 83 c0 01 eb bf 48 83 c4 30 5b 5d 41 5c 41 5d 41 5e 2024-09-25-09:16:08: [ 163.777655] RSP: 0018:ffffb9e0001f7c18 EFLAGS: 00000202 2024-09-25-09:16:08: [ 163.777658] RAX: 0000000000000001 RBX: 0000000000000000 RCX: 0000000000000011 2024-09-25-09:16:08: [ 163.777663] RDX: 0000000000000001 RSI: ffff9ea7fbcbd300 RDI: ffff9ea780054500 2024-09-25-09:16:08: [ 163.777665] RBP: 0000000000000000 R08: 0000000000000001 R09: 00000000fffffffc 2024-09-25-09:16:08: [ 163.777667] R10: ffff9ea7800542d0 R11: 0000000000000000 R12: ffff9ea7fbdb6c40 2024-09-25-09:16:08: [ 163.777668] R13: ffff9ea7800542d0 R14: 0000000000000003 R15: 0000000000000000 2024-09-25-09:16:08: [ 163.777670] FS: 0000000000000000(0000) GS:ffff9ea7fbd80000(0000) knlGS:0000000000000000 2024-09-25-09:16:08: [ 163.777675] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 2024-09-25-09:16:08: [ 163.777690] CR2: 00007f20d4c0a0f8 CR3: 0000000063222004 CR4: 0000000000170ef0 2024-09-25-09:16:08: [ 163.777697] Call Trace: 2024-09-25-09:16:08: [ 163.777706] <IRQ> 2024-09-25-09:16:08: [ 163.777709] ? show_trace_log_lvl+0x1b0/0x2f0 2024-09-25-09:16:08: [ 163.777740] ? show_trace_log_lvl+0x1b0/0x2f0 2024-09-25-09:16:08: [ 163.777762] ? smp_call_function+0x2c/0x50 2024-09-25-09:16:08: [ 163.777766] ? watchdog_timer_fn.cold+0x3d/0xa0 2024-09-25-09:16:08: [ 163.777777] ? __pfx_watchdog_timer_fn+0x10/0x10 2024-09-25-09:16:08: [ 163.777795] ? __hrtimer_run_queues+0x13c/0x2a0 2024-09-25-09:16:08: [ 163.777810] ? hrtimer_interrupt+0xfa/0x210 2024-09-25-09:16:08: [ 163.777813] ? __sysvec_apic_timer_interrupt+0x55/0x100 2024-09-25-09:16:08: [ 163.777830] ? sysvec_apic_timer_interrupt+0x6c/0x90 2024-09-25-09:16:08: [ 163.777856] </IRQ> 2024-09-25-09:16:08: [ 163.777857] <TASK> 2024-09-25-09:16:08: [ 163.777858] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 2024-09-25-09:16:08: [ 163.777875] ? smp_call_function_many_cond+0x112/0x4c0 2024-09-25-09:16:08: [ 163.777879] ? smp_call_function_many_cond+0xee/0x4c0 2024-09-25-09:16:08: [ 163.777882] ? __pfx_tlb_remove_table_smp_sync+0x10/0x10 2024-09-25-09:16:08: [ 163.777909] smp_call_function+0x2c/0x50 2024-09-25-09:16:08: [ 163.777913] collapse_huge_page+0x462/0x790 2024-09-25-09:16:08: [ 163.777923] hpage_collapse_scan_pmd+0x659/0x710 2024-09-25-09:16:08: [ 163.777927] khugepaged_scan_mm_slot.constprop.0+0x3ce/0x580 2024-09-25-09:16:08: [ 163.777931] khugepaged+0x10c/0x200 2024-09-25-09:16:08: [ 163.777934] ? __pfx_khugepaged+0x10/0x10 2024-09-25-09:16:08: [ 163.777936] kthread+0xd2/0x100 2024-09-25-09:16:08: [ 163.777950] ? __pfx_kthread+0x10/0x10 2024-09-25-09:16:08: [ 163.777952] ret_from_fork+0x34/0x50 2024-09-25-09:16:08: [ 163.777958] ? __pfx_kthread+0x10/0x10 2024-09-25-09:16:08: [ 163.777959] ret_from_fork_asm+0x1a/0x30 2024-09-25-09:16:08: [ 163.777972] </TASK> 2024-09-25-09:16:08: [ 164.152020] rfkill: input handler disabled 2024-09-25-09:16:08: [ 200.134938] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [kworker/3:3:667] 2024-09-25-09:16:08: [ 200.135025] Modules linked in: nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 rfkill nf_tables qrtr vfat fat intel_rapl_msr intel_rapl_common kvm_intel kvm iTCO_wdt iTCO_vendor_support rapl pcspkr lpc_ich virtio_balloon i2c_i801 i2c_smbus joydev sg fuse loop nfnetlink xfs nvme_tcp ahci crct10dif_pclmul bochs drm_vram_helper libahci nvme_keyring crc32_pclmul sd_mod nvme_fabrics drm_ttm_helper crc32c_intel nvme_core virtio_scsi virtio_net ttm libata net_failover ghash_clmulni_intel failover nvme_auth serio_raw dm_mirror dm_region_hash dm_log be2iscsi iscsi_boot_sysfs cxgb4i cxgb4 tls libcxgbi libcxgb iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi dm_multipath dm_mod 2024-09-25-09:16:08: [ 200.135524] CPU: 3 UID: 0 PID: 667 Comm: kworker/3:3 Kdump: loaded Tainted: G L ------- --- 6.11.0-25.el10.x86_64 #1 2024-09-25-09:16:08: [ 200.135534] Tainted: [L]=SOFTLOCKUP 2024-09-25-09:16:08: [ 200.135541] Hardware name: Red Hat KVM/RHEL, BIOS edk2-20240524-6.el10 05/24/2024 2024-09-25-09:16:08: [ 200.135551] Workqueue: events netstamp_clear 2024-09-25-09:16:08: [ 200.135612] RIP: 0010:smp_call_function_many_cond+0x10f/0x4c0 2024-09-25-09:16:08: [ 200.135638] Code: 08 48 63 d0 e8 e2 b1 4a 00 3b 05 4c 2e 1a 02 73 26 48 63 d0 49 8b 34 24 48 03 34 d5 60 7d b3 85 8b 56 08 83 e2 01 74 0a f3 90 <8b> 4e 08 83 e1 01 75 f6 83 c0 01 eb bf 48 83 c4 30 5b 5d 41 5c 41 2024-09-25-09:16:08: [ 200.135641] RSP: 0018:ffffb9e00050fd68 EFLAGS: 00000202 2024-09-25-09:16:08: [ 200.135647] RAX: 0000000000000001 RBX: 0000000000000246 RCX: 0000000000000001 2024-09-25-09:16:08: [ 200.135651] RDX: 0000000000000001 RSI: ffff9ea7fbcbd300 RDI: ffff9ea780054500 2024-09-25-09:16:08: [ 200.135653] RBP: 0000000000000000 R08: 0000000000000001 R09: 00000000fffffffc 2024-09-25-09:16:08: [ 200.135654] R10: ffff9ea7800542d0 R11: 0000000000000000 R12: ffff9ea7fbdb6c40 2024-09-25-09:16:08: [ 200.135656] R13: ffff9ea7800542d0 R14: 0000000000000003 R15: 0000000000000001 2024-09-25-09:16:08: [ 200.135658] FS: 0000000000000000(0000) GS:ffff9ea7fbd80000(0000) knlGS:0000000000000000 2024-09-25-09:16:08: [ 200.135662] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 2024-09-25-09:16:08: [ 200.135672] CR2: 0000561403926c98 CR3: 0000000063222006 CR4: 0000000000170ef0 2024-09-25-09:16:08: [ 200.135680] Call Trace: 2024-09-25-09:16:08: [ 200.135695] <IRQ> 2024-09-25-09:16:08: [ 200.135745] ? show_trace_log_lvl+0x1b0/0x2f0 2024-09-25-09:16:08: [ 200.135777] ? show_trace_log_lvl+0x1b0/0x2f0 2024-09-25-09:16:08: [ 200.135781] ? on_each_cpu_cond_mask+0x24/0x40 2024-09-25-09:16:08: [ 200.135783] ? watchdog_timer_fn.cold+0x3d/0xa0 2024-09-25-09:16:08: [ 200.135790] ? __pfx_watchdog_timer_fn+0x10/0x10 2024-09-25-09:16:08: [ 200.135805] ? __hrtimer_run_queues+0x13c/0x2a0 2024-09-25-09:16:08: [ 200.135817] ? hrtimer_interrupt+0xfa/0x210 2024-09-25-09:16:08: [ 200.135819] ? __sysvec_apic_timer_interrupt+0x55/0x100 2024-09-25-09:16:08: [ 200.135836] ? sysvec_apic_timer_interrupt+0x6c/0x90 2024-09-25-09:16:08: [ 200.135853] </IRQ> 2024-09-25-09:16:08: [ 200.135854] <TASK> 2024-09-25-09:16:08: [ 200.135855] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 2024-09-25-09:16:08: [ 200.135899] ? smp_call_function_many_cond+0x10f/0x4c0 2024-09-25-09:16:08: [ 200.135902] ? smp_call_function_many_cond+0xee/0x4c0 2024-09-25-09:16:08: [ 200.135906] ? __pfx_do_sync_core+0x10/0x10 2024-09-25-09:16:08: [ 200.135922] on_each_cpu_cond_mask+0x24/0x40 2024-09-25-09:16:08: [ 200.135924] text_poke_bp_batch+0xc1/0x300 2024-09-25-09:16:08: [ 200.135935] ? arch_jump_label_transform_queue+0x57/0x80 2024-09-25-09:16:08: [ 200.135947] text_poke_finish+0x1f/0x30 2024-09-25-09:16:08: [ 200.135950] arch_jump_label_transform_apply+0x1a/0x30 2024-09-25-09:16:08: [ 200.135958] static_key_enable_cpuslocked+0x63/0x90 2024-09-25-09:16:08: [ 200.135989] static_key_enable+0x1a/0x20 2024-09-25-09:16:08: [ 200.135992] process_one_work+0x177/0x330 2024-09-25-09:16:08: [ 200.136016] worker_thread+0x252/0x390 2024-09-25-09:16:08: [ 200.136020] ? __pfx_worker_thread+0x10/0x10 2024-09-25-09:16:08: [ 200.136023] kthread+0xd2/0x100 2024-09-25-09:16:08: [ 200.136033] ? __pfx_kthread+0x10/0x10 2024-09-25-09:16:08: [ 200.136035] ret_from_fork+0x34/0x50 2024-09-25-09:16:08: [ 200.136040] ? __pfx_kthread+0x10/0x10 2024-09-25-09:16:08: [ 200.136042] ret_from_fork_asm+0x1a/0x30 2024-09-25-09:16:08: [ 200.136059] </TASK> 2024-09-25-09:16:08: [ 207.174682] watchdog: BUG: soft lockup - CPU#0 stuck for 25s! [irqbalance:1109] 2024-09-25-09:16:08: [ 207.174711] Modules linked in: nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 rfkill nf_tables qrtr vfat fat intel_rapl_msr intel_rapl_common kvm_intel kvm iTCO_wdt iTCO_vendor_support rapl pcspkr lpc_ich virtio_balloon i2c_i801 i2c_smbus joydev sg fuse loop nfnetlink xfs nvme_tcp ahci crct10dif_pclmul bochs drm_vram_helper libahci nvme_keyring crc32_pclmul sd_mod nvme_fabrics drm_ttm_helper crc32c_intel nvme_core virtio_scsi virtio_net ttm libata net_failover ghash_clmulni_intel failover nvme_auth serio_raw dm_mirror dm_region_hash dm_log be2iscsi iscsi_boot_sysfs cxgb4i cxgb4 tls libcxgbi libcxgb iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi dm_multipath dm_mod 2024-09-25-09:16:08: [ 207.174804] CPU: 0 UID: 0 PID: 1109 Comm: irqbalance Kdump: loaded Tainted: G L ------- --- 6.11.0-25.el10.x86_64 #1 2024-09-25-09:16:08: [ 207.174808] Tainted: [L]=SOFTLOCKUP 2024-09-25-09:16:08: [ 207.174810] Hardware name: Red Hat KVM/RHEL, BIOS edk2-20240524-6.el10 05/24/2024 2024-09-25-09:16:08: [ 207.174812] RIP: 0010:_raw_spin_unlock_irqrestore+0x1f/0x30 2024-09-25-09:16:08: [ 207.174824] Code: 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 0f 1f 44 00 00 e8 82 07 00 00 90 f7 c6 00 02 00 00 74 01 fb 65 ff 0d 89 d0 e3 7a <74> 05 c3 cc cc cc cc 0f 1f 44 00 00 c3 cc cc cc cc 90 90 90 90 90 2024-09-25-09:16:08: [ 207.174826] RSP: 0018:ffffb9e000527cc0 EFLAGS: 00000286 2024-09-25-09:16:08: [ 207.174829] RAX: 0000000000000001 RBX: 0000000000000000 RCX: 00000000000003fb 2024-09-25-09:16:08: [ 207.174831] RDX: ffff9ea78cb23000 RSI: 0000000000000246 RDI: ffff9ea780c9a8a4 2024-09-25-09:16:08: [ 207.174833] RBP: ffff9ea798297b40 R08: 000000000000000f R09: 0000000000000004 2024-09-25-09:16:08: [ 207.174834] R10: ffffffffffffffff R11: 0000000000000000 R12: ffff9ea780c9a800 2024-09-25-09:16:08: [ 207.174854] R13: 0000000000000246 R14: ffff9ea780c9a8a4 R15: 0000000000000000 2024-09-25-09:16:08: [ 207.174856] FS: 00007f9cf329e840(0000) GS:ffff9ea7fbc00000(0000) knlGS:0000000000000000 2024-09-25-09:16:08: [ 207.174861] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 2024-09-25-09:16:08: [ 207.174863] CR2: 00007fd21ad62950 CR3: 000000010ca5e003 CR4: 0000000000170ef0 2024-09-25-09:16:08: [ 207.174865] Call Trace: 2024-09-25-09:16:08: [ 207.174885] <IRQ> 2024-09-25-09:16:08: [ 207.174889] ? show_trace_log_lvl+0x1b0/0x2f0 2024-09-25-09:16:08: [ 207.174894] ? show_trace_log_lvl+0x1b0/0x2f0 2024-09-25-09:16:08: [ 207.174898] ? show_interrupts+0x243/0x2e0 2024-09-25-09:16:08: [ 207.174914] ? watchdog_timer_fn.cold+0x3d/0xa0 2024-09-25-09:16:08: [ 207.174918] ? __pfx_watchdog_timer_fn+0x10/0x10 2024-09-25-09:16:08: [ 207.174921] ? __hrtimer_run_queues+0x13c/0x2a0 2024-09-25-09:16:08: [ 207.174925] ? hrtimer_interrupt+0xfa/0x210 2024-09-25-09:16:08: [ 207.174928] ? __sysvec_apic_timer_interrupt+0x55/0x100 2024-09-25-09:16:08: [ 207.174931] ? sysvec_apic_timer_interrupt+0x6c/0x90 2024-09-25-09:16:08: [ 207.174935] </IRQ> 2024-09-25-09:16:08: [ 207.174936] <TASK> 2024-09-25-09:16:08: [ 207.174937] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 2024-09-25-09:16:08: [ 207.174943] ? _raw_spin_unlock_irqrestore+0x1f/0x30 2024-09-25-09:16:08: [ 207.174945] show_interrupts+0x243/0x2e0 2024-09-25-09:16:08: [ 207.174961] seq_read_iter+0x2b1/0x460 2024-09-25-09:16:08: [ 207.174996] proc_reg_read_iter+0x53/0x90 2024-09-25-09:16:08: [ 207.175022] vfs_read+0x299/0x370 2024-09-25-09:16:08: [ 207.175055] ksys_read+0x6d/0xf0 2024-09-25-09:16:08: [ 207.175059] do_syscall_64+0x7d/0x160 2024-09-25-09:16:08: [ 207.175078] ? syscall_exit_work+0xf3/0x120 2024-09-25-09:16:08: [ 207.175088] ? syscall_exit_to_user_mode+0x10/0x1f0 2024-09-25-09:16:08: [ 207.175091] ? do_syscall_64+0x89/0x160 2024-09-25-09:16:08: [ 207.175093] ? syscall_exit_to_user_mode+0x10/0x1f0 2024-09-25-09:16:08: [ 207.175099] ? do_syscall_64+0x89/0x160 2024-09-25-09:16:08: [ 207.175121] entry_SYSCALL_64_after_hwframe+0x76/0x7e 2024-09-25-09:16:08: [ 207.175130] RIP: 0033:0x7f9cf3479b7c 2024-09-25-09:16:08: [ 207.175512] Code: ec 28 48 89 54 24 18 48 89 74 24 10 89 7c 24 08 e8 b9 8f f9 ff 48 8b 54 24 18 48 8b 74 24 10 41 89 c0 8b 7c 24 08 31 c0 0f 05 <48> 3d 00 f0 ff ff 77 34 44 89 c7 48 89 44 24 08 e8 0f 90 f9 ff 48 2024-09-25-09:16:08: [ 207.175525] RSP: 002b:00007ffe298ea660 EFLAGS: 00000246 ORIG_RAX: 0000000000000000 2024-09-25-09:16:08: [ 207.175528] RAX: ffffffffffffffda RBX: 0000556fd38d72a0 RCX: 00007f9cf3479b7c 2024-09-25-09:16:08: [ 207.175530] RDX: 0000000000000400 RSI: 0000556fd38d7500 RDI: 0000000000000007 2024-09-25-09:16:08: [ 207.175531] RBP: 00007f9cf354e050 R08: 0000000000000000 R09: 0000000000000000 2024-09-25-09:16:08: [ 207.175533] R10: 00007f9cf34f9fe0 R11: 0000000000000246 R12: 00007f9cf354df00 2024-09-25-09:16:08: [ 207.175534] R13: 0000556fd38d789b R14: 0000000000000000 R15: 0000000000000065 2024-09-25-09:16:08: [ 207.175537] </TASK> 2024-09-25-09:16:08: [ 207.175541] watchdog: BUG: soft lockup - CPU#1 stuck for 28s! [watchdogd:69] 2024-09-25-09:16:08: [ 207.175578] Modules linked in: nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 rfkill nf_tables qrtr vfat fat intel_rapl_msr intel_rapl_common kvm_intel kvm iTCO_wdt iTCO_vendor_support rapl pcspkr lpc_ich virtio_balloon i2c_i801 i2c_smbus joydev sg fuse loop nfnetlink xfs nvme_tcp ahci crct10dif_pclmul bochs drm_vram_helper libahci nvme_keyring crc32_pclmul sd_mod nvme_fabrics drm_ttm_helper crc32c_intel nvme_core virtio_scsi virtio_net ttm libata net_failover ghash_clmulni_intel failover nvme_auth serio_raw dm_mirror dm_region_hash dm_log be2iscsi iscsi_boot_sysfs cxgb4i cxgb4 tls libcxgbi libcxgb iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi dm_multipath dm_mod 2024-09-25-09:16:08: [ 207.175664] CPU: 1 UID: 0 PID: 69 Comm: watchdogd Kdump: loaded Tainted: G L ------- --- 6.11.0-25.el10.x86_64 #1 2024-09-25-09:16:08: [ 207.175668] Tainted: [L]=SOFTLOCKUP 2024-09-25-09:16:08: [ 207.175670] Hardware name: Red Hat KVM/RHEL, BIOS edk2-20240524-6.el10 05/24/2024 2024-09-25-09:16:08: [ 207.175673] RIP: 0010:iTCO_wdt_ping+0x3b/0x80 [iTCO_wdt] 2024-09-25-09:16:08: [ 207.175719] Code: 00 48 8d ab e0 00 00 00 48 89 ef e8 2f 1e 5e c4 83 bb c0 00 00 00 01 76 22 48 8b 83 c8 00 00 00 48 8b 10 b8 01 00 00 00 66 ef <48> 89 ef e8 0d 1f 5e c4 31 c0 5b 5d c3 cc cc cc cc 75 ed 48 8b 83 2024-09-25-09:16:08: [ 207.175739] RSP: 0018:ffffb9e000257e60 EFLAGS: 00000202 2024-09-25-09:16:08: [ 207.175742] RAX: 0000000000000001 RBX: ffff9ea780caf228 RCX: 0000000000139b4f 2024-09-25-09:16:08: [ 207.175744] RDX: 0000000000000660 RSI: ffff9ea799ce5bd8 RDI: ffff9ea780caf308 2024-09-25-09:16:08: [ 207.175746] RBP: ffff9ea780caf308 R08: 0000000000000000 R09: 0000000000000000 2024-09-25-09:16:08: [ 207.175748] R10: 00000000f5257d14 R11: 0000000000000002 R12: ffff9ea799ce5800 2024-09-25-09:16:08: [ 207.175750] R13: ffff9ea780d28444 R14: 0000000000000001 R15: ffffffff84e26450 2024-09-25-09:16:08: [ 207.175752] FS: 0000000000000000(0000) GS:ffff9ea7fbc80000(0000) knlGS:0000000000000000 2024-09-25-09:16:08: [ 207.175757] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 2024-09-25-09:16:08: [ 207.175759] CR2: 00007fd38e040000 CR3: 000000010ca4c003 CR4: 0000000000170ef0 2024-09-25-09:16:08: [ 207.175761] Call Trace: 2024-09-25-09:16:08: [ 207.175776] <IRQ> 2024-09-25-09:16:08: [ 207.175782] ? show_trace_log_lvl+0x1b0/0x2f0 2024-09-25-09:16:08: [ 207.175790] ? show_trace_log_lvl+0x1b0/0x2f0 2024-09-25-09:16:08: [ 207.175794] ? __watchdog_ping+0x50/0x120 2024-09-25-09:16:08: [ 207.175871] ? watchdog_timer_fn.cold+0x3d/0xa0 2024-09-25-09:16:08: [ 207.175875] ? __pfx_watchdog_timer_fn+0x10/0x10 2024-09-25-09:16:08: [ 207.175878] ? __hrtimer_run_queues+0x13c/0x2a0 2024-09-25-09:16:08: [ 207.175883] ? hrtimer_interrupt+0xfa/0x210 2024-09-25-09:16:08: [ 207.175885] ? __sysvec_apic_timer_interrupt+0x55/0x100 2024-09-25-09:16:08: [ 207.175889] ? sysvec_apic_timer_interrupt+0x6c/0x90 2024-09-25-09:16:08: [ 207.175893] </IRQ> 2024-09-25-09:16:08: [ 207.175894] <TASK> 2024-09-25-09:16:08: [ 207.175895] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 2024-09-25-09:16:08: [ 207.175898] ? __pfx_watchdog_ping_work+0x10/0x10 2024-09-25-09:16:08: [ 207.175905] ? iTCO_wdt_ping+0x3b/0x80 [iTCO_wdt] 2024-09-25-09:16:08: [ 207.175907] __watchdog_ping+0x50/0x120 2024-09-25-09:16:08: [ 207.175942] watchdog_ping_work+0x44/0x50 2024-09-25-09:16:08: [ 207.175954] kthread_worker_fn+0xa1/0x220 2024-09-25-09:16:08: [ 207.175969] ? __pfx_kthread_worker_fn+0x10/0x10 2024-09-25-09:16:08: [ 207.175972] kthread+0xd2/0x100 2024-09-25-09:16:08: [ 207.175974] ? __pfx_kthread+0x10/0x10 2024-09-25-09:16:08: [ 207.175976] ret_from_fork+0x34/0x50 2024-09-25-09:16:08: [ 207.175980] ? __pfx_kthread+0x10/0x10 2024-09-25-09:16:08: [ 207.175982] ret_from_fork_asm+0x1a/0x30 2024-09-25-09:16:08: [ 207.175987] </TASK>
Expected results
No call trace in VM.
Actual results
The qemu command lines of two cases are almost same, case 2 appends "-S":
/usr/libexec/qemu-kvm \ -name "mouse-vm" \ -sandbox on \ -machine q35,memory-backend=pc.ram,pflash0=drive_ovmf_code,pflash1=drive_ovmf_vars \ -cpu Haswell-noTSX-IBRS,vmx=on,pdcm=on,f16c=on,rdrand=on,hypervisor=on,vme=on,ss=on,arat=on,tsc-adjust=on,umip=on,md-clear=on,stibp=on,flush-l1d=on,arch-capabilities=on,ssbd=on,xsaveopt=on,abm=on,pdpe1gb=on,invtsc=on,ibpb=on,ibrs=on,amd-stibp=on,amd-ssbd=on,skip-l1dfl-vmentry=on,pschange-mc-no=on,gds-no=on,vmx-ins-outs=on,vmx-true-ctls=on,vmx-store-lma=on,vmx-activity-hlt=on,vmx-activity-wait-sipi=on,vmx-vmwrite-vmexit-fields=on,vmx-apicv-xapic=on,vmx-ept=on,vmx-desc-exit=on,vmx-rdtscp-exit=on,vmx-apicv-x2apic=on,vmx-vpid=on,vmx-wbinvd-exit=on,vmx-unrestricted-guest=on,vmx-apicv-register=on,vmx-apicv-vid=on,vmx-rdrand-exit=on,vmx-invpcid-exit=on,vmx-vmfunc=on,vmx-shadow-vmcs=on,vmx-pml=on,vmx-ept-execonly=on,vmx-page-walk-4=on,vmx-ept-2mb=on,vmx-ept-1gb=on,vmx-invept=on,vmx-eptad=on,vmx-invept-single-context=on,vmx-invept-all-context=on,vmx-invvpid=on,vmx-invvpid-single-addr=on,vmx-invvpid-all-context=on,vmx-intr-exit=on,vmx-nmi-exit=on,vmx-vnmi=on,vmx-preemption-timer=on,vmx-posted-intr=on,vmx-vintr-pending=on,vmx-tsc-offset=on,vmx-hlt-exit=on,vmx-invlpg-exit=on,vmx-mwait-exit=on,vmx-rdpmc-exit=on,vmx-rdtsc-exit=on,vmx-cr3-load-noexit=on,vmx-cr3-store-noexit=on,vmx-cr8-load-exit=on,vmx-cr8-store-exit=on,vmx-flexpriority=on,vmx-vnmi-pending=on,vmx-movdr-exit=on,vmx-io-exit=on,vmx-io-bitmap=on,vmx-mtf=on,vmx-msr-bitmap=on,vmx-monitor-exit=on,vmx-pause-exit=on,vmx-secondary-ctls=on,vmx-exit-nosave-debugctl=on,vmx-exit-load-perf-global-ctrl=on,vmx-exit-ack-intr=on,vmx-exit-save-pat=on,vmx-exit-load-pat=on,vmx-exit-save-efer=on,vmx-exit-load-efer=on,vmx-exit-save-preemption-timer=on,vmx-entry-noload-debugctl=on,vmx-entry-ia32e-mode=on,vmx-entry-load-perf-global-ctrl=on,vmx-entry-load-pat=on,vmx-entry-load-efer=on,vmx-eptp-switching=on,tsc-frequency=1899996000,host-phys-bits=on,host-phys-bits-limit=46 \ -nodefaults \ -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpmonitor1,server=on,wait=off \ -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/monitor-catch_monitor,server=on,wait=off \ -mon chardev=qmp_id_qmpmonitor1,mode=control \ -mon chardev=qmp_id_catch_monitor,mode=control \ -device '{"driver":"pcie-root-port","id":"root0","multifunction":true,"bus":"pcie.0","addr":"0x2","chassis":1}' \ -device '{"driver":"pcie-root-port","id":"root1","port":11,"addr":"0x2.0x1","bus":"pcie.0","chassis":2}' \ -device '{"driver":"pcie-root-port","id":"root2","port":12,"addr":"0x2.0x2","bus":"pcie.0","chassis":3}' \ -device '{"driver":"pcie-root-port","id":"root3","port":13,"addr":"0x2.0x3","bus":"pcie.0","chassis":4}' \ -device '{"driver":"pcie-root-port","id":"root4","port":14,"addr":"0x2.0x4","bus":"pcie.0","chassis":5}' \ -device '{"driver":"pcie-root-port","id":"root5","port":15,"addr":"0x2.0x5","bus":"pcie.0","chassis":6}' \ -device '{"driver":"pcie-root-port","id":"root6","port":16,"addr":"0x2.0x6","bus":"pcie.0","chassis":7}' \ -device '{"driver":"pcie-root-port","id":"root7","port":17,"addr":"0x2.0x7","bus":"pcie.0","chassis":8}' \ -device '{"driver":"pcie-root-port","id":"extra_root0","multifunction":true,"bus":"pcie.0","addr":"0x3","chassis":21}' \ -device '{"driver":"pcie-root-port","id":"extra_root1","port":21,"addr":"0x3.0x1","bus":"pcie.0","chassis":22}' \ -device '{"driver":"pcie-root-port","id":"extra_root2","port":22,"addr":"0x3.0x2","bus":"pcie.0","chassis":23}' \ -device '{"driver":"nec-usb-xhci","id":"usb1","bus":"root0","addr":"0x0"}' \ -device '{"driver":"virtio-scsi-pci","id":"virtio_scsi_pci0","bus":"root1","addr":"0x0"}' \ -device '{"driver":"scsi-hd","id":"image1","device_id":"drive-image1","drive":"drive_image1","bus":"virtio_scsi_pci0.0","channel":0,"scsi-id":0,"lun":0,"bootindex":0,"write-cache":"on"}' \ -device '{"driver":"virtio-net-pci","mac":"9a:8a:8b:8c:8d:8e","id":"net0","netdev":"tap0","bus":"root2","addr":"0x0"}' \ -device '{"driver":"usb-tablet","id":"usb-tablet1","bus":"usb1.0","port":"1"}' \ -device '{"driver":"virtio-balloon-pci","id":"balloon0","bus":"root3","addr":"0x0"}' \ -device '{"driver":"VGA","id":"video0","vgamem_mb":16,"bus":"pcie.0","addr":"0x1"}' \ -blockdev '{"driver":"file","auto-read-only":true,"discard":"unmap","aio":"threads","cache":{"direct":true,"no-flush":false},"filename":"/mnt/xiaohli/rhel100-64-virtio-scsi-ovmf.qcow2","node-name":"drive_sys1"}' \ -blockdev '{"driver":"qcow2","node-name":"drive_image1","read-only":false,"cache":{"direct":true,"no-flush":false},"file":"drive_sys1"}' \ -blockdev '{"node-name":"file_ovmf_code","driver":"file","filename":"/usr/share/OVMF/OVMF_CODE.secboot.fd","auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"drive_ovmf_code","driver":"raw","read-only":true,"file":"file_ovmf_code"}' \ -blockdev '{"node-name":"file_ovmf_vars","driver":"file","filename":"/mnt/xiaohli/rhel100-64-virtio-scsi-ovmf.qcow2_VARS.fd","auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"drive_ovmf_vars","driver":"raw","read-only":false,"file":"file_ovmf_vars"}' \ -netdev tap,id=tap0,vhost=on \ -m 4096 \ -object '{"qom-type":"memory-backend-ram","id":"pc.ram","size":4294967296}' \ -smp 4,maxcpus=4,cores=2,threads=1,sockets=2 \ -vnc :10 \ -rtc base=utc,clock=host \ -boot menu=off,strict=off,order=cdn,once=c \ -enable-kvm \ -qmp tcp:0:3333,server=on,wait=off \ -qmp tcp:0:9999,server=on,wait=off \ -qmp tcp:0:9888,server=on,wait=off \ -serial tcp:0:4444,server=on,wait=off \ -monitor stdio \ -msg timestamp=on \