Uploaded image for project: 'RHEL'
  1. RHEL
  2. RHEL-60558

Guest hit call trace when boot multi VMs on two hosts, or when migrate VM during the early boot stage

    • Yes
    • Important
    • CustomerScenariosInitiative
    • rhel-sst-virtualization
    • ssg_virtualization
    • None
    • False
    • Hide

      None

      Show
      None
    • None
    • None
    • Unspecified
    • None

      What were you trying to do that didn't work?

      Hit guest call trace under two cases:
      1. First boot two VM on the source host, and then boot the third VM3 on the destination host. the VM3 hit call trace on the destination
      2. Migrate the VM from the source to the destination during the early stage of boot

      What is the impact of this issue to you?

      About case 1: the VM3 automatically quit after call trace;
      About case 2: migration completes, only hit the guest call trace info when check dmesg on the destination host after migration.

      Please provide the package NVR for which the bug is seen:

      hosts: kernel-6.11.0-25.el10.x86_64 && qemu-kvm-9.1.0-1.el10.x86_64
      guest: kernel-6.11.0-25.el10.x86_64

      How reproducible is this bug?:

      100%

      Steps to reproduce

      1. Boot VMs with below qemu command lines

      For case1:
      Boot two VMs on the source host, then boot VM3 on the destination host,
      the VM3 on the dst host hit call trace:

      2024-09-25-06:48:24: [  101.780517] watchdog: BUG: soft lockup - CPU#2 stuck for 24s! [systemd:1]
      2024-09-25-06:48:24: [  101.786263] Modules linked in: fuse loop nfnetlink xfs nvme_tcp nvme_keyring sd_mod nvme_fabrics ahci nvme_core bochs libahci drm_vram_helper crct10dif_pclmul crc32_pclmul crc32c_intel nvme_auth virtio_net drm_ttm_helper libata net_failover ttm ghash_clmulni_intel virtio_scsi failover serio_raw dm_mirror dm_region_hash dm_log be2iscsi iscsi_boot_sysfs cxgb4i cxgb4 tls libcxgbi libcxgb iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi dm_multipath dm_mod
      2024-09-25-06:48:24: [  101.788048] CPU: 2 UID: 0 PID: 1 Comm: systemd Not tainted 6.11.0-25.el10.x86_64 #1
      2024-09-25-06:48:24: g syste[  101.788555] Hardware name: Red Hat KVM/RHEL, BIOS edk2-20240524-6.el10 05/24/2024
      2024-09-25-06:48:24: md-modules-load.[  101.789073] RIP: 0010:_raw_spin_unlock_irqrestore+0x1f/0x30
      2024-09-25-06:48:24: service - Lo[  101.789582] Code: 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 0f 1f 44 00 00 e8 82 07 00 00 90 f7 c6 00 02 00 00 74 01 fb 65 ff 0d 89 d0 c3 73 <74> 05 c3 cc cc cc cc 0f 1f 44 00 00 c3 cc cc cc cc 90 90 90 90 90
      2024-09-25-06:48:24: ad Kernel Module[  101.790609] RSP: 0018:ffffa5a24001b8f8 EFLAGS: 00000286
      2024-09-25-06:48:24: s...
      2024-09-25-06:48:24: 
      2024-09-25-06:48:24: 
      2024-09-25-06:48:24: [0;21;3[  101.791170] RAX: 0000000000000001 RBX: ffffffff8e5bd060 RCX: 0000000000000000
      2024-09-25-06:48:24: 9msystemd[1]: St[  101.791697] RDX: 0000000000000002 RSI: 0000000000000246 RDI: ffffffff8e5bd060
      2024-09-25-06:48:24: arting systemd-n[  101.792214] RBP: ffff8efbaf943100 R08: 0000000000000002 R09: 0000000000000001
      2024-09-25-06:48:24: etwork-generator[  101.792737] R10: ffffffff8d721be8 R11: 0000000000000003 R12: ffff8efb80294d40
      2024-09-25-06:48:24: .service - Gener[  101.793219] R13: 0000000000001002 R14: ffff8efb811b0000 R15: ffff8efb81593750
      2024-09-25-06:48:24: ate network unit[  101.793722] FS:  00007f94dc448440(0000) GS:ffff8efbfbd00000(0000) knlGS:0000000000000000
      2024-09-25-06:48:24: s from Kernel co[  101.794227] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
      2024-09-25-06:48:24: mmand line...[0[  101.794715] CR2: 00007f0a1bc51080 CR3: 00000001026c6003 CR4: 0000000000170ef0
      2024-09-25-06:48:24: m
      2024-09-25-06:48:24: Add[  101.795225] Call Trace:
      2024-09-25-06:48:24: ing 2097148k swa[  101.795723]  <IRQ>
      2024-09-25-06:48:24: p on /dev/mapper[  101.796234]  ? show_trace_log_lvl+0x1b0/0x2f0
      2024-09-25-06:48:24: /rhel-swap.  Pri[  101.796722]  ? show_trace_log_lvl+0x1b0/0x2f0
      2024-09-25-06:48:24: ority:-2 extents[  101.797208]  ? uart_dtr_rts+0x86/0x90
      2024-09-25-06:48:24: :1 across:209714[  101.797677]  ? watchdog_timer_fn.cold+0x3d/0xa0
      2024-09-25-06:48:24: 8k 
      2024-09-25-06:48:24:        [  101.798154]  ? __pfx_watchdog_timer_fn+0x10/0x10
      2024-09-25-06:48:24:   Starting [0;1[  101.798611]  ? __hrtimer_run_queues+0x13c/0x2a0
      2024-09-25-06:48:24: ;39msystemd-netw[  101.799066]  ? hrtimer_interrupt+0xfa/0x210
      2024-09-25-06:48:24: ork-generator[  101.799500]  ? __sysvec_apic_timer_interrupt+0x55/0x100
      2024-09-25-06:48:24: k units from[  101.799936]  ? sysvec_apic_timer_interrupt+0x6c/0x90
      2024-09-25-06:48:24:  Kernel command [  101.800358]  </IRQ>
      2024-09-25-06:48:24: line...
      2024-09-25-06:48:24: 
      2024-09-25-06:48:24: 
      2024-09-25-06:48:24: [0;2[  101.800780]  <TASK>
      2024-09-25-06:48:24: 1;39msystemd[1]:[  101.801186]  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
      2024-09-25-06:48:24:  systemd-pcrmach[  101.801605]  ? _raw_spin_unlock_irqrestore+0x1f/0x30
      2024-09-25-06:48:24: ine.service - TP[  101.802023]  ? _raw_spin_unlock_irqrestore+0xe/0x30
      2024-09-25-06:48:24: M PCR Machine ID[  101.802429]  uart_dtr_rts+0x86/0x90
      2024-09-25-06:48:24:  Measurement was[  101.802849]  tty_port_block_til_ready+0x27d/0x330
      2024-09-25-06:48:24:  skipped because[  101.803253]  ? __pfx_autoremove_wake_function+0x10/0x10
      2024-09-25-06:48:24:  of an unmet con[  101.803650]  uart_open+0x1e/0x30
      2024-09-25-06:48:24: dition check (Co[  101.804048]  tty_open+0x161/0x6a0
      2024-09-25-06:48:24: nditionSecurity=[  101.804427]  chrdev_open+0xcb/0x240
      2024-09-25-06:48:24: measured-uki).[[  101.804809]  ? __pfx_chrdev_open+0x10/0x10
      2024-09-25-06:48:24: 0m[  101.805164]  do_dentry_open+0x25a/0x490
      2024-09-25-06:48:24: [  101.805514]  vfs_open+0x34/0xf0
      2024-09-25-06:48:24: [  101.805867]  do_open+0x165/0x3d0
      2024-09-25-06:48:24: 
      2024-09-25-06:48:24: [  101.806206]  path_openat+0x124/0x2d0
      2024-09-25-06:48:24: [  101.806539]  ? pick_next_task+0x50/0xb10
      2024-09-25-06:48:24: [  101.806875]  do_filp_open+0xc4/0x170
      2024-09-25-06:48:24:          Startin[  101.807191]  do_sys_openat2+0xae/0xe0
      2024-09-25-06:48:24: g syste[  101.807492]  __x64_sys_openat+0x55/0xa0
      2024-09-25-06:48:24: md-remount-fs.se[  101.807795]  do_syscall_64+0x7d/0x160
      2024-09-25-06:48:24: rvic…unt R[  101.808089]  ? __do_sys_newfstat+0x68/0x70
      2024-09-25-06:48:24: oot and Kernel F[  101.808366]  ? __memcg_slab_free_hook+0x100/0x150
      2024-09-25-06:48:24: ile Systems...
      2024-09-25-06:48:24: 
      2024-09-25-06:48:24: [  101.808633]  ? __x64_sys_close+0x3c/0x80
      2024-09-25-06:48:24: [  101.808895]  ? kmem_cache_free+0x3d5/0x400
      2024-09-25-06:48:24: 
      2024-09-25-06:48:24: [  101.809139]  ? syscall_exit_to_user_mode+0x10/0x1f0
      2024-09-25-06:48:24:          Startin[  101.809381]  ? do_syscall_64+0x89/0x160
      2024-09-25-06:48:24: g syste[  101.809618]  ? filp_flush+0x56/0x70
      2024-09-25-06:48:24: md-udev-load-cre[  101.809851]  ? syscall_exit_to_user_mode+0x10/0x1f0
      2024-09-25-06:48:24: dentLoad [  101.810064]  ? do_syscall_64+0x89/0x160
      2024-09-25-06:48:24: udev Rules from [  101.810287]  ? syscall_exit_to_user_mode+0x10/0x1f0
      2024-09-25-06:48:24: Credentials...
      2024-09-25-06:48:24: 
      2024-09-25-06:48:24: [  101.810495]  ? do_syscall_64+0x89/0x160
      2024-09-25-06:48:24: [  101.810700]  ? do_syscall_64+0x89/0x160
      2024-09-25-06:48:24: 
      2024-09-25-06:48:24: [  101.810907]  ? exc_page_fault+0x73/0x160
      2024-09-25-06:48:24: [  101.811106]  entry_SYSCALL_64_after_hwframe+0x76/0x7e
      2024-09-25-06:48:24:          Startin[  101.811310] RIP: 0033:0x7f94dbb20210
      2024-09-25-06:48:24: g syste[  101.811648] Code: 83 e2 40 75 55 89 f0 f7 d0 a9 00 00 41 00 74 4a 80 3d 43 de 0d 00 00 74 6e 89 da 48 89 ee bf 9c ff ff ff b8 01 01 00 00 0f 05 <48> 3d 00 f0 ff ff 0f 87 94 00 00 00 48 8b 54 24 28 64 48 2b 14 25
      2024-09-25-06:48:24: [  101.812084] RSP: 002b:00007ffc30704f80 EFLAGS: 00000202 ORIG_RAX: 0000000000000101
      2024-09-25-06:48:24: md-udev-trigger.[  101.812304] RAX: ffffffffffffffda RBX: 0000000000080101 RCX: 00007f94dbb20210
      2024-09-25-06:48:24: service - Co[  101.812527] RDX: 0000000000080101 RSI: 00007f94dc37a5d6 RDI: 00000000ffffff9c
      2024-09-25-06:48:24: ldplug All udev [  101.812772] RBP: 00007f94dc37a5d6 R08: 0000000000000073 R09: 00000000ffffffff
      2024-09-25-06:48:24: Devices...
      2024-09-25-06:48:24: 
      2024-09-25-06:48:24: [  101.813058] R10: 0000000000000000 R11: 0000000000000202 R12: 0000000000080101
      2024-09-25-06:48:24: 
      2024-09-25-06:48:24: [  101.813289] R13: 00007f94dc37a5d6 R14: 00007ffc30705000 R15: 000055f8c591def0
      2024-09-25-06:48:24: [  101.813522]  </TASK>
      2024-09-25-06:48:24: [  OK  [[  101.813761] watchdog: BUG: soft lockup - CPU#0 stuck for 24s! [plymouthd:618]
      2024-09-25-06:48:24: 0m] Activated sw[  101.814059] Modules linked in: fuse loop nfnetlink xfs nvme_tcp nvme_keyring sd_mod nvme_fabrics ahci nvme_core bochs libahci drm_vram_helper crct10dif_pclmul crc32_pclmul crc32c_intel nvme_auth virtio_net drm_ttm_helper libata net_failover ttm ghash_clmulni_intel virtio_scsi failover serio_raw dm_mirror dm_region_hash dm_log be2iscsi iscsi_boot_sysfs cxgb4i cxgb4 tls libcxgbi libcxgb iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi dm_multipath dm_mod
      2024-09-25-06:48:24: [  101.815154] CPU: 0 UID: 0 PID: 618 Comm: plymouthd Tainted: G             L    -------  ---  6.11.0-25.el10.x86_64 #1
      2024-09-25-06:48:24: ap dev-[  101.815550] Tainted: [L]=SOFTLOCKUP
      2024-09-25-06:48:24: disk-by\x2duuid[  101.815947] Hardware name: Red Hat KVM/RHEL, BIOS edk2-20240524-6.el10 05/24/2024
      2024-09-25-06:48:24: 4cb060-21a[  101.816331] RIP: 0010:_raw_spin_unlock_irqrestore+0x1f/0x30
      2024-09-25-06:48:24: 8-434a-91e6-83e0[  101.816736] Code: 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 0f 1f 44 00 00 e8 82 07 00 00 90 f7 c6 00 02 00 00 74 01 fb 65 ff 0d 89 d0 c3 73 <74> 05 c3 cc cc cc cc 0f 1f 44 00 00 c3 cc cc cc cc 90 90 90 90 90
      2024-09-25-06:48:24: 369e1296.
      2024-09-25-06:48:24: 
      2024-09-25-06:48:24: [  101.817604] RSP: 0018:ffffa5a2403a3be0 EFLAGS: 00000286
      2024-09-25-06:48:24: 
      2024-09-25-06:48:24: [  101.818040] RAX: 0000000000000001 RBX: 0000000000000e75 RCX: 0000000000000000
      2024-09-25-06:48:24: [  101.818454] RDX: 0000000000000000 RSI: 0000000000000286 RDI: ffffffff8e5bd060
      2024-09-25-06:48:24: [  OK  [[  101.818849] RBP: ffffffff8e5bd060 R08: 0000000000000000 R09: 0000000000000001
      2024-09-25-06:48:24: 0m] Started [0;[  101.819181] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000286
      2024-09-25-06:48:24: 1;39msystemd-jou[  101.819506] R13: ffffa5a2403d1000 R14: ffff8efb815ec000 R15: 00000000000000a7
      2024-09-25-06:48:24: rnald.service[0[  101.819838] FS:  00007fd043521280(0000) GS:ffff8efbfbc00000(0000) knlGS:0000000000000000
      2024-09-25-06:48:24: m - Journal Serv[  101.820174] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
      2024-09-25-06:48:24: ice.
      2024-09-25-06:48:24: 
      2024-09-25-06:48:24: [  101.820510] CR2: 00007f32e1aee080 CR3: 00000001137b8004 CR4: 0000000000170ef0
      2024-09-25-06:48:24: 
      2024-09-25-06:48:24: [  101.820863] Call Trace:
      2024-09-25-06:48:24: [  101.821211]  <IRQ>
      2024-09-25-06:48:24: [  OK  [[  101.821561]  ? show_trace_log_lvl+0x1b0/0x2f0
      2024-09-25-06:48:24: 0m] Mounted [0;[  101.821924]  ? show_trace_log_lvl+0x1b0/0x2f0
      2024-09-25-06:48:24: 1;39mdev-hugepag[  101.822279]  ? uart_write_room+0x74/0x190
      2024-09-25-06:48:24: es.mount - H[  101.822634]  ? watchdog_timer_fn.cold+0x3d/0xa0
      2024-09-25-06:48:24: uge Pages File S[  101.822996]  ? __pfx_watchdog_timer_fn+0x10/0x10
      2024-09-25-06:48:24: ystem.
      2024-09-25-06:48:24: 
      2024-09-25-06:48:24: 
      2024-09-25-06:48:24: [[0;3[  101.823353]  ? __hrtimer_run_queues+0x13c/0x2a0
      2024-09-25-06:48:24: 2m  OK  ] Mo[  101.823710]  ? hrtimer_interrupt+0xfa/0x210
      2024-09-25-06:48:24: unted d[  101.824070]  ? __sysvec_apic_timer_interrupt+0x55/0x100
      2024-09-25-06:48:24: ev-mqueue.mount[  101.824429]  ? sysvec_apic_timer_interrupt+0x6c/0x90
      2024-09-25-06:48:24: [0m - POSIX Mess[  101.824792]  </IRQ>
      2024-09-25-06:48:24: age Queue File S[  101.825144]  <TASK>
      2024-09-25-06:48:24: ystem.
      2024-09-25-06:48:24: 
      2024-09-25-06:48:24: 
      2024-09-25-06:48:24: [[0;3[  101.825497]  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
      2024-09-25-06:48:24: 2m  OK  ] Mo[  101.825861]  ? _raw_spin_unlock_irqrestore+0x1f/0x30
      2024-09-25-06:48:24: unted s[  101.826218]  uart_write_room+0x74/0x190
      2024-09-25-06:48:24: ys-kernel-debug.[  101.826577]  process_output_block+0x3a/0x1c0
      2024-09-25-06:48:24: mount - Kern[  101.826954]  ? add_wait_queue+0x7e/0xb0
      2024-09-25-06:48:24: el Debug File Sy[  101.827312]  n_tty_write+0x1a4/0x3c0
      2024-09-25-06:48:24: stem.
      2024-09-25-06:48:24: 
      2024-09-25-06:48:24: 
      2024-09-25-06:48:24: [[0;32[  101.827668]  ? __pfx_woken_wake_function+0x10/0x10
      2024-09-25-06:48:24: m  OK  ] Mou[  101.828033]  iterate_tty_write+0x114/0x240
      2024-09-25-06:48:24: nted sy[  101.828392]  file_tty_write.isra.0+0x86/0xb0
      2024-09-25-06:48:24: s-kernel-tracing[  101.828746]  vfs_write+0x294/0x460
      2024-09-25-06:48:24: .mount - Ker[  101.829087]  ksys_write+0x6d/0xf0
      2024-09-25-06:48:24: nel Trace File S[  101.829426]  do_syscall_64+0x7d/0x160
      2024-09-25-06:48:24: ystem.
      2024-09-25-06:48:24: 
      2024-09-25-06:48:24: 
      2024-09-25-06:48:24: [  101.829769]  ? syscall_exit_to_user_mode+0x10/0x1f0
      2024-09-25-06:48:24: [  101.830107]  ? do_syscall_64+0x89/0x160
      2024-09-25-06:48:24: [  101.830448]  ? ksys_read+0x6d/0xf0
      2024-09-25-06:48:24: [  101.830787]  ? syscall_exit_to_user_mode+0x10/0x1f0
      2024-09-25-06:48:24: [  101.831121]  ? do_syscall_64+0x89/0x160
      2024-09-25-06:48:24: [  101.831434]  entry_SYSCALL_64_after_hwframe+0x76/0x7e
      2024-09-25-06:48:24: [  101.831738] RIP: 0033:0x7fd0437ac6a4
      2024-09-25-06:48:24: [  101.832048] Code: c7 00 16 00 00 00 b8 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 80 3d a5 c9 0d 00 00 74 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 48 83 ec 28 48 89 54 24 18 48
      2024-09-25-06:48:24: [  101.832667] RSP: 002b:00007ffeb2d85198 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
      2024-09-25-06:48:24: fuse: [  101.832997] RAX: ffffffffffffffda RBX: 0000563fd8c56110 RCX: 00007fd0437ac6a4
      2024-09-25-06:48:24: init (API versio[  101.833299] RDX: 00000000000000a7 RSI: 0000563fd8c81f60 RDI: 000000000000000a
      2024-09-25-06:48:24: n 7.40)[  101.833584] RBP: 00000000000000a7 R08: 0000563fd8c844c0 R09: 00000000000000d0
      2024-09-25-06:48:24: [  101.833871] R10: 0000000000000009 R11: 0000000000000202 R12: 0000563fd8c890c0
      2024-09-25-06:48:24: [  101.834152] R13: 0000563fd8c80640 R14: 0000563fcc03e4c2 R15: 0000563fd8c89180
      2024-09-25-06:48:24: 
      2024-09-25-06:48:24: [  101.834442]  </TASK>
      2024-09-25-06:48:24: [  101.834705] watchdog: BUG: soft lockup - CPU#1 stuck for 24s! [kworker/1:2:301]
      2024-09-25-06:48:24: [  101.835127] Modules linked in: fuse loop nfnetlink xfs nvme_tcp nvme_keyring sd_mod nvme_fabrics ahci nvme_core bochs libahci drm_vram_helper crct10dif_pclmul crc32_pclmul crc32c_intel nvme_auth virtio_net drm_ttm_helper libata net_failover ttm ghash_clmulni_intel virtio_scsi failover serio_raw dm_mirror dm_region_hash dm_log be2iscsi iscsi_boot_sysfs cxgb4i cxgb4 tls libcxgbi libcxgb iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi dm_multipath dm_mod
      2024-09-25-06:48:24: [  101.836467] CPU: 1 UID: 0 PID: 301 Comm: kworker/1:2 Tainted: G             L    -------  ---  6.11.0-25.el10.x86_64 #1
      2024-09-25-06:48:24: [  101.836946] Tainted: [L]=SOFTLOCKUP
      2024-09-25-06:48:24: [  OK  [[  101.837422] Hardware name: Red Hat KVM/RHEL, BIOS edk2-20240524-6.el10 05/24/2024
      2024-09-25-06:48:24: 0m] Finished [0[  101.837846] Workqueue: pm pm_runtime_work
      2024-09-25-06:48:24: ;1;39mkmod-stati[  101.838238] RIP: 0010:_raw_spin_unlock_irqrestore+0x1f/0x30
      2024-09-25-06:48:24: c-nodes.service[  101.838662] Code: 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 0f 1f 44 00 00 e8 82 07 00 00 90 f7 c6 00 02 00 00 74 01 fb 65 ff 0d 89 d0 c3 73 <74> 05 c3 cc cc cc cc 0f 1f 44 00 00 c3 cc cc cc cc 90 90 90 90 90
      2024-09-25-06:48:24: [  101.839604] RSP: 0018:ffffa5a2405c3d50 EFLAGS: 00000286
      2024-09-25-06:48:24: [  101.840102] RAX: 0000000000000001 RBX: ffffffff8e5bd060 RCX: 0000000000000000
      2024-09-25-06:48:24: [0mCreate Lis[  101.840595] RDX: 0000000000000001 RSI: 0000000000000246 RDI: ffffffff8e5bd060
      2024-09-25-06:48:24: t of Static Devi[  101.841099] RBP: ffff8efb838ecc00 R08: 0000000000000001 R09: 0000000000000001
      2024-09-25-06:48:24: ce Nodes.
      2024-09-25-06:48:24: 
      2024-09-25-06:48:24: 
      2024-09-25-06:48:24: [[[  101.841623] R10: 0000000000000003 R11: 0000000000000000 R12: 0000000000000246
      2024-09-25-06:48:24: 0;32m  OK  ][  101.842128] R13: 0000000000000001 R14: 0000000000000000 R15: ffff8efb901499c0
      2024-09-25-06:48:24:  Finished [0;1;[  101.842628] FS:  0000000000000000(0000) GS:ffff8efbfbc80000(0000) knlGS:0000000000000000
      2024-09-25-06:48:24: 39mlvm2-monitor.[  101.843144] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
      2024-09-25-06:48:24: service - Mo[  101.843652] CR2: 00005618866fa9d0 CR3: 00000001113c4005 CR4: 0000000000170ef0
      2024-09-25-06:48:24: …using dmevent[  101.844174] Call Trace:
      2024-09-25-06:48:24: d or progress po[  101.844682]  <IRQ>
      2024-09-25-06:48:24: lling.
      2024-09-25-06:48:24: 
      2024-09-25-06:48:24: 
      2024-09-25-06:48:24: [[0;3[  101.845201]  ? show_trace_log_lvl+0x1b0/0x2f0
      2024-09-25-06:48:24: 2m  OK  ] Fi[  101.845715]  ? show_trace_log_lvl+0x1b0/0x2f0
      2024-09-25-06:48:24: nished [  101.846224]  ? serial_port_runtime_suspend+0x12b/0x230
      2024-09-25-06:48:24: modprobe@configf[  101.846743]  ? watchdog_timer_fn.cold+0x3d/0xa0
      2024-09-25-06:48:24: s.service - [  101.847265]  ? __pfx_watchdog_timer_fn+0x10/0x10
      2024-09-25-06:48:24: Load Kernel Modu[  101.847777]  ? __hrtimer_run_queues+0x13c/0x2a0
      2024-09-25-06:48:24: le configfs.
      2024-09-25-06:48:24: 
      2024-09-25-06:48:24: 
      2024-09-25-06:48:24: [  101.848282]  ? hrtimer_interrupt+0xfa/0x210
      2024-09-25-06:48:24: [  OK  [[  101.848796]  ? __sysvec_apic_timer_interrupt+0x55/0x100
      2024-09-25-06:48:24: 0m] Finished [0[  101.849300]  ? sysvec_apic_timer_interrupt+0x6c/0x90
      2024-09-25-06:48:24: ;1;39mmodprobe@d[  101.849816]  </IRQ>
      2024-09-25-06:48:24: m_mod.service[0[  101.850319]  <TASK>
      2024-09-25-06:48:24: m - Load Kernel [  101.850832]  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
      2024-09-25-06:48:24: Module dm_mod.
      2024-09-25-06:48:24: 
      2024-09-25-06:48:24: [  101.851344]  ? _raw_spin_unlock_irqrestore+0x1f/0x30
      2024-09-25-06:48:24: [  101.851866]  serial_port_runtime_suspend+0x12b/0x230
      2024-09-25-06:48:24: 
      2024-09-25-06:48:24: [  OK  ] Finished [  101.852372]  ? __pfx_serial_port_runtime_suspend+0x10/0x10
      2024-09-25-06:48:24: [0;1;39mmodprobe[  101.852895]  __rpm_callback+0x44/0x170
      2024-09-25-06:48:24: @dm_multipath.se[  101.853417]  ? __pfx_serial_port_runtime_suspend+0x10/0x10
      2024-09-25-06:48:24: r - Load [  101.853941]  rpm_callback+0x55/0x60
      2024-09-25-06:48:24: Kernel Module dm[  101.854449]  ? __pfx_serial_port_runtime_suspend+0x10/0x10
      2024-09-25-06:48:24: _multipath.
      2024-09-25-06:48:24: 
      2024-09-25-06:48:24: 
      2024-09-25-06:48:24: [[  101.854962]  rpm_suspend+0xe6/0x5f0
      2024-09-25-06:48:24:   OK  [0[  101.855449]  ? _raw_spin_unlock+0xe/0x30
      2024-09-25-06:48:24: m] Finished [0;[  101.855982]  ? finish_task_switch.isra.0+0x99/0x2b0
      2024-09-25-06:48:24: 1;39mmodprobe@dr[  101.856461]  pm_runtime_work+0x98/0xb0
      2024-09-25-06:48:24: m.service - [  101.856949]  process_one_work+0x177/0x330
      2024-09-25-06:48:24: Load Kernel Modu[  101.857453]  worker_thread+0x252/0x390
      2024-09-25-06:48:24: le drm.
      2024-09-25-06:48:24: 
      2024-09-25-06:48:24: 
      2024-09-25-06:48:24: [[0;[  101.857954]  ? __pfx_worker_thread+0x10/0x10
      2024-09-25-06:48:24: 32m  OK  ] F[  101.858424]  kthread+0xd2/0x100
      2024-09-25-06:48:24: inished [0;1;39[  101.858798]  ? __pfx_kthread+0x10/0x10
      2024-09-25-06:48:24: mmodprobe@efi_ps[  101.859209]  ret_from_fork+0x34/0x50
      2024-09-25-06:48:24: tore.service[  101.859649]  ? __pfx_kthread+0x10/0x10
      2024-09-25-06:48:24:  - Load Kernel M[  101.860088]  ret_from_fork_asm+0x1a/0x30
      2024-09-25-06:48:24: odule efi_pstore[  101.860485]  </TASK>
      2024-09-25-06:48:24: .
      2024-09-25-06:48:24: 
      2024-09-25-06:48:24: 
      2024-09-25-06:48:24: [  O[  101.996834] systemd-journald[916]: Collecting audit messages is disabled.
      2024-09-25-06:48:24: K  ] Finishe[  104.147963] systemd[1]: Starting systemd-udev-trigger.service - Coldplug All udev Devices...
      2024-09-25-06:48:24: [  104.158162] systemd[1]: Activated swap dev-disk-by\x2duuid-794cb060\x2d21a8\x2d434a\x2d91e6\x2d83e0369e1296.swap - /dev/disk/by-uuid/794cb060-21a8-434a-91e6-83e0369e1296.
      2024-09-25-06:48:24: [  104.161685] systemd[1]: Started systemd-journald.service - Journal Service.
      2024-09-25-06:48:24: d modprobe@fuse.service - Load Kernel Module fuse.
      2024-09-25-06:48:24: 
      2024-09-25-06:48:24: 
      2024-09-25-06:48:24: [  OK  ] Finished modprobe@loop.service - Load Kernel Module loop.
      2024-09-25-06:48:24: 
      2024-09-25-06:48:24: 
      2024-09-25-06:48:24: [  OK  ] Finished systemd-modules-load.service - Load Kernel Modules.
      2024-09-25-06:48:24: 
      2024-09-25-06:48:24: 
      2024-09-25-06:48:24: [  OK  ] Finished systemd-network-generator…ork units from Kernel command line.
      2024-09-25-06:48:24: 
      2024-09-25-06:48:24: 
      2024-09-25-06:48:24: [  OK  ] Finished systemd-remount-fs.servicmount Root and Kernel File Systems.
      2024-09-25-06:48:24: 
      2024-09-25-06:48:24: 
      2024-09-25-06:48:24: [  OK  ] Reached target swap.target - Swaps.
      2024-09-25-06:48:24: 
      2024-09-25-06:48:24: 
      2024-09-25-06:48:24:          Mounting sys-fs-fuse-connections.mount - FUSE Control File System...
      2024-09-25-06:48:24: 
      2024-09-25-06:48:24: 
      2024-09-25-06:48:24: systemd[1]: Starting systemd-remount-fs.service - Remount Root and Kernel File Systems...
      2024-09-25-06:48:24: systemd[1]: systemd-tpm2-setup-early.service - Early TPM SRK Setup was skipped because of an unmet condition check (ConditionSecurity=measured-uki).
      2024-09-25-06:48:24: systemd[1]: Starting systemd-udev-load-credentials.service - Load udev Rules from Credentials...
      2024-09-25-06:48:24: watchdog: BUG: soft lockup - CPU#2 stuck for 24s! [systemd:1]
      2024-09-25-06:48:24: Modules linked in: fuse loop nfnetlink xfs nvme_tcp nvme_keyring sd_mod nvme_fabrics ahci nvme_core bochs libahci drm_vram_helper crct10dif_pclmul crc32_pclmul crc32c_intel nvme_auth virtio_net drm_ttm_helper libata net_failover ttm ghash_clmulni_intel virtio_scsi failover serio_raw dm_mirror dm_region_hash dm_log be2iscsi iscsi_boot_sysfs cxgb4i cxgb4 tls libcxgbi libcxgb iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi dm_multipath dm_mod
      2024-09-25-06:48:24: CPU: 2 UID: 0 PID: 1 Comm: systemd Not tainted 6.11.0-25.el10.x86_64 #1
      2024-09-25-06:48:24: Hardware name: Red Hat KVM/RHEL, BIOS edk2-20240524-6.el10 05/24/2024
      2024-09-25-06:48:24: RIP: 0010:_raw_spin_unlock_irqrestore+0x1f/0x30
      2024-09-25-06:48:24: Code: 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 0f 1f 44 00 00 e8 82 07 00 00 90 f7 c6 00 02 00 00 74 01 fb 65 ff 0d 89 d0 c3 73 <74> 05 c3 cc cc cc cc 0f 1f 44 00 00 c3 cc cc cc cc 90 90 90 90 90
      2024-09-25-06:48:24: RSP: 0018:ffffa5a24001b8f8 EFLAGS: 00000286
      2024-09-25-06:48:24: RAX: 0000000000000001 RBX: ffffffff8e5bd060 RCX: 0000000000000000
      2024-09-25-06:48:24: RDX: 0000000000000002 RSI: 0000000000000246 RDI: ffffffff8e5bd060
      2024-09-25-06:48:24: RBP: ffff8efbaf943100 R08: 0000000000000002 R09: 0000000000000001
      2024-09-25-06:48:24: R10: ffffffff8d721be8 R11: 0000000000000003 R12: ffff8efb80294d40
      2024-09-25-06:48:24: R13: 0000000000001002 R14: ffff8efb811b0000 R15: ffff8efb81593750
      2024-09-25-06:48:24: FS:  00007f94dc448440(0000) GS:ffff8efbfbd00000(0000) knlGS:0000000000000000
      2024-09-25-06:48:24: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
      2024-09-25-06:48:24: CR2: 00007f0a1bc51080 CR3: 00000001026c6003 CR4: 0000000000170ef0
      2024-09-25-06:48:24: Call Trace:
      2024-09-25-06:48:24:  <IRQ>
      2024-09-25-06:48:24:  ? show_trace_log_lvl+0x1b0/0x2f0
      2024-09-25-06:48:24:  ? show_trace_log_lvl+0x1b0/0x2f0
      2024-09-25-06:48:24:  ? uart_dtr_rts+0x86/0x90
      2024-09-25-06:48:24:  ? watchdog_timer_fn.cold+0x3d/0xa0
      2024-09-25-06:48:24:  ? __pfx_watchdog_timer_fn+0x10/0x10
      2024-09-25-06:48:24:  ? __hrtimer_run_queues+0x13c/0x2a0
      2024-09-25-06:48:24:  ? hrtimer_interrupt+0xfa/0x210
      2024-09-25-06:48:24:  ? __sysvec_apic_timer_interrupt+0x55/0x100
      2024-09-25-06:48:24:  ? sysvec_apic_timer_interrupt+0x6c/0x90
      2024-09-25-06:48:24:  </IRQ>
      2024-09-25-06:48:24:  <TASK>
      2024-09-25-06:48:24:  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
      2024-09-25-06:48:24:  ? _raw_spin_unlock_irqrestore+0x1f/0x30
      2024-09-25-06:48:24:  ? _raw_spin_unlock_irqrestore+0xe/0x30
      2024-09-25-06:48:24:  uart_dtr_rts+0x86/0x90
      2024-09-25-06:48:24:  tty_port_block_til_ready+0x27d/0x330
      2024-09-25-06:48:24:  ? __pfx_autoremove_wake_function+0x10/0x10
      2024-09-25-06:48:24:  uart_open+0x1e/0x30
      2024-09-25-06:48:24:  tty_open+0x161/0x6a0
      2024-09-25-06:48:24:  chrdev_open+0xcb/0x240
      2024-09-25-06:48:24:  ? __pfx_chrdev_open+0x10/0x10
      2024-09-25-06:48:24:  do_dentry_open+0x25a/0x490
      2024-09-25-06:48:24:  vfs_open+0x34/0xf0
      2024-09-25-06:48:24:  do_open+0x165/0x3d0
      2024-09-25-06:48:24:  path_openat+0x124/0x2d0
      2024-09-25-06:48:24:  ? pick_next_task+0x50/0xb10
      2024-09-25-06:48:24:  do_filp_open+0xc4/0x170
      2024-09-25-06:48:24:  do_sys_openat2+0xae/0xe0
      2024-09-25-06:48:24:  __x64_sys_openat+0x55/0xa0
      2024-09-25-06:48:24:  do_syscall_64+0x7d/0x160
      2024-09-25-06:48:24:  ? __do_sys_newfstat+0x68/0x70
      2024-09-25-06:48:24:  ? __memcg_slab_free_hook+0x100/0x150
      2024-09-25-06:48:24:  ? __x64_sys_close+0x3c/0x80
      2024-09-25-06:48:24:  ? kmem_cache_free+0x3d5/0x400
      2024-09-25-06:48:24:  ? syscall_exit_to_user_mode+0x10/0x1f0
      2024-09-25-06:48:24:  ? do_syscall_64+0x89/0x160
      2024-09-25-06:48:24:  ? filp_flush+0x56/0x70
      2024-09-25-06:48:24:  ? syscall_exit_to_user_mode+0x10/0x1f0
      2024-09-25-06:48:24:  ? do_syscall_64+0x89/0x160
      2024-09-25-06:48:24:  ? syscall_exit_to_user_mode+0x10/0x1f0
      2024-09-25-06:48:24:  ? do_syscall_64+0x89/0x160
      2024-09-25-06:48:24:  ? do_syscall_64+0x89/0x160
      2024-09-25-06:48:24:  ? exc_page_fault+0x73/0x160
      2024-09-25-06:48:24:  entry_SYSCALL_64_after_hwframe+0x76/0x7e
      2024-09-25-06:48:24: RIP: 0033:0x7f94dbb20210
      2024-09-25-06:48:24: Code: 83 e2 40 75 55 89 f0 f7 d0 a9 00 00 41 00 74 4a 80 3d 43 de 0d 00 00 74 6e 89 da 48 89 ee bf 9c ff ff ff b8 01 01 00 00 0f 05 <48> 3d 00 f0 ff ff 0f 87 94 00 00 00 48 8b 54 24 28 64 48 2b 14 25
      2024-09-25-06:48:24: RSP: 002b:00007ffc30704f80 EFLAGS: 00000202 ORIG_RAX: 0000000000000101
      2024-09-25-06:48:24: RAX: ffffffffffffffda RBX: 0000000000080101 RCX: 00007f94dbb20210
      2024-09-25-06:48:24: RDX: 0000000000080101 RSI: 00007f94dc37a5d6 RDI: 00000000ffffff9c
      2024-09-25-06:48:24: RBP: 00007f94dc37a5d6 R08: 0000000000000073 R09: 00000000ffffffff
      2024-09-25-06:48:24: R10: 0000000000000000 R11: 0000000000000202 R12: 0000000000080101
      2024-09-25-06:48:24: R13: 00007f94dc37a5d6 R14: 00007ffc30705000 R15: 000055f8c591def0
      2024-09-25-06:48:24:  </TASK>
      2024-09-25-06:48:24: watchdog: BUG: soft lockup - CPU#0 stuck for 24s! [plymouthd:618]
      2024-09-25-06:48:24: Modules linked in: fuse loop nfnetlink xfs nvme_tcp nvme_keyring sd_mod nvme_fabrics ahci nvme_core bochs libahci drm_vram_helper crct10dif_pclmul crc32_pclmul crc32c_intel nvme_auth virtio_net drm_ttm_helper libata net_failover ttm ghash_clmulni_intel virtio_scsi failover serio_raw dm_mirror dm_region_hash dm_log be2iscsi iscsi_boot_sysfs cxgb4i cxgb4 tls libcxgbi libcxgb iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi dm_multipath dm_mod
      2024-09-25-06:48:24: CPU: 0 UID: 0 PID: 618 Comm: plymouthd Tainted: G             L    -------  ---  6.11.0-25.el10.x86_64 #1
      2024-09-25-06:48:24: Tainted: [L]=SOFTLOCKUP
      2024-09-25-06:48:24: Hardware name: Red Hat KVM/RHEL, BIOS edk2-20240524-6.el10 05/24/2024
      2024-09-25-06:48:24: RIP: 0010:_raw_spin_unlock_irqrestore+0x1f/0x30
      2024-09-25-06:48:24: Code: 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 0f 1f 44 00 00 e8 82 07 00 00 90 f7 c6 00 02 00 00 74 01 fb 65 ff 0d 89 d0 c3 73 <74> 05 c3 cc cc cc cc 0f 1f 44 00 00 c3 cc cc cc cc 90 90 90 90 90
      2024-09-25-06:48:24: RSP: 0018:ffffa5a2403a3be0 EFLAGS: 00000286
      2024-09-25-06:48:24: RAX: 0000000000000001 RBX: 0000000000000e75 RCX: 0000000000000000
      2024-09-25-06:48:24: RDX: 0000000000000000 RSI: 0000000000000286 RDI: ffffffff8e5bd060
      2024-09-25-06:48:24: RBP: ffffffff8e5bd060 R08: 0000000000000000 R09: 0000000000000001
      2024-09-25-06:48:24: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000286
      2024-09-25-06:48:24: R13: ffffa5a2403d1000 R14: ffff8efb815ec000 R15: 00000000000000a7
      2024-09-25-06:48:24: FS:  00007fd043521280(0000) GS:ffff8efbfbc00000(0000) knlGS:0000000000000000
      2024-09-25-06:48:24: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
      2024-09-25-06:48:24: CR2: 00007f32e1aee080 CR3: 00000001137b8004 CR4: 0000000000170ef0
      2024-09-25-06:48:24: Call Trace:
      2024-09-25-06:48:24:  <IRQ>
      2024-09-25-06:48:24:  ? show_trace_log_lvl+0x1b0/0x2f0
      2024-09-25-06:48:24:  ? show_trace_log_lvl+0x1b0/0x2f0
      2024-09-25-06:48:24:  ? uart_write_room+0x74/0x190
      2024-09-25-06:48:24:  ? watchdog_timer_fn.cold+0x3d/0xa0
      2024-09-25-06:48:24:  ? __pfx_watchdog_timer_fn+0x10/0x10
      2024-09-25-06:48:24:  ? __hrtimer_run_queues+0x13c/0x2a0
      2024-09-25-06:48:24:  ? hrtimer_interrupt+0xfa/0x210
      2024-09-25-06:48:24:  ? __sysvec_apic_timer_interrupt+0x55/0x100
      2024-09-25-06:48:24:  ? sysvec_apic_timer_interrupt+0x6c/0x90
      2024-09-25-06:48:24:  </IRQ>
      2024-09-25-06:48:24:  <TASK>
      2024-09-25-06:48:24:  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
      2024-09-25-06:48:24:  ? _raw_spin_unlock_irqrestore+0x1f/0x30
      2024-09-25-06:48:24:  uart_write_room+0x74/0x190
      2024-09-25-06:48:24:  process_output_block+0x3a/0x1c0
      2024-09-25-06:48:24:  ? add_wait_queue+0x7e/0xb0
      2024-09-25-06:48:24:  n_tty_write+0x1a4/0x3c0
      2024-09-25-06:48:24:  ? __pfx_woken_wake_function+0x10/0x10
      2024-09-25-06:48:24:  iterate_tty_write+0x114/0x240
      2024-09-25-06:48:24:  file_tty_write.isra.0+0x86/0xb0
      2024-09-25-06:48:24:  vfs_write+0x294/0x460
      2024-09-25-06:48:24:  ksys_write+0x6d/0xf0
      2024-09-25-06:48:24:  do_syscall_64+0x7d/0x160
      2024-09-25-06:48:24:  ? syscall_exit_to_user_mode+0x10/0x1f0
      2024-09-25-06:48:24:  ? do_syscall_64+0x89/0x160
      2024-09-25-06:48:24:  ? ksys_read+0x6d/0xf0
      2024-09-25-06:48:24:  ? syscall_exit_to_user_mode+0x10/0x1f0
      2024-09-25-06:48:24:  ? do_syscall_64+0x89/0x160
      2024-09-25-06:48:24:  entry_SYSCALL_64_after_hwframe+0x76/0x7e
      2024-09-25-06:48:24: RIP: 0033:0x7fd0437ac6a4
      2024-09-25-06:48:24: Code: c7 00 16 00 00 00 b8 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 80 3d a5 c9 0d 00 00 74 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 48 83 ec 28 48 89 54 24 18 48
      2024-09-25-06:48:24: RSP: 002b:00007ffeb2d85198 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
      2024-09-25-06:48:24: RAX: ffffffffffffffda RBX: 0000563fd8c56110 RCX: 00007fd0437ac6a4
      2024-09-25-06:48:24: RDX: 00000000000000a7 RSI: 0000563fd8c81f60 RDI: 000000000000000a
      2024-09-25-06:48:24: RBP: 00000000000000a7 R08: 0000563fd8c844c0 R09: 00000000000000d0
      2024-09-25-06:48:24: R10: 0000000000000009 R11: 0000000000000202 R12: 0000563fd8c890c0
      2024-09-25-06:48:24: R13: 0000563fd8c80640 R14: 0000563fcc03e4c2 R15: 0000563fd8c89180
      2024-09-25-06:48:24:  </TASK>
      2024-09-25-06:48:24: watchdog: BUG: soft lockup - CPU#1 stuck for 24s! [kworker/1:2:301]
      2024-09-25-06:48:24: Modules linked in: fuse loop nfnetlink xfs nvme_tcp nvme_keyring sd_mod nvme_fabrics ahci nvme_core bochs libahci drm_vram_helper crct10dif_pclmul crc32_pclmul crc32c_intel nvme_auth virtio_net drm_ttm_helper libata net_failover ttm ghash_clmulni_intel virtio_scsi failover serio_raw dm_mirror dm_region_hash dm_log be2iscsi iscsi_boot_sysfs cxgb4i cxgb4 tls libcxgbi libcxgb iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi dm_multipath dm_mod
      2024-09-25-06:48:24: CPU: 1 UID: 0 PID: 301 Comm: kworker/1:2 Tainted: G             L    -------  ---  6.11.0-25.el10.x86_64 #1
      2024-09-25-06:48:24: Tainted: [L]=SOFTLOCKUP
      2024-09-25-06:48:24: Hardware name: Red Hat KVM/RHEL, BIOS edk2-20240524-6.el10 05/24/2024
      2024-09-25-06:48:24: Workqueue: pm pm_runtime_work
      2024-09-25-06:48:24: RIP: 0010:_raw_spin_unlock_irqrestore+0x1f/0x30
      2024-09-25-06:48:24: Code: 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 0f 1f 44 00 00 e8 82 07 00 00 90 f7 c6 00 02 00 00 74 01 fb 65 ff 0d 89 d0 c3 73 <74> 05 c3 cc cc cc cc 0f 1f 44 00 00 c3 cc cc cc cc 90 90 90 90 90
      2024-09-25-06:48:24: RSP: 0018:ffffa5a2405c3d50 EFLAGS: 00000286
      2024-09-25-06:48:24: RAX: 0000000000000001 RBX: ffffffff8e5bd060 RCX: 0000000000000000
      2024-09-25-06:48:24: RDX: 0000000000000001 RSI: 0000000000000246 RDI: ffffffff8e5bd060
      2024-09-25-06:48:24: RBP: ffff8efb838ecc00 R08: 0000000000000001 R09: 0000000000000001
      2024-09-25-06:48:24: R10: 0000000000000003 R11: 0000000000000000 R12: 0000000000000246
      2024-09-25-06:48:24: R13: 0000000000000001 R14: 0000000000000000 R15: ffff8efb901499c0
      2024-09-25-06:48:24: FS:  0000000000000000(0000) GS:ffff8efbfbc80000(0000) knlGS:0000000000000000
      2024-09-25-06:48:24: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
      2024-09-25-06:48:24: CR2: 00005618866fa9d0 CR3: 00000001113c4005 CR4: 0000000000170ef0
      2024-09-25-06:48:24: Call Trace:
      2024-09-25-06:48:24:  <IRQ>
      2024-09-25-06:48:24:  ? show_trace_log_lvl+0x1b0/0x2f0
      2024-09-25-06:48:24:  ? show_trace_log_lvl+0x1b0/0x2f0
      2024-09-25-06:48:24:  ? serial_port_runtime_suspend+0x12b/0x230
      2024-09-25-06:48:24:  ? watchdog_timer_fn.cold+0x3d/0xa0
      2024-09-25-06:48:24:  ? __pfx_watchdog_timer_fn+0x10/0x10
      2024-09-25-06:48:24:  ? __hrtimer_run_queues+0x13c/0x2a0
      2024-09-25-06:48:24:  ? hrtimer_interrupt+0xfa/0x210
      2024-09-25-06:48:24:  ? __sysvec_apic_timer_interrupt+0x55/0x100
      2024-09-25-06:48:24:  ? sysvec_apic_timer_interrupt+0x6c/0x90
      2024-09-25-06:48:24:  </IRQ>
      2024-09-25-06:48:24:  <TASK>
      2024-09-25-06:48:24:  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
      2024-09-25-06:48:24:  ? _raw_spin_unlock_irqrestore+0x1f/0x30
      2024-09-25-06:48:24:  serial_port_runtime_suspend+0x12b/0x230
      2024-09-25-06:48:24:  ? __pfx_serial_port_runtime_suspend+0x10/0x10
      2024-09-25-06:48:24:  __rpm_callback+0x44/0x170
      2024-09-25-06:48:24:  ? __pfx_serial_port_runtime_suspend+0x10/0x10
      2024-09-25-06:48:24:  rpm_callback+0x55/0x60
      2024-09-25-06:48:24:  ? __pfx_serial_port_runtime_suspend+0x10/0x10
      2024-09-25-06:48:24:  rpm_suspend+0xe6/0x5f0
      2024-09-25-06:48:24:  ? _raw_spin_unlock+0xe/0x30
      2024-09-25-06:48:24:  ? finish_task_switch.isra.0+0x99/0x2b0
      2024-09-25-06:48:24:  pm_runtime_work+0x98/0xb0
      2024-09-25-06:48:24:  process_one_work+0x177/0x330
      2024-09-25-06:48:24:  worker_thread+0x252/0x390
      2024-09-25-06:48:24:  ? __pfx_worker_thread+0x10/0x10
      2024-09-25-06:48:24:  kthread+0xd2/0x100
      2024-09-25-06:48:24:  ? __pfx_kthread+0x10/0x10
      2024-09-25-06:48:24:  ret_from_fork+0x34/0x50
      2024-09-25-06:48:24:  ? __pfx_kthread+0x10/0x10
      2024-09-25-06:48:24:  ret_from_fork_asm+0x1a/0x30
      2024-09-25-06:48:24:  </TASK>
      2024-09-25-06:48:24: systemd-journald[916]: Collecting audit messages is disabled.
      2024-09-25-06:48:24: systemd[1]: Starting systemd-udev-trigger.service - Coldplug All udev Devices...
      2024-09-25-06:48:24: systemd[1]: Activated swap dev-disk-by\x2duuid-794cb060\x2d21a8\x2d434a\x2d91e6\x2d83e0369e1296.swap - /dev/disk/by-uuid/794cb060-21a8-434a-91e6-83e0369e1296.
      2024-09-25-06:48:24: systemd[1]: Started systemd-journald.service - Journal Service.
      

      For case2:
      Boot the VM also with '-S', cont VM and then start migration:

      Sending qmp command   : {"execute": "migrate-incoming", "arguments": {"uri": "tcp:[::]:4000"}, "id": "R0NhJlXA"}Responding qmp command: {"return": {}, "id": "R0NhJlXA"}
      Sending qmp command   : {"execute": "cont", "id": "ByzCEP23"}
      Responding qmp command: {"return": {}, "id": "ByzCEP23"}
      onds": 1727269941, "microseconds": 612960}, "event": "RTC_CHANGE", "data": {"offset": -1, "qom-path": "/machine/unattached/device[6]/rtc"}}
      Sending qmp command   : {"execute": "query-status", "id": "n5TqS8U3"}
      Responding qmp command: {"return": {"status": "running", "running": true}, "id": "n5TqS8U3"}
      Sending qmp command   : {"execute": "query-version", "id": "RWJZBqxm"}
      Responding qmp command: {"return": {"qemu": {"micro": 0, "minor": 1, "major": 9}, "package": "qemu-kvm-9.1.0-1.el10"}, "id": "RWJZBqxm"}
      Sending qmp command   : {"execute": "migrate", "arguments": {"uri": "tcp:10.72.136.64:4000", "detach": true, "resume": false}, "id": "Dsk5HErl"}
      

      After migration, found VM call trace:

      2024-09-25-09:16:08: [  163.777468] watchdog: BUG: soft lockup - CPU#3 stuck for 37s! [khugepaged:57]
      2024-09-25-09:16:08: [  163.777503] Modules linked in: nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 rfkill nf_tables qrtr vfat fat intel_rapl_msr intel_rapl_common kvm_intel kvm iTCO_wdt iTCO_vendor_support rapl pcspkr lpc_ich virtio_balloon i2c_i801 i2c_smbus joydev sg fuse loop nfnetlink xfs nvme_tcp ahci crct10dif_pclmul bochs drm_vram_helper libahci nvme_keyring crc32_pclmul sd_mod nvme_fabrics drm_ttm_helper crc32c_intel nvme_core virtio_scsi virtio_net ttm libata net_failover ghash_clmulni_intel failover nvme_auth serio_raw dm_mirror dm_region_hash dm_log be2iscsi iscsi_boot_sysfs cxgb4i cxgb4 tls libcxgbi libcxgb iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi dm_multipath dm_mod
      2024-09-25-09:16:08: [  163.777595] CPU: 3 UID: 0 PID: 57 Comm: khugepaged Kdump: loaded Not tainted 6.11.0-25.el10.x86_64 #1
      2024-09-25-09:16:08: [  163.777602] Hardware name: Red Hat KVM/RHEL, BIOS edk2-20240524-6.el10 05/24/2024
      2024-09-25-09:16:08: [  163.777607] RIP: 0010:smp_call_function_many_cond+0x112/0x4c0
      2024-09-25-09:16:08: [  163.777653] Code: d0 e8 e2 b1 4a 00 3b 05 4c 2e 1a 02 73 26 48 63 d0 49 8b 34 24 48 03 34 d5 60 7d b3 85 8b 56 08 83 e2 01 74 0a f3 90 8b 4e 08 <83> e1 01 75 f6 83 c0 01 eb bf 48 83 c4 30 5b 5d 41 5c 41 5d 41 5e
      2024-09-25-09:16:08: [  163.777655] RSP: 0018:ffffb9e0001f7c18 EFLAGS: 00000202
      2024-09-25-09:16:08: [  163.777658] RAX: 0000000000000001 RBX: 0000000000000000 RCX: 0000000000000011
      2024-09-25-09:16:08: [  163.777663] RDX: 0000000000000001 RSI: ffff9ea7fbcbd300 RDI: ffff9ea780054500
      2024-09-25-09:16:08: [  163.777665] RBP: 0000000000000000 R08: 0000000000000001 R09: 00000000fffffffc
      2024-09-25-09:16:08: [  163.777667] R10: ffff9ea7800542d0 R11: 0000000000000000 R12: ffff9ea7fbdb6c40
      2024-09-25-09:16:08: [  163.777668] R13: ffff9ea7800542d0 R14: 0000000000000003 R15: 0000000000000000
      2024-09-25-09:16:08: [  163.777670] FS:  0000000000000000(0000) GS:ffff9ea7fbd80000(0000) knlGS:0000000000000000
      2024-09-25-09:16:08: [  163.777675] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
      2024-09-25-09:16:08: [  163.777690] CR2: 00007f20d4c0a0f8 CR3: 0000000063222004 CR4: 0000000000170ef0
      2024-09-25-09:16:08: [  163.777697] Call Trace:
      2024-09-25-09:16:08: [  163.777706]  <IRQ>
      2024-09-25-09:16:08: [  163.777709]  ? show_trace_log_lvl+0x1b0/0x2f0
      2024-09-25-09:16:08: [  163.777740]  ? show_trace_log_lvl+0x1b0/0x2f0
      2024-09-25-09:16:08: [  163.777762]  ? smp_call_function+0x2c/0x50
      2024-09-25-09:16:08: [  163.777766]  ? watchdog_timer_fn.cold+0x3d/0xa0
      2024-09-25-09:16:08: [  163.777777]  ? __pfx_watchdog_timer_fn+0x10/0x10
      2024-09-25-09:16:08: [  163.777795]  ? __hrtimer_run_queues+0x13c/0x2a0
      2024-09-25-09:16:08: [  163.777810]  ? hrtimer_interrupt+0xfa/0x210
      2024-09-25-09:16:08: [  163.777813]  ? __sysvec_apic_timer_interrupt+0x55/0x100
      2024-09-25-09:16:08: [  163.777830]  ? sysvec_apic_timer_interrupt+0x6c/0x90
      2024-09-25-09:16:08: [  163.777856]  </IRQ>
      2024-09-25-09:16:08: [  163.777857]  <TASK>
      2024-09-25-09:16:08: [  163.777858]  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
      2024-09-25-09:16:08: [  163.777875]  ? smp_call_function_many_cond+0x112/0x4c0
      2024-09-25-09:16:08: [  163.777879]  ? smp_call_function_many_cond+0xee/0x4c0
      2024-09-25-09:16:08: [  163.777882]  ? __pfx_tlb_remove_table_smp_sync+0x10/0x10
      2024-09-25-09:16:08: [  163.777909]  smp_call_function+0x2c/0x50
      2024-09-25-09:16:08: [  163.777913]  collapse_huge_page+0x462/0x790
      2024-09-25-09:16:08: [  163.777923]  hpage_collapse_scan_pmd+0x659/0x710
      2024-09-25-09:16:08: [  163.777927]  khugepaged_scan_mm_slot.constprop.0+0x3ce/0x580
      2024-09-25-09:16:08: [  163.777931]  khugepaged+0x10c/0x200
      2024-09-25-09:16:08: [  163.777934]  ? __pfx_khugepaged+0x10/0x10
      2024-09-25-09:16:08: [  163.777936]  kthread+0xd2/0x100
      2024-09-25-09:16:08: [  163.777950]  ? __pfx_kthread+0x10/0x10
      2024-09-25-09:16:08: [  163.777952]  ret_from_fork+0x34/0x50
      2024-09-25-09:16:08: [  163.777958]  ? __pfx_kthread+0x10/0x10
      2024-09-25-09:16:08: [  163.777959]  ret_from_fork_asm+0x1a/0x30
      2024-09-25-09:16:08: [  163.777972]  </TASK>
      2024-09-25-09:16:08: [  164.152020] rfkill: input handler disabled
      2024-09-25-09:16:08: [  200.134938] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [kworker/3:3:667]
      2024-09-25-09:16:08: [  200.135025] Modules linked in: nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 rfkill nf_tables qrtr vfat fat intel_rapl_msr intel_rapl_common kvm_intel kvm iTCO_wdt iTCO_vendor_support rapl pcspkr lpc_ich virtio_balloon i2c_i801 i2c_smbus joydev sg fuse loop nfnetlink xfs nvme_tcp ahci crct10dif_pclmul bochs drm_vram_helper libahci nvme_keyring crc32_pclmul sd_mod nvme_fabrics drm_ttm_helper crc32c_intel nvme_core virtio_scsi virtio_net ttm libata net_failover ghash_clmulni_intel failover nvme_auth serio_raw dm_mirror dm_region_hash dm_log be2iscsi iscsi_boot_sysfs cxgb4i cxgb4 tls libcxgbi libcxgb iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi dm_multipath dm_mod
      2024-09-25-09:16:08: [  200.135524] CPU: 3 UID: 0 PID: 667 Comm: kworker/3:3 Kdump: loaded Tainted: G             L    -------  ---  6.11.0-25.el10.x86_64 #1
      2024-09-25-09:16:08: [  200.135534] Tainted: [L]=SOFTLOCKUP
      2024-09-25-09:16:08: [  200.135541] Hardware name: Red Hat KVM/RHEL, BIOS edk2-20240524-6.el10 05/24/2024
      2024-09-25-09:16:08: [  200.135551] Workqueue: events netstamp_clear
      2024-09-25-09:16:08: [  200.135612] RIP: 0010:smp_call_function_many_cond+0x10f/0x4c0
      2024-09-25-09:16:08: [  200.135638] Code: 08 48 63 d0 e8 e2 b1 4a 00 3b 05 4c 2e 1a 02 73 26 48 63 d0 49 8b 34 24 48 03 34 d5 60 7d b3 85 8b 56 08 83 e2 01 74 0a f3 90 <8b> 4e 08 83 e1 01 75 f6 83 c0 01 eb bf 48 83 c4 30 5b 5d 41 5c 41
      2024-09-25-09:16:08: [  200.135641] RSP: 0018:ffffb9e00050fd68 EFLAGS: 00000202
      2024-09-25-09:16:08: [  200.135647] RAX: 0000000000000001 RBX: 0000000000000246 RCX: 0000000000000001
      2024-09-25-09:16:08: [  200.135651] RDX: 0000000000000001 RSI: ffff9ea7fbcbd300 RDI: ffff9ea780054500
      2024-09-25-09:16:08: [  200.135653] RBP: 0000000000000000 R08: 0000000000000001 R09: 00000000fffffffc
      2024-09-25-09:16:08: [  200.135654] R10: ffff9ea7800542d0 R11: 0000000000000000 R12: ffff9ea7fbdb6c40
      2024-09-25-09:16:08: [  200.135656] R13: ffff9ea7800542d0 R14: 0000000000000003 R15: 0000000000000001
      2024-09-25-09:16:08: [  200.135658] FS:  0000000000000000(0000) GS:ffff9ea7fbd80000(0000) knlGS:0000000000000000
      2024-09-25-09:16:08: [  200.135662] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
      2024-09-25-09:16:08: [  200.135672] CR2: 0000561403926c98 CR3: 0000000063222006 CR4: 0000000000170ef0
      2024-09-25-09:16:08: [  200.135680] Call Trace:
      2024-09-25-09:16:08: [  200.135695]  <IRQ>
      2024-09-25-09:16:08: [  200.135745]  ? show_trace_log_lvl+0x1b0/0x2f0
      2024-09-25-09:16:08: [  200.135777]  ? show_trace_log_lvl+0x1b0/0x2f0
      2024-09-25-09:16:08: [  200.135781]  ? on_each_cpu_cond_mask+0x24/0x40
      2024-09-25-09:16:08: [  200.135783]  ? watchdog_timer_fn.cold+0x3d/0xa0
      2024-09-25-09:16:08: [  200.135790]  ? __pfx_watchdog_timer_fn+0x10/0x10
      2024-09-25-09:16:08: [  200.135805]  ? __hrtimer_run_queues+0x13c/0x2a0
      2024-09-25-09:16:08: [  200.135817]  ? hrtimer_interrupt+0xfa/0x210
      2024-09-25-09:16:08: [  200.135819]  ? __sysvec_apic_timer_interrupt+0x55/0x100
      2024-09-25-09:16:08: [  200.135836]  ? sysvec_apic_timer_interrupt+0x6c/0x90
      2024-09-25-09:16:08: [  200.135853]  </IRQ>
      2024-09-25-09:16:08: [  200.135854]  <TASK>
      2024-09-25-09:16:08: [  200.135855]  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
      2024-09-25-09:16:08: [  200.135899]  ? smp_call_function_many_cond+0x10f/0x4c0
      2024-09-25-09:16:08: [  200.135902]  ? smp_call_function_many_cond+0xee/0x4c0
      2024-09-25-09:16:08: [  200.135906]  ? __pfx_do_sync_core+0x10/0x10
      2024-09-25-09:16:08: [  200.135922]  on_each_cpu_cond_mask+0x24/0x40
      2024-09-25-09:16:08: [  200.135924]  text_poke_bp_batch+0xc1/0x300
      2024-09-25-09:16:08: [  200.135935]  ? arch_jump_label_transform_queue+0x57/0x80
      2024-09-25-09:16:08: [  200.135947]  text_poke_finish+0x1f/0x30
      2024-09-25-09:16:08: [  200.135950]  arch_jump_label_transform_apply+0x1a/0x30
      2024-09-25-09:16:08: [  200.135958]  static_key_enable_cpuslocked+0x63/0x90
      2024-09-25-09:16:08: [  200.135989]  static_key_enable+0x1a/0x20
      2024-09-25-09:16:08: [  200.135992]  process_one_work+0x177/0x330
      2024-09-25-09:16:08: [  200.136016]  worker_thread+0x252/0x390
      2024-09-25-09:16:08: [  200.136020]  ? __pfx_worker_thread+0x10/0x10
      2024-09-25-09:16:08: [  200.136023]  kthread+0xd2/0x100
      2024-09-25-09:16:08: [  200.136033]  ? __pfx_kthread+0x10/0x10
      2024-09-25-09:16:08: [  200.136035]  ret_from_fork+0x34/0x50
      2024-09-25-09:16:08: [  200.136040]  ? __pfx_kthread+0x10/0x10
      2024-09-25-09:16:08: [  200.136042]  ret_from_fork_asm+0x1a/0x30
      2024-09-25-09:16:08: [  200.136059]  </TASK>
      2024-09-25-09:16:08: [  207.174682] watchdog: BUG: soft lockup - CPU#0 stuck for 25s! [irqbalance:1109]
      2024-09-25-09:16:08: [  207.174711] Modules linked in: nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 rfkill nf_tables qrtr vfat fat intel_rapl_msr intel_rapl_common kvm_intel kvm iTCO_wdt iTCO_vendor_support rapl pcspkr lpc_ich virtio_balloon i2c_i801 i2c_smbus joydev sg fuse loop nfnetlink xfs nvme_tcp ahci crct10dif_pclmul bochs drm_vram_helper libahci nvme_keyring crc32_pclmul sd_mod nvme_fabrics drm_ttm_helper crc32c_intel nvme_core virtio_scsi virtio_net ttm libata net_failover ghash_clmulni_intel failover nvme_auth serio_raw dm_mirror dm_region_hash dm_log be2iscsi iscsi_boot_sysfs cxgb4i cxgb4 tls libcxgbi libcxgb iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi dm_multipath dm_mod
      2024-09-25-09:16:08: [  207.174804] CPU: 0 UID: 0 PID: 1109 Comm: irqbalance Kdump: loaded Tainted: G             L    -------  ---  6.11.0-25.el10.x86_64 #1
      2024-09-25-09:16:08: [  207.174808] Tainted: [L]=SOFTLOCKUP
      2024-09-25-09:16:08: [  207.174810] Hardware name: Red Hat KVM/RHEL, BIOS edk2-20240524-6.el10 05/24/2024
      2024-09-25-09:16:08: [  207.174812] RIP: 0010:_raw_spin_unlock_irqrestore+0x1f/0x30
      2024-09-25-09:16:08: [  207.174824] Code: 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 0f 1f 44 00 00 e8 82 07 00 00 90 f7 c6 00 02 00 00 74 01 fb 65 ff 0d 89 d0 e3 7a <74> 05 c3 cc cc cc cc 0f 1f 44 00 00 c3 cc cc cc cc 90 90 90 90 90
      2024-09-25-09:16:08: [  207.174826] RSP: 0018:ffffb9e000527cc0 EFLAGS: 00000286
      2024-09-25-09:16:08: [  207.174829] RAX: 0000000000000001 RBX: 0000000000000000 RCX: 00000000000003fb
      2024-09-25-09:16:08: [  207.174831] RDX: ffff9ea78cb23000 RSI: 0000000000000246 RDI: ffff9ea780c9a8a4
      2024-09-25-09:16:08: [  207.174833] RBP: ffff9ea798297b40 R08: 000000000000000f R09: 0000000000000004
      2024-09-25-09:16:08: [  207.174834] R10: ffffffffffffffff R11: 0000000000000000 R12: ffff9ea780c9a800
      2024-09-25-09:16:08: [  207.174854] R13: 0000000000000246 R14: ffff9ea780c9a8a4 R15: 0000000000000000
      2024-09-25-09:16:08: [  207.174856] FS:  00007f9cf329e840(0000) GS:ffff9ea7fbc00000(0000) knlGS:0000000000000000
      2024-09-25-09:16:08: [  207.174861] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
      2024-09-25-09:16:08: [  207.174863] CR2: 00007fd21ad62950 CR3: 000000010ca5e003 CR4: 0000000000170ef0
      2024-09-25-09:16:08: [  207.174865] Call Trace:
      2024-09-25-09:16:08: [  207.174885]  <IRQ>
      2024-09-25-09:16:08: [  207.174889]  ? show_trace_log_lvl+0x1b0/0x2f0
      2024-09-25-09:16:08: [  207.174894]  ? show_trace_log_lvl+0x1b0/0x2f0
      2024-09-25-09:16:08: [  207.174898]  ? show_interrupts+0x243/0x2e0
      2024-09-25-09:16:08: [  207.174914]  ? watchdog_timer_fn.cold+0x3d/0xa0
      2024-09-25-09:16:08: [  207.174918]  ? __pfx_watchdog_timer_fn+0x10/0x10
      2024-09-25-09:16:08: [  207.174921]  ? __hrtimer_run_queues+0x13c/0x2a0
      2024-09-25-09:16:08: [  207.174925]  ? hrtimer_interrupt+0xfa/0x210
      2024-09-25-09:16:08: [  207.174928]  ? __sysvec_apic_timer_interrupt+0x55/0x100
      2024-09-25-09:16:08: [  207.174931]  ? sysvec_apic_timer_interrupt+0x6c/0x90
      2024-09-25-09:16:08: [  207.174935]  </IRQ>
      2024-09-25-09:16:08: [  207.174936]  <TASK>
      2024-09-25-09:16:08: [  207.174937]  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
      2024-09-25-09:16:08: [  207.174943]  ? _raw_spin_unlock_irqrestore+0x1f/0x30
      2024-09-25-09:16:08: [  207.174945]  show_interrupts+0x243/0x2e0
      2024-09-25-09:16:08: [  207.174961]  seq_read_iter+0x2b1/0x460
      2024-09-25-09:16:08: [  207.174996]  proc_reg_read_iter+0x53/0x90
      2024-09-25-09:16:08: [  207.175022]  vfs_read+0x299/0x370
      2024-09-25-09:16:08: [  207.175055]  ksys_read+0x6d/0xf0
      2024-09-25-09:16:08: [  207.175059]  do_syscall_64+0x7d/0x160
      2024-09-25-09:16:08: [  207.175078]  ? syscall_exit_work+0xf3/0x120
      2024-09-25-09:16:08: [  207.175088]  ? syscall_exit_to_user_mode+0x10/0x1f0
      2024-09-25-09:16:08: [  207.175091]  ? do_syscall_64+0x89/0x160
      2024-09-25-09:16:08: [  207.175093]  ? syscall_exit_to_user_mode+0x10/0x1f0
      2024-09-25-09:16:08: [  207.175099]  ? do_syscall_64+0x89/0x160
      2024-09-25-09:16:08: [  207.175121]  entry_SYSCALL_64_after_hwframe+0x76/0x7e
      2024-09-25-09:16:08: [  207.175130] RIP: 0033:0x7f9cf3479b7c
      2024-09-25-09:16:08: [  207.175512] Code: ec 28 48 89 54 24 18 48 89 74 24 10 89 7c 24 08 e8 b9 8f f9 ff 48 8b 54 24 18 48 8b 74 24 10 41 89 c0 8b 7c 24 08 31 c0 0f 05 <48> 3d 00 f0 ff ff 77 34 44 89 c7 48 89 44 24 08 e8 0f 90 f9 ff 48
      2024-09-25-09:16:08: [  207.175525] RSP: 002b:00007ffe298ea660 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
      2024-09-25-09:16:08: [  207.175528] RAX: ffffffffffffffda RBX: 0000556fd38d72a0 RCX: 00007f9cf3479b7c
      2024-09-25-09:16:08: [  207.175530] RDX: 0000000000000400 RSI: 0000556fd38d7500 RDI: 0000000000000007
      2024-09-25-09:16:08: [  207.175531] RBP: 00007f9cf354e050 R08: 0000000000000000 R09: 0000000000000000
      2024-09-25-09:16:08: [  207.175533] R10: 00007f9cf34f9fe0 R11: 0000000000000246 R12: 00007f9cf354df00
      2024-09-25-09:16:08: [  207.175534] R13: 0000556fd38d789b R14: 0000000000000000 R15: 0000000000000065
      2024-09-25-09:16:08: [  207.175537]  </TASK>
      2024-09-25-09:16:08: [  207.175541] watchdog: BUG: soft lockup - CPU#1 stuck for 28s! [watchdogd:69]
      2024-09-25-09:16:08: [  207.175578] Modules linked in: nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 rfkill nf_tables qrtr vfat fat intel_rapl_msr intel_rapl_common kvm_intel kvm iTCO_wdt iTCO_vendor_support rapl pcspkr lpc_ich virtio_balloon i2c_i801 i2c_smbus joydev sg fuse loop nfnetlink xfs nvme_tcp ahci crct10dif_pclmul bochs drm_vram_helper libahci nvme_keyring crc32_pclmul sd_mod nvme_fabrics drm_ttm_helper crc32c_intel nvme_core virtio_scsi virtio_net ttm libata net_failover ghash_clmulni_intel failover nvme_auth serio_raw dm_mirror dm_region_hash dm_log be2iscsi iscsi_boot_sysfs cxgb4i cxgb4 tls libcxgbi libcxgb iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi dm_multipath dm_mod
      2024-09-25-09:16:08: [  207.175664] CPU: 1 UID: 0 PID: 69 Comm: watchdogd Kdump: loaded Tainted: G             L    -------  ---  6.11.0-25.el10.x86_64 #1
      2024-09-25-09:16:08: [  207.175668] Tainted: [L]=SOFTLOCKUP
      2024-09-25-09:16:08: [  207.175670] Hardware name: Red Hat KVM/RHEL, BIOS edk2-20240524-6.el10 05/24/2024
      2024-09-25-09:16:08: [  207.175673] RIP: 0010:iTCO_wdt_ping+0x3b/0x80 [iTCO_wdt]
      2024-09-25-09:16:08: [  207.175719] Code: 00 48 8d ab e0 00 00 00 48 89 ef e8 2f 1e 5e c4 83 bb c0 00 00 00 01 76 22 48 8b 83 c8 00 00 00 48 8b 10 b8 01 00 00 00 66 ef <48> 89 ef e8 0d 1f 5e c4 31 c0 5b 5d c3 cc cc cc cc 75 ed 48 8b 83
      2024-09-25-09:16:08: [  207.175739] RSP: 0018:ffffb9e000257e60 EFLAGS: 00000202
      2024-09-25-09:16:08: [  207.175742] RAX: 0000000000000001 RBX: ffff9ea780caf228 RCX: 0000000000139b4f
      2024-09-25-09:16:08: [  207.175744] RDX: 0000000000000660 RSI: ffff9ea799ce5bd8 RDI: ffff9ea780caf308
      2024-09-25-09:16:08: [  207.175746] RBP: ffff9ea780caf308 R08: 0000000000000000 R09: 0000000000000000
      2024-09-25-09:16:08: [  207.175748] R10: 00000000f5257d14 R11: 0000000000000002 R12: ffff9ea799ce5800
      2024-09-25-09:16:08: [  207.175750] R13: ffff9ea780d28444 R14: 0000000000000001 R15: ffffffff84e26450
      2024-09-25-09:16:08: [  207.175752] FS:  0000000000000000(0000) GS:ffff9ea7fbc80000(0000) knlGS:0000000000000000
      2024-09-25-09:16:08: [  207.175757] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
      2024-09-25-09:16:08: [  207.175759] CR2: 00007fd38e040000 CR3: 000000010ca4c003 CR4: 0000000000170ef0
      2024-09-25-09:16:08: [  207.175761] Call Trace:
      2024-09-25-09:16:08: [  207.175776]  <IRQ>
      2024-09-25-09:16:08: [  207.175782]  ? show_trace_log_lvl+0x1b0/0x2f0
      2024-09-25-09:16:08: [  207.175790]  ? show_trace_log_lvl+0x1b0/0x2f0
      2024-09-25-09:16:08: [  207.175794]  ? __watchdog_ping+0x50/0x120
      2024-09-25-09:16:08: [  207.175871]  ? watchdog_timer_fn.cold+0x3d/0xa0
      2024-09-25-09:16:08: [  207.175875]  ? __pfx_watchdog_timer_fn+0x10/0x10
      2024-09-25-09:16:08: [  207.175878]  ? __hrtimer_run_queues+0x13c/0x2a0
      2024-09-25-09:16:08: [  207.175883]  ? hrtimer_interrupt+0xfa/0x210
      2024-09-25-09:16:08: [  207.175885]  ? __sysvec_apic_timer_interrupt+0x55/0x100
      2024-09-25-09:16:08: [  207.175889]  ? sysvec_apic_timer_interrupt+0x6c/0x90
      2024-09-25-09:16:08: [  207.175893]  </IRQ>
      2024-09-25-09:16:08: [  207.175894]  <TASK>
      2024-09-25-09:16:08: [  207.175895]  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
      2024-09-25-09:16:08: [  207.175898]  ? __pfx_watchdog_ping_work+0x10/0x10
      2024-09-25-09:16:08: [  207.175905]  ? iTCO_wdt_ping+0x3b/0x80 [iTCO_wdt]
      2024-09-25-09:16:08: [  207.175907]  __watchdog_ping+0x50/0x120
      2024-09-25-09:16:08: [  207.175942]  watchdog_ping_work+0x44/0x50
      2024-09-25-09:16:08: [  207.175954]  kthread_worker_fn+0xa1/0x220
      2024-09-25-09:16:08: [  207.175969]  ? __pfx_kthread_worker_fn+0x10/0x10
      2024-09-25-09:16:08: [  207.175972]  kthread+0xd2/0x100
      2024-09-25-09:16:08: [  207.175974]  ? __pfx_kthread+0x10/0x10
      2024-09-25-09:16:08: [  207.175976]  ret_from_fork+0x34/0x50
      2024-09-25-09:16:08: [  207.175980]  ? __pfx_kthread+0x10/0x10
      2024-09-25-09:16:08: [  207.175982]  ret_from_fork_asm+0x1a/0x30
      2024-09-25-09:16:08: [  207.175987]  </TASK>
      

      Expected results

      No call trace in VM.

      Actual results

      The qemu command lines of two cases are almost same, case 2 appends "-S":

      /usr/libexec/qemu-kvm  \
      -name "mouse-vm" \
      -sandbox on \
      -machine q35,memory-backend=pc.ram,pflash0=drive_ovmf_code,pflash1=drive_ovmf_vars \
      -cpu Haswell-noTSX-IBRS,vmx=on,pdcm=on,f16c=on,rdrand=on,hypervisor=on,vme=on,ss=on,arat=on,tsc-adjust=on,umip=on,md-clear=on,stibp=on,flush-l1d=on,arch-capabilities=on,ssbd=on,xsaveopt=on,abm=on,pdpe1gb=on,invtsc=on,ibpb=on,ibrs=on,amd-stibp=on,amd-ssbd=on,skip-l1dfl-vmentry=on,pschange-mc-no=on,gds-no=on,vmx-ins-outs=on,vmx-true-ctls=on,vmx-store-lma=on,vmx-activity-hlt=on,vmx-activity-wait-sipi=on,vmx-vmwrite-vmexit-fields=on,vmx-apicv-xapic=on,vmx-ept=on,vmx-desc-exit=on,vmx-rdtscp-exit=on,vmx-apicv-x2apic=on,vmx-vpid=on,vmx-wbinvd-exit=on,vmx-unrestricted-guest=on,vmx-apicv-register=on,vmx-apicv-vid=on,vmx-rdrand-exit=on,vmx-invpcid-exit=on,vmx-vmfunc=on,vmx-shadow-vmcs=on,vmx-pml=on,vmx-ept-execonly=on,vmx-page-walk-4=on,vmx-ept-2mb=on,vmx-ept-1gb=on,vmx-invept=on,vmx-eptad=on,vmx-invept-single-context=on,vmx-invept-all-context=on,vmx-invvpid=on,vmx-invvpid-single-addr=on,vmx-invvpid-all-context=on,vmx-intr-exit=on,vmx-nmi-exit=on,vmx-vnmi=on,vmx-preemption-timer=on,vmx-posted-intr=on,vmx-vintr-pending=on,vmx-tsc-offset=on,vmx-hlt-exit=on,vmx-invlpg-exit=on,vmx-mwait-exit=on,vmx-rdpmc-exit=on,vmx-rdtsc-exit=on,vmx-cr3-load-noexit=on,vmx-cr3-store-noexit=on,vmx-cr8-load-exit=on,vmx-cr8-store-exit=on,vmx-flexpriority=on,vmx-vnmi-pending=on,vmx-movdr-exit=on,vmx-io-exit=on,vmx-io-bitmap=on,vmx-mtf=on,vmx-msr-bitmap=on,vmx-monitor-exit=on,vmx-pause-exit=on,vmx-secondary-ctls=on,vmx-exit-nosave-debugctl=on,vmx-exit-load-perf-global-ctrl=on,vmx-exit-ack-intr=on,vmx-exit-save-pat=on,vmx-exit-load-pat=on,vmx-exit-save-efer=on,vmx-exit-load-efer=on,vmx-exit-save-preemption-timer=on,vmx-entry-noload-debugctl=on,vmx-entry-ia32e-mode=on,vmx-entry-load-perf-global-ctrl=on,vmx-entry-load-pat=on,vmx-entry-load-efer=on,vmx-eptp-switching=on,tsc-frequency=1899996000,host-phys-bits=on,host-phys-bits-limit=46 \
      -nodefaults  \
      -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpmonitor1,server=on,wait=off \
      -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/monitor-catch_monitor,server=on,wait=off \
      -mon chardev=qmp_id_qmpmonitor1,mode=control \
      -mon chardev=qmp_id_catch_monitor,mode=control \
      -device '{"driver":"pcie-root-port","id":"root0","multifunction":true,"bus":"pcie.0","addr":"0x2","chassis":1}' \
      -device '{"driver":"pcie-root-port","id":"root1","port":11,"addr":"0x2.0x1","bus":"pcie.0","chassis":2}' \
      -device '{"driver":"pcie-root-port","id":"root2","port":12,"addr":"0x2.0x2","bus":"pcie.0","chassis":3}' \
      -device '{"driver":"pcie-root-port","id":"root3","port":13,"addr":"0x2.0x3","bus":"pcie.0","chassis":4}' \
      -device '{"driver":"pcie-root-port","id":"root4","port":14,"addr":"0x2.0x4","bus":"pcie.0","chassis":5}' \
      -device '{"driver":"pcie-root-port","id":"root5","port":15,"addr":"0x2.0x5","bus":"pcie.0","chassis":6}' \
      -device '{"driver":"pcie-root-port","id":"root6","port":16,"addr":"0x2.0x6","bus":"pcie.0","chassis":7}' \
      -device '{"driver":"pcie-root-port","id":"root7","port":17,"addr":"0x2.0x7","bus":"pcie.0","chassis":8}' \
      -device '{"driver":"pcie-root-port","id":"extra_root0","multifunction":true,"bus":"pcie.0","addr":"0x3","chassis":21}' \
      -device '{"driver":"pcie-root-port","id":"extra_root1","port":21,"addr":"0x3.0x1","bus":"pcie.0","chassis":22}' \
      -device '{"driver":"pcie-root-port","id":"extra_root2","port":22,"addr":"0x3.0x2","bus":"pcie.0","chassis":23}' \
      -device '{"driver":"nec-usb-xhci","id":"usb1","bus":"root0","addr":"0x0"}' \
      -device '{"driver":"virtio-scsi-pci","id":"virtio_scsi_pci0","bus":"root1","addr":"0x0"}' \
      -device '{"driver":"scsi-hd","id":"image1","device_id":"drive-image1","drive":"drive_image1","bus":"virtio_scsi_pci0.0","channel":0,"scsi-id":0,"lun":0,"bootindex":0,"write-cache":"on"}' \
      -device '{"driver":"virtio-net-pci","mac":"9a:8a:8b:8c:8d:8e","id":"net0","netdev":"tap0","bus":"root2","addr":"0x0"}' \
      -device '{"driver":"usb-tablet","id":"usb-tablet1","bus":"usb1.0","port":"1"}' \
      -device '{"driver":"virtio-balloon-pci","id":"balloon0","bus":"root3","addr":"0x0"}' \
      -device '{"driver":"VGA","id":"video0","vgamem_mb":16,"bus":"pcie.0","addr":"0x1"}' \
      -blockdev '{"driver":"file","auto-read-only":true,"discard":"unmap","aio":"threads","cache":{"direct":true,"no-flush":false},"filename":"/mnt/xiaohli/rhel100-64-virtio-scsi-ovmf.qcow2","node-name":"drive_sys1"}' \
      -blockdev '{"driver":"qcow2","node-name":"drive_image1","read-only":false,"cache":{"direct":true,"no-flush":false},"file":"drive_sys1"}' \
      -blockdev '{"node-name":"file_ovmf_code","driver":"file","filename":"/usr/share/OVMF/OVMF_CODE.secboot.fd","auto-read-only":true,"discard":"unmap"}' \
      -blockdev '{"node-name":"drive_ovmf_code","driver":"raw","read-only":true,"file":"file_ovmf_code"}' \
      -blockdev '{"node-name":"file_ovmf_vars","driver":"file","filename":"/mnt/xiaohli/rhel100-64-virtio-scsi-ovmf.qcow2_VARS.fd","auto-read-only":true,"discard":"unmap"}' \
      -blockdev '{"node-name":"drive_ovmf_vars","driver":"raw","read-only":false,"file":"file_ovmf_vars"}' \
      -netdev tap,id=tap0,vhost=on \
      -m 4096 \
      -object '{"qom-type":"memory-backend-ram","id":"pc.ram","size":4294967296}' \
      -smp 4,maxcpus=4,cores=2,threads=1,sockets=2 \
      -vnc :10 \
      -rtc base=utc,clock=host \
      -boot menu=off,strict=off,order=cdn,once=c \
      -enable-kvm  \
      -qmp tcp:0:3333,server=on,wait=off \
      -qmp tcp:0:9999,server=on,wait=off \
      -qmp tcp:0:9888,server=on,wait=off \
      -serial tcp:0:4444,server=on,wait=off \
      -monitor stdio \
      -msg timestamp=on \
      

              jsuvorov@redhat.com Julia Suvorova
              rhn-support-xiaohli Xiaohui Li
              virt-maint virt-maint
              Xiaohui Li Xiaohui Li
              Votes:
              0 Vote for this issue
              Watchers:
              20 Start watching this issue

                Created:
                Updated: