Loading...

Type: Bug
Resolution: Done-Errata
Priority: Normal
Fix Version/s: rhel-9.5
Affects Version/s: rhel-9.3.0
Component/s: kernel / Virtualization / KVM
Labels:
- Unset

Fixed in Build:
kernel-5.14.0-444.el9
Regression:
None
Severity:
Low

Pool Team:

rhel-sst-virtualization-hwe
Sub-System Group:

ssg_virtualization

Dev Target Milestone:
8
Internal Target Milestone:
11
Story Points:
None
Blocked:
False
Blocked Reason:

Hide

None

Show
None
Product Documentation Required:
None
Sprint:
None

Preliminary Testing:
Pass
Testable Builds:

Hide
The following Merge Request has pipeline job artifacts available:

Title: KVM: x86/mmu: Retry fault before acquiring mmu_lock if mapping is changing
MR: https://gitlab.com/redhat/centos-stream/src/kernel/centos-stream-9/-/merge_requests/4051
MR Pipeline: https://gitlab.com/redhat/centos-stream/src/kernel/centos-stream-9/-/pipelines/1254650274

The Repo URLs are *not* accessible from a web browser! They only function as a dnf or yum baseurl.

Artifacts expire six weeks after creation. If artifacts are needed after that time please rerun the pipeline by visiting the MR's 'Pipelines' tab and clicking the 'Run pipeline' button.

----------------------------------------

Downstream Pipeline Name: RHEL_COMPAT (c9s_rhel9_compat_merge_request)
DataWarehouse Checkout: https://datawarehouse.cki-project.org/kcidb/checkouts/redhat:1254650448
Repo URL: https://s3.upshift.redhat.com/DH-PROD-CKI/internal/1254650448/$basearch/5.14.0-439.4051_1254650274.el9.$basearch
Debug Repo URL: https://s3.upshift.redhat.com/DH-PROD-CKI/internal/1254650448/$basearch-debug/5.14.0-439.4051_1254650274.el9.$basearch

5.14.0-439.4051_1254650274.el9.s390x-debug:
Artifacts (RPMs): https://s3.upshift.redhat.com/DH-PROD-CKI/index.html?prefix=internal-artifacts/1254650448/publish_s390x_debug/6634499256/artifacts/repo/5.14.0-439.4051_1254650274.el9.s390x/

5.14.0-439.4051_1254650274.el9.s390x:
Artifacts (RPMs): https://s3.upshift.redhat.com/DH-PROD-CKI/index.html?prefix=internal-artifacts/1254650448/publish_s390x/6634499254/artifacts/repo/5.14.0-439.4051_1254650274.el9.s390x/

5.14.0-439.4051_1254650274.el9.aarch64-debug:
Artifacts (RPMs): https://s3.upshift.redhat.com/DH-PROD-CKI/index.html?prefix=internal-artifacts/1254650448/publish_aarch64_debug/6634499250/artifacts/repo/5.14.0-439.4051_1254650274.el9.aarch64/

5.14.0-439.4051_1254650274.el9.aarch64:
Artifacts (RPMs): https://s3.upshift.redhat.com/DH-PROD-CKI/index.html?prefix=internal-artifacts/1254650448/publish_aarch64/6634499245/artifacts/repo/5.14.0-439.4051_1254650274.el9.aarch64/

5.14.0-439.4051_1254650274.el9.ppc64le-debug:
Artifacts (RPMs): https://s3.upshift.redhat.com/DH-PROD-CKI/index.html?prefix=internal-artifacts/1254650448/publish_ppc64le_debug/6634499221/artifacts/repo/5.14.0-439.4051_1254650274.el9.ppc64le/

5.14.0-439.4051_1254650274.el9.ppc64le:
Artifacts (RPMs): https://s3.upshift.redhat.com/DH-PROD-CKI/index.html?prefix=internal-artifacts/1254650448/publish_ppc64le/6634499196/artifacts/repo/5.14.0-439.4051_1254650274.el9.ppc64le/

5.14.0-439.4051_1254650274.el9.x86_64-debug:
Artifacts (RPMs): https://s3.upshift.redhat.com/DH-PROD-CKI/index.html?prefix=internal-artifacts/1254650448/publish_x86_64_debug/6634499154/artifacts/repo/5.14.0-439.4051_1254650274.el9.x86_64/

5.14.0-439.4051_1254650274.el9.x86_64:
Artifacts (RPMs): https://s3.upshift.redhat.com/DH-PROD-CKI/index.html?prefix=internal-artifacts/1254650448/publish_x86_64/6634499135/artifacts/repo/5.14.0-439.4051_1254650274.el9.x86_64/

----------------------------------------

Downstream Pipeline Name: REALTIME (c9s_rt_merge_request)
DataWarehouse Checkout: https://datawarehouse.cki-project.org/kcidb/checkouts/redhat:1254650392

5.14.0-439.4051_1254650274.el9.x86_64-debug:
Artifacts (RPMs): https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/index.html?prefix=trusted-artifacts/1254650392/publish_x86_64_debug/6634498651/artifacts/repo/5.14.0-439.4051_1254650274.el9.x86_64/
Repo URL: https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/trusted-artifacts/1254650392/publish_x86_64_debug/6634498651/artifacts/repo/5.14.0-439.4051_1254650274.el9.x86_64/

5.14.0-439.4051_1254650274.el9.x86_64:
Artifacts (RPMs): https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/index.html?prefix=trusted-artifacts/1254650392/publish_x86_64/6634498644/artifacts/repo/5.14.0-439.4051_1254650274.el9.x86_64/
Repo URL: https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/trusted-artifacts/1254650392/publish_x86_64/6634498644/artifacts/repo/5.14.0-439.4051_1254650274.el9.x86_64/

----------------------------------------

Downstream Pipeline Name: 64K (c9s_64k_merge_request)
DataWarehouse Checkout: https://datawarehouse.cki-project.org/kcidb/checkouts/redhat:1254650433

5.14.0-439.4051_1254650274.el9.aarch64-debug:
Artifacts (RPMs): https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/index.html?prefix=trusted-artifacts/1254650433/publish_aarch64_debug/6634498967/artifacts/repo/5.14.0-439.4051_1254650274.el9.aarch64/
Repo URL: https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/trusted-artifacts/1254650433/publish_aarch64_debug/6634498967/artifacts/repo/5.14.0-439.4051_1254650274.el9.aarch64/

5.14.0-439.4051_1254650274.el9.aarch64:
Artifacts (RPMs): https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/index.html?prefix=trusted-artifacts/1254650433/publish_aarch64/6634498957/artifacts/repo/5.14.0-439.4051_1254650274.el9.aarch64/
Repo URL: https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/trusted-artifacts/1254650433/publish_aarch64/6634498957/artifacts/repo/5.14.0-439.4051_1254650274.el9.aarch64/

----------------------------------------

Updated 2024-04-16 11:05 UTC by buglinker:
- KWF FAQ: https://red.ht/kernel_workflow_doc
- Slack #team-kernel-workflow: https://redhat-internal.slack.com/archives/C04LRUPMJQ5
- Source: https://gitlab.com/cki-project/kernel-workflow/-/blob/main/webhook/buglinker.py
- Documentation: https://gitlab.com/cki-project/kernel-workflow/-/blob/main/docs/README.buglinker.md
- Report an issue: https://gitlab.com/cki-project/kernel-workflow/-/issues/new?issue%5Btitle%5D=buglinker%20webhook%20issue

Show
The following Merge Request has pipeline job artifacts available: Title: KVM: x86/mmu: Retry fault before acquiring mmu_lock if mapping is changing MR: https://gitlab.com/redhat/centos-stream/src/kernel/centos-stream-9/-/merge_requests/4051 MR Pipeline: https://gitlab.com/redhat/centos-stream/src/kernel/centos-stream-9/-/pipelines/1254650274 The Repo URLs are *not* accessible from a web browser! They only function as a dnf or yum baseurl. Artifacts expire six weeks after creation. If artifacts are needed after that time please rerun the pipeline by visiting the MR's 'Pipelines' tab and clicking the 'Run pipeline' button. ---------------------------------------- Downstream Pipeline Name: RHEL_COMPAT (c9s_rhel9_compat_merge_request) DataWarehouse Checkout: https://datawarehouse.cki-project.org/kcidb/checkouts/redhat:1254650448 Repo URL: https://s3.upshift.redhat.com/DH-PROD-CKI/internal/1254650448/$basearch/5.14.0-439.4051_1254650274.el9.$basearch Debug Repo URL: https://s3.upshift.redhat.com/DH-PROD-CKI/internal/1254650448/$basearch-debug/5.14.0-439.4051_1254650274.el9.$basearch 5.14.0-439.4051_1254650274.el9.s390x-debug: Artifacts (RPMs): https://s3.upshift.redhat.com/DH-PROD-CKI/index.html?prefix=internal-artifacts/1254650448/publish_s390x_debug/6634499256/artifacts/repo/5.14.0-439.4051_1254650274.el9.s390x/ 5.14.0-439.4051_1254650274.el9.s390x: Artifacts (RPMs): https://s3.upshift.redhat.com/DH-PROD-CKI/index.html?prefix=internal-artifacts/1254650448/publish_s390x/6634499254/artifacts/repo/5.14.0-439.4051_1254650274.el9.s390x/ 5.14.0-439.4051_1254650274.el9.aarch64-debug: Artifacts (RPMs): https://s3.upshift.redhat.com/DH-PROD-CKI/index.html?prefix=internal-artifacts/1254650448/publish_aarch64_debug/6634499250/artifacts/repo/5.14.0-439.4051_1254650274.el9.aarch64/ 5.14.0-439.4051_1254650274.el9.aarch64: Artifacts (RPMs): https://s3.upshift.redhat.com/DH-PROD-CKI/index.html?prefix=internal-artifacts/1254650448/publish_aarch64/6634499245/artifacts/repo/5.14.0-439.4051_1254650274.el9.aarch64/ 5.14.0-439.4051_1254650274.el9.ppc64le-debug: Artifacts (RPMs): https://s3.upshift.redhat.com/DH-PROD-CKI/index.html?prefix=internal-artifacts/1254650448/publish_ppc64le_debug/6634499221/artifacts/repo/5.14.0-439.4051_1254650274.el9.ppc64le/ 5.14.0-439.4051_1254650274.el9.ppc64le: Artifacts (RPMs): https://s3.upshift.redhat.com/DH-PROD-CKI/index.html?prefix=internal-artifacts/1254650448/publish_ppc64le/6634499196/artifacts/repo/5.14.0-439.4051_1254650274.el9.ppc64le/ 5.14.0-439.4051_1254650274.el9.x86_64-debug: Artifacts (RPMs): https://s3.upshift.redhat.com/DH-PROD-CKI/index.html?prefix=internal-artifacts/1254650448/publish_x86_64_debug/6634499154/artifacts/repo/5.14.0-439.4051_1254650274.el9.x86_64/ 5.14.0-439.4051_1254650274.el9.x86_64: Artifacts (RPMs): https://s3.upshift.redhat.com/DH-PROD-CKI/index.html?prefix=internal-artifacts/1254650448/publish_x86_64/6634499135/artifacts/repo/5.14.0-439.4051_1254650274.el9.x86_64/ ---------------------------------------- Downstream Pipeline Name: REALTIME (c9s_rt_merge_request) DataWarehouse Checkout: https://datawarehouse.cki-project.org/kcidb/checkouts/redhat:1254650392 5.14.0-439.4051_1254650274.el9.x86_64-debug: Artifacts (RPMs): https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/index.html?prefix=trusted-artifacts/1254650392/publish_x86_64_debug/6634498651/artifacts/repo/5.14.0-439.4051_1254650274.el9.x86_64/ Repo URL: https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/trusted-artifacts/1254650392/publish_x86_64_debug/6634498651/artifacts/repo/5.14.0-439.4051_1254650274.el9.x86_64/ 5.14.0-439.4051_1254650274.el9.x86_64: Artifacts (RPMs): https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/index.html?prefix=trusted-artifacts/1254650392/publish_x86_64/6634498644/artifacts/repo/5.14.0-439.4051_1254650274.el9.x86_64/ Repo URL: https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/trusted-artifacts/1254650392/publish_x86_64/6634498644/artifacts/repo/5.14.0-439.4051_1254650274.el9.x86_64/ ---------------------------------------- Downstream Pipeline Name: 64K (c9s_64k_merge_request) DataWarehouse Checkout: https://datawarehouse.cki-project.org/kcidb/checkouts/redhat:1254650433 5.14.0-439.4051_1254650274.el9.aarch64-debug: Artifacts (RPMs): https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/index.html?prefix=trusted-artifacts/1254650433/publish_aarch64_debug/6634498967/artifacts/repo/5.14.0-439.4051_1254650274.el9.aarch64/ Repo URL: https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/trusted-artifacts/1254650433/publish_aarch64_debug/6634498967/artifacts/repo/5.14.0-439.4051_1254650274.el9.aarch64/ 5.14.0-439.4051_1254650274.el9.aarch64: Artifacts (RPMs): https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/index.html?prefix=trusted-artifacts/1254650433/publish_aarch64/6634498957/artifacts/repo/5.14.0-439.4051_1254650274.el9.aarch64/ Repo URL: https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/trusted-artifacts/1254650433/publish_aarch64/6634498957/artifacts/repo/5.14.0-439.4051_1254650274.el9.aarch64/ ---------------------------------------- Updated 2024-04-16 11:05 UTC by buglinker: - KWF FAQ: https://red.ht/kernel_workflow_doc - Slack #team-kernel-workflow: https://redhat-internal.slack.com/archives/C04LRUPMJQ5 - Source: https://gitlab.com/cki-project/kernel-workflow/-/blob/main/webhook/buglinker.py - Documentation: https://gitlab.com/cki-project/kernel-workflow/-/blob/main/docs/README.buglinker.md - Report an issue: https://gitlab.com/cki-project/kernel-workflow/-/issues/new?issue%5Btitle%5D=buglinker%20webhook%20issue
Errata Link:
https://errata.engineering.redhat.com/advisory/128795
Test Coverage:
None

Release Note Type:
If docs needed, set a value

Experience:
Architecture:

Unspecified
Bugzilla Bug:
RHBZ: 2224509

SFDC Cases Counter:
SFDC Cases Open:
SFDC Cases Links:

Planning:
None

Description of problem:
hit "watchdog: BUG: soft lockup - CPU#9 stuck for 44s! [swapper/9:0]" problem during hotplug vcpus.

Version-Release number of selected component (if applicable):
host version:
kernel-5.14.0-340.el9.x86_64
qemu-kvm-8.0.0-7.el9.x86_64
edk2-ovmf-20230524-2.el9.noarch
guest: rhel9.3.0

How reproducible:
2/2

Steps to Reproduce:
1.Boot guest with "cpus_hotplug.sh" script

sh cpus_hotplug.sh ovmf

2.After the cpus_hotplug.sh script is complete, dmesg message in guest.

dmesg

Actual results:
[ 1150.754057] watchdog: BUG: soft lockup - CPU#9 stuck for 44s! [swapper/9:0]
[ 1151.178612] Modules linked in: nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set rfkill nf_tables nfnetlink qrtr vfat fat intel_rapl_msr intel_rapl_common rapl iTCO_wdt iTCO_vendor_support i2c_i801 pcspkr lpc_ich i2c_smbus joydev fuse xfs libcrc32c bochs drm_vram_helper drm_kms_helper syscopyarea sysfillrect sysimgblt drm_ttm_helper ttm ahci libahci drm sd_mod sg nvme_tcp nvme_fabrics nvme crct10dif_pclmul crc32_pclmul nvme_core crc32c_intel nvme_common libata t10_pi ghash_clmulni_intel e1000e virtio_scsi serio_raw dm_multipath dm_mirror dm_region_hash dm_log dm_mod be2iscsi bnx2i cnic uio cxgb4i cxgb4 tls libcxgbi libcxgb qla4xxx iscsi_boot_sysfs iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi
[ 1151.234293] CPU: 9 PID: 0 Comm: swapper/9 Not tainted 5.14.0-340.el9.x86_64 #1
[ 1151.234762] Hardware name: Red Hat KVM/RHEL, BIOS edk2-20230524-2.el9 05/24/2023
[ 1151.235369] RIP: 0010:default_idle+0x10/0x20
[ 1151.235771] Code: 8b 04 25 40 ef 01 00 f0 80 60 02 df c3 cc cc cc cc 0f ae 38 eb bb 0f 1f 40 00 0f 1f 44 00 00 eb 07 0f 00 2d 3e 3c 47 00 fb f4 <c3> cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc 0f 1f 44 00 00 65
[ 1151.236764] RSP: 0018:ffffa81140113ed0 EFLAGS: 00000212
[ 1151.237040] RAX: ffffffff95197340 RBX: ffff92c200bb8000 RCX: ffff92c28c440e00
[ 1151.237402] RDX: 4000000000000000 RSI: 0000000000000009 RDI: 00000000025f488c
[ 1151.237833] RBP: 0000000000000000 R08: 00000100d51e74e7 R09: 00000000000486fd
[ 1151.238212] R10: 00000000000486fd R11: 0000000000000000 R12: 0000000000000000
[ 1151.259021] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[ 1151.259440] FS: 0000000000000000(0000) GS:ffff92d139840000(0000) knlGS:0000000000000000
[ 1151.266655] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1151.266971] CR2: 00007fe00c7d24f4 CR3: 000000022e610003 CR4: 00000000003706e0
[ 1151.267348] Call Trace:
[ 1151.267532] <IRQ>
[ 1151.267675] ? show_trace_log_lvl+0x1c4/0x2df
[ 1151.267918] ? show_trace_log_lvl+0x1c4/0x2df
[ 1151.268156] ? default_idle_call+0x33/0xe0
[ 1151.268386] ? watchdog_timer_fn+0x1b2/0x210
[ 1151.268690] ? lockup_detector_update_enable+0x50/0x50
[ 1151.269035] ? __hrtimer_run_queues+0x12a/0x2c0
[ 1151.269358] ? hrtimer_interrupt+0xfc/0x210
[ 1151.269605] ? __sysvec_apic_timer_interrupt+0x5f/0x110
[ 1151.269880] ? sysvec_apic_timer_interrupt+0x6d/0x90
[ 1151.270141] </IRQ>
[ 1151.270278] <TASK>
[ 1151.270432] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[ 1151.270784] ? mwait_idle+0x70/0x70
[ 1151.271050] ? default_idle+0x10/0x20
[ 1151.271258] default_idle_call+0x33/0xe0
[ 1151.271505] cpuidle_idle_call+0x125/0x160
[ 1151.271797] ? kvm_sched_clock_read+0x14/0x30
[ 1151.272102] do_idle+0x78/0xe0
[ 1151.272335] cpu_startup_entry+0x19/0x20
[ 1151.272575] start_secondary+0x10d/0x130
[ 1151.272797] secondary_startup_64_no_verify+0xe5/0xeb
[ 1151.273071] </TASK>