-
Bug
-
Resolution: Duplicate
-
Undefined
-
None
-
rhel-9.3.0
-
Yes
-
Critical
-
Regression
-
rhel-sst-kernel-ft
-
ssg_core_kernel
-
None
-
False
-
-
None
-
None
-
Rejected Blocker
-
None
-
None
-
-
x86_64
-
None
What were you trying to do that didn't work?
Boot current cloud image and try to poweroff
Please provide the package NVR for which bug is seen:
kernel-5.14.0-362.el9.x86_64
How reproducible: Always
Steps to reproduce
- Boot http://download.devel.redhat.com/rhel-9/nightly/RHEL-9/latest-RHEL-9.3/compose/BaseOS/x86_64/images/rhel-guest-image-9.3-20230901.0.x86_64.qcow2 or https://cloud.centos.org/centos/9-stream/x86_64/images/CentOS-Stream-GenericCloud-9-latest.x86_64.qcow2 with some suitable cloud-init.iso; you are welcome to use cockpit CI's which provides root:foobar and admin:foobar accounts, but it's not magic in any way. You just need some cloud-init data to be able to log in at all.
- poweroff
Expected results
Machine powers off.
Actual results
Kernel oopses during late shutdown, complaining that init gets killed. Well yes, that's what shutdown does..
[ OK ] Reached target System Shutdown. [ OK ] Reached target Late Shutdown Services. [ OK ] Finished System Power Off. [ OK ] Reached target System Power Off. [...] [ 136.473094] systemd-shutdown[1]: Sending SIGKILL to remaining processes... [ 136.474582] systemd-shutdown[1]: Unmounting file systems. [ 136.474897] [4117]: Remounting '/' read-only with options 'seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota'. [ 136.479713] systemd-shutdown[1]: All filesystems unmounted. [ 136.485781] systemd-shutdown[1]: Powering off. [ 136.486908] ACPI: PM: Preparing to enter system sleep state S5 [ 136.487425] kvm: exiting hardware virtualization [ 136.487730] reboot: Power down [ 136.488059] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000000 [ 136.488581] CPU: 0 PID: 1 Comm: systemd-shutdow Kdump: loaded Not tainted 5.14.0-362.el9.x86_64 #1 [ 136.489174] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 136.489722] Call Trace: [ 136.490266] <TASK> [ 136.490783] dump_stack_lvl+0x34/0x48 [ 136.491369] panic+0xea/0x2e4 [ 136.491860] do_exit.cold+0x15/0x15 [ 136.492411] __do_sys_reboot+0x1e5/0x220 [ 136.492952] ? do_iter_readv_writev+0x124/0x190 [ 136.493533] ? do_iter_write+0x81/0x150 [ 136.494065] ? vfs_writev+0xcb/0x170 [ 136.494570] do_syscall_64+0x59/0x90 [ 136.495064] ? __x64_sys_rt_sigprocmask+0x76/0xd0 [ 136.495611] ? do_writev+0x6f/0x120 [ 136.496069] ? syscall_exit_to_user_mode+0x12/0x30 [ 136.496607] ? do_syscall_64+0x69/0x90 [ 136.497067] ? exc_page_fault+0x62/0x150 [ 136.497539] entry_SYSCALL_64_after_hwframe+0x72/0xdc [ 136.498083] RIP: 0033:0x7f9a8a3453e7 [ 136.498524] Code: 0b 00 f7 d8 64 89 02 b8 ff ff ff ff eb b8 0f 1f 44 00 00 f3 0f 1e fa 89 fa be 69 19 12 28 bf ad de e1 fe b8 a9 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 01 c3 48 8b 15 09 4a 0b 00 f7 d8 64 89 02 b8 [ 136.500059] RSP: 002b:00007ffcc34c4f38 EFLAGS: 00000202 ORIG_RAX: 00000000000000a9 [ 136.500743] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f9a8a3453e7 [ 136.501421] RDX: 000000004321fedc RSI: 0000000028121969 RDI: 00000000fee1dead [ 136.502087] RBP: 00007ffcc34c5140 R08: 0000000000000000 R09: 00007ffcc34c4330 [ 136.502738] R10: 00007ffcc34c44f0 R11: 0000000000000202 R12: 0000000000000000 [ 136.503360] R13: 00007ffcc34c4f88 R14: 00007ffcc34c4f90 R15: 0000000000000000 [ 136.503992] </TASK>
After that, it immediately reboots into a kdump boot:
[ 0.000000] Linux version 5.14.0-362.el9.x86_64 (mockbuild@x86-64-01.build.eng.rdu2.redhat.com) (gcc (GCC) 11.4.1 20230605 (Red Hat 11.4.1-2), GNU ld version 2.35.2-42.el9) #1 SMP PREEMPT_DYNAMIC Mon Aug 28 18:48:22 EDT 2023 [...] [ 1.817772] kdump[382]: Kdump is using the default log level(3). [ 1.846962] kdump[418]: saving to /sysroot/var/crash/127.0.0.1-2023-09-01-02:07:42/ [ 1.852206] kdump[423]: saving vmcore-dmesg.txt to /sysroot/var/crash/127.0.0.1-2023-09-01-02:07:42/ [ 1.862219] kdump[429]: saving vmcore-dmesg.txt complete [ 1.864143] kdump[431]: saving vmcore Copying data : [100.0 %] / eta: 0s [ 2.127216] kdump.sh[432]: The dumpfile is saved to /sysroot/var/crash/127.0.0.1-2023-09-01-02:07:42//vmcore-incomplete. [ 2.128155] kdump.sh[432]: makedumpfile Completed. [ 2.136409] kdump[436]: saving vmcore complete [ 2.138384] kdump[438]: saving the /run/initramfs/kexec-dmesg.log to /sysroot/var/crash/127.0.0.1-2023-09-01-02:07:42// [ 2.153255] kdump[444]: Executing final action systemctl reboot -f
.. and eventually ends up again in a booted system:
OK ] Finished Crash recovery kernel arming. Red Hat Enterprise Linux 9.3 Beta (Plow) Kernel 5.14.0-362.el9.x86_64 on an x86_64 Activate the web console with: systemctl enable --now cockpit.socket rhel-9-3-127-0-0-2-2201 login:
See https://github.com/cockpit-project/bots/issues/5169 – this started somewhere between rhel-guest-image-9.3-20230817.2.x86_64.qcow2 from roughly two weeks ago (which was fine), and rhel-guest-image-9.3-20230828.65.x86_64.qcow2 from 3 days ago (which was broken).
