What were you trying to do that didn't work?

tl;dr: a `Power key pressed short.` results in a reboot of the host instead of a shutdown/halt

We build AMI images in AWS on top of the "ec2" images published at https://cloud.centos.org/centos/9-stream/x86_64/images/

The packer build process essentially boots up an existing AMI, executes some provisioners to make changes, shuts down the host, then takes a block-level snapshot to create a new AMI image.

Starting with image `CentOS-Stream-ec2-9-20230828.0.x86_64.raw.xz` / 5.14.0-361.el9.x86_64 we started seeing a behavior where a `Power key pressed short.` resulted in a reboot of the host instead of a shutdown in prior releases.

In the packer workflow specifically this means that after provisioners make changes to an image and then shut the host down it actually boots up once before a snapshot is taken resulting in unexpected and unintentional first boot operations being applied to what should be a clean image.

Please provide the package NVR for which bug is seen:

Affected:
CentOS-Stream-ec2-9-20230828.0.x86_64.raw.xz
CentOS-Stream-ec2-9-20230904.0.x86_64.raw.xz

Last known good:
CentOS-Stream-ec2-9-20230821.0.x86_64.raw.xz

How reproducible:

Every boot

Steps to reproduce

1. boot one of the images 20230828.0 or newer
2. send an power key press short event
3. observe the host reboots instead of shutting down/halting

Testing in AWS we spun up the newest 3 ec2 images: 20230821.0, 20230828.0, and 20230904.0.  Once all 3 were running we issued a StopInstances() API call to each host.  After waiting 5-10 minutes we then issued a StartInstances() API call and observed that images newer than 20230821.0 had 3 boot ups instead of the expected 2.

ip-172-31-2-7 - CentOS-Stream-ec2-9-20230821.0.x86_64.raw.xz

02:10:00 - STOP issued
02:15:00 - START issued
[ec2-user@ip-172-31-2-7 ~]$ uname -a
Linux ip-172-31-2-7.us-west-2.compute.internal 5.14.0-354.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Aug 10 21:06:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
[ec2-user@ip-172-31-2-7 ~]$ sudo grep "Command line" /var/log/messages
Sep  8 02:05:58 ip-172-31-2-7 kernel: Command line: BOOT_IMAGE=(hd0,gpt2)/boot/vmlinuz-5.14.0-354.el9.x86_64 root=UUID=318adc97-7acb-4f53-99ff-a07e8e8db0c0 ro console=ttyS0,115200n8 console=tty0 net.ifnames=0 rd.blacklist=nouveau nvme_core.io_timeout=4294967295 crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M
Sep  8 02:15:16 ip-172-31-2-7 kernel: Command line: BOOT_IMAGE=(hd0,gpt2)/boot/vmlinuz-5.14.0-354.el9.x86_64 root=UUID=318adc97-7acb-4f53-99ff-a07e8e8db0c0 ro console=ttyS0,115200n8 console=tty0 net.ifnames=0 rd.blacklist=nouveau nvme_core.io_timeout=4294967295 crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M

---

ip-172-31-2-126 - CentOS-Stream-ec2-9-20230828.0.x86_64.raw.xz

01:52:35 - STOP issued.
02:00:00 - START issued.
[ec2-user@ip-172-31-2-126 ~]$ uname -a
Linux ip-172-31-2-126.us-west-2.compute.internal 5.14.0-361.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Aug 24 13:40:45 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
[ec2-user@ip-172-31-2-126 ~]$ sudo grep "Command line" /var/log/messages
Sep  8 01:29:46 ip-172-31-2-126 kernel: Command line: BOOT_IMAGE=(hd0,gpt2)/boot/vmlinuz-5.14.0-361.el9.x86_64 root=UUID=0f01cec4-7e50-471a-8ea6-06acce1cc6dd ro console=ttyS0,115200n8 console=tty0 net.ifnames=0 rd.blacklist=nouveau nvme_core.io_timeout=4294967295 crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M
Sep  8 01:53:01 ip-172-31-2-126 kernel: Command line: BOOT_IMAGE=(hd0,gpt2)/boot/vmlinuz-5.14.0-361.el9.x86_64 root=UUID=0f01cec4-7e50-471a-8ea6-06acce1cc6dd ro console=ttyS0,115200n8 console=tty0 net.ifnames=0 rd.blacklist=nouveau nvme_core.io_timeout=4294967295 crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M
Sep  8 02:00:19 ip-172-31-2-126 kernel: Command line: BOOT_IMAGE=(hd0,gpt2)/boot/vmlinuz-5.14.0-361.el9.x86_64 root=UUID=0f01cec4-7e50-471a-8ea6-06acce1cc6dd ro console=ttyS0,115200n8 console=tty0 net.ifnames=0 rd.blacklist=nouveau nvme_core.io_timeout=4294967295 crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M

------
ip-172-31-2-228 - CentOS-Stream-ec2-9-20230904.0.x86_64.raw.xz

01:52:35 - STOP issued.
02:00:00 - START issued.
[ec2-user@ip-172-31-2-228 log]$ uname -a
Linux ip-172-31-2-228.us-west-2.compute.internal 5.14.0-362.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Mon Aug 28 20:43:49 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
[ec2-user@ip-172-31-2-228 log]$ sudo grep "Command line" /var/log/messages
Sep  8 01:28:36 ip-172-31-2-228 kernel: Command line: BOOT_IMAGE=(hd0,gpt2)/boot/vmlinuz-5.14.0-362.el9.x86_64 root=UUID=90c99e1c-4fc8-4b54-9451-7f56173692bd ro console=ttyS0,115200n8 console=tty0 net.ifnames=0 rd.blacklist=nouveau nvme_core.io_timeout=4294967295 crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M
Sep  8 01:53:05 ip-172-31-2-228 kernel: Command line: BOOT_IMAGE=(hd0,gpt2)/boot/vmlinuz-5.14.0-362.el9.x86_64 root=UUID=90c99e1c-4fc8-4b54-9451-7f56173692bd ro console=ttyS0,115200n8 console=tty0 net.ifnames=0 rd.blacklist=nouveau nvme_core.io_timeout=4294967295 crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M
Sep  8 02:00:15 ip-172-31-2-228 kernel: Command line: BOOT_IMAGE=(hd0,gpt2)/boot/vmlinuz-5.14.0-362.el9.x86_64 root=UUID=90c99e1c-4fc8-4b54-9451-7f56173692bd ro console=ttyS0,115200n8 console=tty0 net.ifnames=0 rd.blacklist=nouveau nvme_core.io_timeout=4294967295 crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M

Expected results

host should shutdown/halt.

Actual results

host reboots.