What were you trying to do that didn't work?

Have Greenboot roll back my system that got unbootable due to bootc update

What is the impact of this issue to you?

I can't use the system. Also, GRUB is not showing so I have no way to roll it back at this point.

Please provide the package NVR for which the bug is seen:

greenboot-0.15.6-2.el9.x86_64

How reproducible is this bug?:

100%

Steps to reproduce

1. Create build container image with Containerfile. In my tests I'm using a local registry as outlined on https://gist.github.com/smitterl/b7d1e1543eddc8b6aefd32b8b70b0b89

   FROM quay.io/centos-bootc/centos-bootc:stream9
   ...
   RUN dnf install -y greenboot && dnf clean all
   RUN systemctl enable greenboot-task-runner greenboot-healthcheck greenboot-status greenboot-loading-message greenboot-grub2-set-counter greenboot-grub2-set-success greenboot-rpm-ostree-grub2-check-fallback redboot-auto-reboot redboot-task-runner
   

2. Create a qcow2 image from the container

   podman run \
       --rm \
       -it \ 
       --privileged \     
       --pull=newer \
       --security-opt label=type:unconfined_t \
       -v $(pwd)/config.toml:/config.toml:ro \
       -v $(pwd)/output:/output \
       -v /var/lib/containers/storage:/var/lib/containers/storage \
       quay.io/centos-bootc/bootc-image-builder:latest \
       --type qcow2 --tls-verify=false --local <hostname>:5000/demo:latest

3. Define and start the VM with a watchdog device

   # qemu-img convert -f qcow2 -O qcow2 -o lazy_refcounts=on ./output/qcow2/disk.qcow2 /var/lib/libvirt/images/disk.qcow2
   # virt-install --name vm --os-variant rhel-unknown --memory 2048 --vcpus 2 --nographics --import --disk /var/lib/libvirt/images/disk.qcow2 --watchdog i6300esb

4. Confirm that greenboot is running and that the watchdog is available

   $ sudo systemctl status greenboot-task-runner
   ● greenboot-task-runner.service - greenboot Success Scripts Runner
        Loaded: loaded (/usr/lib/systemd/system/greenboot-task-runner.service; ena>
   

5. Build and push a container image that can't be booted into because of a corrupt vmlinuz

   $ sudo wdctl
   Device:        /dev/watchdog0
   Identity:      i6300ESB timer [version 0]
   Timeout:       30 seconds
   Pre-timeout:    0 seconds
   FLAG           DESCRIPTION               STATUS BOOT-STATUS
   KEEPALIVEPING  Keep alive ping reply          1           0
   MAGICCLOSE     Supports magic close char      0           0
   SETTIMEOUT     Set timeout (in seconds)       0           0
   

   cat Containerfile
   ...
   RUN echo "foo" > /usr/lib/modules/5.14.0-512.el9.x86_64/vmlinuz
   

6. In the VM

   sudo bootc update --apply

   (or wait for its timer to trigger)

   Expected results

The system can't boot, it's reverted into the last working container image state

Actual results

The system doesn't boot.