What were you trying to do that didn't work?

Installation of RHEL-10.0-20240717.79 using a volume group on top of a local disk and iSCSI disk failed to reboot into the installed system with the following message in console:

Failed to execute shutdown binary.

Going a bit backward in the console log, there are some squashfs and swap read errors:

Read-error on swap-device (253:0:56)   
Read-error on swap-device (253:0:832)   
Read-error on swap-device (253:0:840)   
Read-error on swap-device (253:0:848) 
...
Buffer I/O error on dev dm-0, logical block 1035248, async page read   
I/O error, dev loop0, sector 694454 op 0x0:(READ) flags 0x800 phys_seg 10 prio class 0   
SQUASHFS error: Failed to read block 0x15316e6b: -5   
SQUASHFS error: Unable to read fragment cache entry [15316e6b]   
SQUASHFS error: Unable to read fragment cache entry [15316e6b]
SQUASHFS error: Unable to read page, block 15316e6b, size 8a64 

Based on those symptoms and configuration, I believe the following is happening (with some level of abstraction):

1. iscsi services are started, Anaconda (installer) starts.
2. Anaconda arranges for a connection to a defined iSCSI target (specified in kickstart file).
3. Anaconda creates new partitioning that includes a volume group across local and iSCSI disk (containing a logical volume for swap), newly created swap (LV) gets activated.
4. Anaconda downloads packages for installation, installs them and performs other necessary steps. Part of the squashfs data potentially gets swapped out from RAM during the installation process.
5. Shutdown phase begins, services are getting stopped.
6. iscsi-shutdown.service gets stopped, and thus the iSCSI session gets terminated.
7. ...
8. swap.target gets stopped and swap devices deactivated.
9. ...
10. Attempt to shutdown (reboot) the system fails, as it's not possible to execute the shutdown binary which is likely on the now inaccessible swap LV on the iSCSI disk.

The way the iSCSI session is terminated should be optimized to ensure that the termination (logout) takes place at a point where it is no more used in any way (i. e. an active swap LV in this case).

A first straight-forward idea is that the iscsi-shutdown.service dependencies could be changed to take into account the swap.target and some other targets/services that may actually use the devices provided by the iSCSI subsystem when being stopped, however I'm not sure if it's that easy and if it couldn't break something else. Moreover, I think a similar issue can, in principle, also happen in an installed system during shutdown.

Please provide the package NVR for which bug is seen:

iscsi-initiator-utils-6.2.1.9-1.gita65a472.el9.x86_64 (anaconda-34.25.5.6-1.el9.x86_64)

How reproducible:

Approximately 1/3.

Steps to reproduce

1. Prepare an iSCSI target.
2. Prepare a kickstart file that will also use an iSCSI disk as part of the storage.
3. Start system installation using the prepared kickstart.

Expected results

Installation is successful, newly installed systems boot after the installation is complete.

Actual results

Installation is successful, however it's not possible to reboot from installation environment, as execution of the shutdown binary fails. System console also contains several error messages related to squashfs and swap read failures.