updates-protectudf-rhel89.img

If we create USB bootable for OS installation using the UDF formatted ISO, anaconda does not recognize this USB as protected device and wipes it during partitioning even though it contains the installation media from where we booted resulting in the USB being wiped and OS installation fails.

Usually the ISO which we provide to customer on customer portal is formatted as ISO9660 format
However some of our customer need to customize to add their own software/repository and kickstart files inside the ISO. Due to limitation of ISO9660 we cannot add a file larger than 4 GiB. So when adding a new file larger than 4 GiB inside the ISO and rebuilding the iso results in an error and we need to pass the `-allow-limited-size` flag to the `genisoimage` which results in the output ISO created as UDF formatted ISO rather than ISO9660.

As per man page of genisoimage(1)

   -allow-limited-size
          When  processing  files  larger  than  2GiB  which cannot be easily represented in ISO9660, add them with a shrunk visible file size to ISO9660 and with the correct visible file size to the UDF system. The result is an
          inconsistent filesystem and users need to make sure that they really use UDF rather than ISO9660 driver to read a such disk. Implies enabling -udf.

When we create a bootable USB using such UDF formatted ISO, and use it to install the OS using Kickstart file the USB is recognized as Hard disk (suppose /dev/sda) but its not added to protected device by anaconda since Anaconda tries to resolve the FS LABEL but because it does not recognizes the UDF format, it is not able to resolve the LABEL to USB device and hence its not added to protected device and hence is formatted as part of partitioning during automatic partitioning.

• Anaconda logs seen when using a bootable USB for ISO created WITHOUT "--allow-limited-size" flag (i.e. iso9660 filesystem)
  We see that anaconda set protected devices, the label for the ISO is looked up and its resolved to the bootable USB and hence its not touched or wiped as part of the installation.

      09:42:20,216 WARNING org.fedoraproject.Anaconda.Modules.Storage:DEBUG:anaconda.modules.storage.disk_selection.selection:Protected devices are set to '['LABEL=RHEL-8-9-0-BaseOS-x86_64']'.
  
      09:42:20,688 WARNING org.fedoraproject.Anaconda.Modules.Storage:DEBUG:blivet:resolved 'LABEL=RHEL-8-9-0-BaseOS-x86_64' to 'sda1' (partition)
      09:42:20,689 WARNING org.fedoraproject.Anaconda.Modules.Storage:DEBUG:anaconda.storage:Protected device spec LABEL=RHEL-8-9-0-BaseOS-x86_64 resolved to sda1.
      

• While in the anaconda logs where bootable USB for ISO created WITH "-allow-limited-size" flag (i.e. udf filesystem)
  We see that the protected device are again set up to the LABEL of the installation source but the lookup for the label fail with "failed to resolve error." due to which we do not find the device which is holding the installation source and hence its wiped in subsequent steps.

      07:00:22,706 WARNING org.fedoraproject.Anaconda.Modules.Storage:DEBUG:anaconda.modules.storage.disk_selection.selection:Protected devices are set to '['LABEL=RHEL-8-9-0-BaseOS-x86_64']'.
  
      07:00:24,451 WARNING org.fedoraproject.Anaconda.Modules.Storage:DEBUG:blivet:failed to resolve 'LABEL=RHEL-8-9-0-BaseOS-x86_64'
      

I found that this protected device logic is stored, we will find this in the pyanaconda python modules storage layout.
File - /usr/lib64/python3.6/site-packages/pyanaconda/modules/storage/devicetree/model.py

277     def _mark_protected_devices(self):
278         """Mark protected devices.
279 
280         If a device is protected, mark it as such now. Once the tree
281         has been populated, devices' protected attribute is how we will
282         identify protected devices.
283         """
284         protected = []
285 
286         # Resolve the protected device specs to devices.
287         for spec in self.protected_devices:
288             dev = self.devicetree.resolve_device(spec)
289 
290             if dev is not None:
291                 log.debug("Protected device spec %s resolved to %s.", spec, dev.name)
292                 protected.append(dev)
293 
294         # Find the live backing device and its parents.
295         live_device = find_live_backing_device(self.devicetree)
296 
297         if live_device:
298             log.debug("Resolved live device to %s.", live_device.name)
299             protected.append(live_device)
300             protected.extend(live_device.parents)
301 
302         # For image installation setup_disk_images method marks all local
303         # storage disks as ignored so they are protected from teardown.
304         # Here we protect also cdrom devices from tearing down that, in case of
305         # cdroms, involves unmounting which is undesirable (see bug #1671713).
306         protected.extend(dev for dev in self.devicetree.devices if dev.type == "cdrom")
307 
308         # Protect also all devices with an iso9660 file system. It will protect
309         # NVDIMM devices that can be used only as an installation source anyway
310         # (see the bug #1856264).
311         protected.extend(dev for dev in self.devicetree.devices if dev.format.type == "iso9660")
312 
313         # RH Case
314         # Protect udf built USB bootable from teardown , copying from above "iso9660" line
315         protected.extend(dev for dev in self.devicetree.devices if dev.format.type == "udf")
316 
317         # Mark the collected devices as protected.
318         for dev in protected:
319             log.debug("Marking device %s as protected.", dev.name)
320             dev.protected = True

I have tried to add a catch all to ignore the UDF by copying the in between Lines 313 and 315 by copying the definition of "iso9660" above but this does did not help as the device still did not resolve UDF filesystem using `dev.format.type == "udf"`

So I searched the existing "iso9660" (since this is being recognized as able to resolve to device name) as to where these filesystem formats definition are defined and found that blivet module where these are defined and I came across these following two files.
I discovered in the in blivet it actually has no fileformats defined for "udf" which is real root cause here.
I have copied the definition present present for "iso9660" to create "udf" equivalents as a test below. (Lines between 1362 and 1372)

File : /usr/lib/python3.6/site-packages/blivet/formats/fs.py

1350 class Iso9660FS(FS):
1351 
1352     """ ISO9660 filesystem. """
1353     _type = "iso9660"
1354     _modules = ["iso9660"]
1355     _supported = True
1356     _mount_class = fsmount.Iso9660FSMount
1357 
1358 
1359 register_device_format(Iso9660FS)
1360 
1361 
1362 # RH case - Copying defination of the above Iso9660FS class for UDF
1363 class UdfFS(FS):
1364 
1365     """ UDF filesystem. """
1366     _type = "udf"
1367     _modules = ["udf"]
1368     _supported = True
1369     _mount_class = fsmount.UdfFSMount
1370 
1371 
1372 register_device_format(UdfFS)

Along with this I needed to edit one other file which contained these "Iso9660FSMount" and "UdfFSMount" definitions.
I added that these lines from Line 165 to Line 167 again just copying the "Iso9660FSMount" class

File: /usr/lib/python3.6/site-packages/blivet/tasks/fsmount.py

162 class Iso9660FSMount(FSMount):
163     options = ["ro"]
164 
165 # RH case - Copying defination form the above Iso9660FSMount class for UDF
166 class UdfFSMount(FSMount):
167     options = ["ro"]

Using this I created an inst.updates image and used it during installation. (Attaching the image)

inst.updates=http://<WEBSERVER_URL>/updates-protectudf-rhel89.img

I was able to install the OS with the Bootable USB and it successfully resolved the LABEL of the installation media to the UDF formatted USB and added it to protected device.

$ lsinitrd ~/temp/updates/updates-protectudf-rhel89.img 
Image: /home/amepatil/temp/updates/updates-protectudf-rhel89.img: 16K
========================================================================
Version: 

Arguments: 
dracut modules:
========================================================================
drwxr-x---   3 root     root            0 May 10 02:29 .
drwxr-xr-x   3 root     root            0 May 10 02:29 usr
dr-xr-xr-x   3 root     root            0 May  1 03:24 usr/lib
drwxr-xr-x   3 root     root            0 May 10 02:29 usr/lib/python3.6
drwxr-xr-x   3 root     root            0 May 10 02:29 usr/lib/python3.6/site-packages
drwxr-xr-x   4 root     root            0 May 10 02:29 usr/lib/python3.6/site-packages/blivet
drwxr-xr-x   2 root     root            0 May 10 02:33 usr/lib/python3.6/site-packages/blivet/formats
-rw-r--r--   1 root     root        51101 May 10 02:33 usr/lib/python3.6/site-packages/blivet/formats/fs.py
drwxr-xr-x   2 root     root            0 May 10 02:33 usr/lib/python3.6/site-packages/blivet/tasks
-rw-r--r--   1 root     root         6209 May 10 02:33 usr/lib/python3.6/site-packages/blivet/tasks/fsmount.py
========================================================================

As we see in the anaconda logs, we are able to resolve the LABEL to "sda" i.e. my USB key and its marked as protected and was not overwritten by anaconda during OS installation.

22:45:01,375 WARNING org.fedoraproject.Anaconda.Modules.Storage:DEBUG:blivet:resolved 'LABEL=RHEL-8-9-0-BaseOS-x86_64' to 'sda' (disk)
22:45:01,375 WARNING org.fedoraproject.Anaconda.Modules.Storage:DEBUG:anaconda.storage:Protected device spec LABEL=RHEL-8-9-0-BaseOS-x86_64 resolved to sda.
22:45:01,375 WARNING org.fedoraproject.Anaconda.Modules.Storage:DEBUG:blivet:               DeviceTree.get_device_by_path: path: /dev/zram0 ; incomplete: False ; hidden: False ;
22:45:01,375 WARNING org.fedoraproject.Anaconda.Modules.Storage:DEBUG:blivet:               DeviceTree.get_device_by_path returned None
22:45:01,376 WARNING org.fedoraproject.Anaconda.Modules.Storage:DEBUG:blivet:failed to resolve '/dev/zram0'
22:45:01,376 WARNING org.fedoraproject.Anaconda.Modules.Storage:DEBUG:anaconda.storage:Marking device sda as protected.
22:45:01,376 WARNING org.fedoraproject.Anaconda.Modules.Storage:DEBUG:anaconda.storage:Marking device sr0 as protected.

Additional note:

The issue does not occur when using UDF formatted ISO directly into the CD-ROM as opposed to the bootable USB
This is because when its in the CR-ROM slot, the logic which we discussed earlier for protected device in the below files kicks in and attempt to add the "cdrom" type device as protected device. Since this is not done by resolving filesystem LABEL's like "['LABEL=RHEL-8-8-0-BaseOS-x86_64']" or with the filesystem type like "iso9660" and "udf" but rather device type (i.e. cdrom)

File - /usr/lib64/python3.6/site-packages/pyanaconda/modules/storage/devicetree/model.py

301 
302         # For image installation setup_disk_images method marks all local
303         # storage disks as ignored so they are protected from teardown.
304         # Here we protect also cdrom devices from tearing down that, in case of
305         # cdroms, involves unmounting which is undesirable (see bug #1671713).
306         protected.extend(dev for dev in self.devicetree.devices if dev.type == "cdrom")
307 

What were you trying to do that didn't work?

If we create USB bootable for OS installation using the UDF formatted ISO, anaconda does not recognize this USB as protected device and wipes it during partitioning even though it contains the installation media from where we booted resulting in the USB being wiped and OS installation fails. See the above details which has more information.

Please provide the package NVR for which bug is seen:

ISO created using content of RHEL 8.9 GA ISO

How reproducible:

Everytime with USB created using UDF formatted ISO

Steps to reproduce

• Create a ISO and add a test file of 4+ GiB and rebuild the ISO and create a bootable USB using this ISO

      # mount /root/rhel-8.9-x86_64-boot.iso /media
      # shopt -s dotglob
      # cp -av /media /mnt
      # dd if=/dev/zero of=/mnt/image.img bs=1M count=5120 status=progress
      # cd /mnt
      # genisoimage -o /root/rhel89boot.iso -allow-limited-size -b isolinux/isolinux.bin -J -R -l -c isolinux/boot.cat -no-emul-boot -boot-load-size 4 -boot-info-table -eltorito-alt-boot -e images/efiboot.img -no-emul-boot -graft-points -joliet-long -V "RHEL-8-9-0-BaseOS-x86_64" .
      # isohybrid --uefi /root/rhel89boot.iso
      

• Create a Bootable USB using this ISO and check that its actually UDF formatted.

      # dd if=/root/rhel89boot.iso of=/dev/sda status=progress
  
      # lsblk -f
      NAME                                          FSTYPE      FSVER LABEL                    UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
      sda                                           udf         1.02  RHEL-8-9-0-BaseOS-x86_64 662ebd7e000014a3                                    
      ├─sda1                                        udf         1.02  RHEL-8-9-0-BaseOS-x86_64 662ebd7e000014a3                                    
      └─sda2                                        vfat        FAT16 ANACONDA                 65C9-7C42                                           
      

• Use this USB to install OS with kickstart file with automatic partition and found system was trying to partition and overwrite my USB drive itself.

      bootloader --append=" crashkernel=auto console=tty0 console=ttyS0,115200n8" --location=mbr
      clearpart --all --initlabel
      autopart --type lvm
      

Expected results

Expecting that anaconda should not wipe the installation source i.e. Bootable USB formatted as UDF during installation

Actual results

USB bootable formatted as UDF is not recognized by anaconda and hence is wiped during installation since its not part of protected device.