-
Bug
-
Resolution: Unresolved
-
Normal
-
None
-
rhel-8.10, rhel-9.4
-
None
-
Low
-
rhel-anaconda
-
ssg_front_door
-
None
-
False
-
False
-
-
No
-
Red Hat Enterprise Linux
-
None
-
None
-
None
-
Unspecified Release Note Type - Unknown
-
None
What were you trying to do that didn't work?
Trying to install a disk image with an ostree using anaconda command line, anaconda crashes right at the end and leaves the disk image's LVM volumes mounted + ostree file system, which makes the system be unusable to create disk images further.
The issue seems "ostree" specific, it doesn't occur with url --url installations.
Please provide the package NVR for which bug is seen:
anaconda-33.16.10.5-1.el8.x86_64
anaconda-34.25.4.9-1.el9_4.x86_64
How reproducible:
Always
Steps to reproduce
Preparation
- Install required package
# yum -y install anaconda osbuild-composer composer-cli bash-completion ostree nginx
- Enable the required services
# systemctl enable osbuild-composer.socket nginx.service --now
- Build an image
# cat > minimal-rhel8.toml << EOF name = "minimal-rhel8" description = "minimal blueprint for ostree commit" version = "1.0.0" modules = [] groups = [] distro = "rhel-8" EOF # composer-cli blueprints push minimal-rhel8.toml # composer-cli compose start-ostree minimal-rhel8 edge-commit Compose e38c972d-7ce1-4d44-a1d0-c3274b245505 added to the queue # while ! composer-cli compose image e38c972d-7ce1-4d44-a1d0-c3274b245505 --filename minimal-rhel8.tar; do sleep 60; done ...
- Extract the ostree to the NGINX html root
# tar xf minimal-rhel8.tar -C /usr/share/nginx/html
- Create a disk image
# qemu-img create -f raw image.raw 20G
- Create the kickstart
# cat > ostree.ks << EOF text ostreesetup --nogpg --osname=rhel --remote=edge --url=http://$(getent ahostsv4 $(hostname) | awk '/STREAM/ { print $1 }')/repo/ --ref=rhel/8/x86_64/edge lang en_US keyboard us zerombr bootloader --disabled clearpart --all --initlabel autopart timezone UTC rootpw redhat reboot firewall --disabled skipx EOF
Reproduce the issue
- Install the disk image
# anaconda --image $PWD/image.raw --kickstart $PWD/ostree.ks
Expected results
Completes successfully
Actual results
Fails:
Installation complete Use of this product is subject to the license agreement found at: /usr/share/redhat-release/EULA Error in atexit._run_exitfuncs: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/dasbus/client/handler.py", line 477, in _get_method_reply return self._handle_method_error(error) File "/usr/lib/python3.6/site-packages/dasbus/client/handler.py", line 497, in _handle_method_error raise exception from None dasbus.error.DBusError: umount of /mnt/sysimage failed (32)
Temporary mounts are remaining on the host:
# mount | grep /mnt /dev/mapper/anaconda_image-root on /mnt/sysimage type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota) /dev/mapper/anaconda_image-root on /mnt/sysroot type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota) devtmpfs on /mnt/sysroot/dev type devtmpfs (rw,nosuid,seclabel,size=892724k,nr_inodes=223181,mode=755) devtmpfs on /mnt/sysimage/ostree/deploy/rhel/deploy/e9ebaf5a52a3492e4ce0c7ce2d465f7830afc1a78236212c652b8f8e169304dd.0/dev type devtmpfs (rw,nosuid,seclabel,size=892724k,nr_inodes=223181,mode=755) sysfs on /mnt/sysroot/sys type sysfs (rw,relatime,seclabel) sysfs on /mnt/sysimage/ostree/deploy/rhel/deploy/e9ebaf5a52a3492e4ce0c7ce2d465f7830afc1a78236212c652b8f8e169304dd.0/sys type sysfs (rw,relatime,seclabel)
Here above we can see the ostree mount still present, preventing the unmount of /mnt/sysimage to occur.
Disk image is still open, disk image SWAP is used:
# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
loop0 7:0 0 20G 0 loop
└─image 253:2 0 20G 0 anaconda
├─image1 253:3 0 1G 0 part
└─image2 253:4 0 19G 0 part
├─anaconda_image-swap 253:5 0 2G 0 lvm [SWAP]
└─anaconda_image-root 253:6 0 17G 0 lvm /mnt/sysimage
[...]
Additional details
The following must be done to recover the situation:
# swapoff /dev/dm-5 # umount /mnt/sysimage/ostree/deploy/rhel/deploy/e9ebaf5a52a3492e4ce0c7ce2d465f7830afc1a78236212c652b8f8e169304dd.0/dev # umount /mnt/sysroot # umount /mnt/sysimage
But this isn't enough, because for some reason, despite having /dev/anaconda_image/root mounted anymore, the LVM volume cannot be deactivated:
# mount | grep /mnt
--> nothing
# vgchange -anl anaconda_image
Logical volume anaconda_image/root contains a filesystem in use.
Can't deactivate volume group "anaconda_image" with 1 open logical volume(s)
It appears "system" services still have `/mnt/sysroot` in their mount table, which tends to indicate a mount namespace issue:
# grep /mnt /proc/*/mounts /proc/1072/mounts:/dev/mapper/anaconda_image-root /mnt/sysroot xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /proc/20410/mounts:/dev/mapper/anaconda_image-root /mnt/sysroot xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /proc/20411/mounts:/dev/mapper/anaconda_image-root /mnt/sysroot xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /proc/20436/mounts:/dev/mapper/anaconda_image-root /mnt/sysroot xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /proc/682/mounts:/dev/mapper/anaconda_image-root /mnt/sysroot xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /proc/754/mounts:/dev/mapper/anaconda_image-root /mnt/sysroot xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /proc/805/mounts:/dev/mapper/anaconda_image-root /mnt/sysroot xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 # for pid in $(grep /mnt /proc/*/mounts | cut -f3 -d'/'); do ps -eaf | grep -w $pid | grep -v grep; done root 1072 1 0 13:45 ? 00:00:00 /usr/sbin/rsyslogd -n root 20410 1 0 13:49 ? 00:00:00 nginx: master process /usr/sbin/nginx nginx 20411 20410 0 13:49 ? 00:00:01 nginx: worker process nginx 20411 20410 0 13:49 ? 00:00:01 nginx: worker process root 20436 1 0 13:51 ? 00:00:00 /usr/libexec/osbuild-composer/osbuild-worker -unix /run/osbuild-composer/job.socket root 682 1 0 13:45 ? 00:00:00 /usr/lib/systemd/systemd-udevd root 754 1 0 13:45 ? 00:00:00 /sbin/auditd root 805 1 0 13:45 ? 00:00:00 /usr/sbin/NetworkManager --no-daemon
Hence, to recover, it's required to restart all those services!
# for pid in $(grep /mnt /proc/*/mounts | cut -f3 -d'/'); do awk -F '/' '/name=systemd/ { print $NF }' "name=systemd" /proc/$pid/cgroup; done) | while read svc; do service $(basename $svc .service) restart; done # vgchange -anl anaconda_image 0 logical volume(s) in volume group "anaconda_image" now active
Only then system is ready again to install a new disk image.
