Uploaded image for project: 'RHEL'
  1. RHEL
  2. RHEL-17394

System drops to emergency when restarting a mount

    • systemd-219-78.el7_9.11
    • Yes
    • Important
    • ZStream, Patch, Regression
    • rhel-sst-cs-plumbers
    • ssg_core_services
    • None
    • False
    • Hide

      None

      Show
      None
    • None
    • None
    • If docs needed, set a value
    • None

      Description of problem:

      When restarting a mount unit using "systemctl restart <mountpoint>.mount" command, the system enters Emergency target due to failure of local-fs.target.
      The issue happens when having some unrelated mount point having a symlink component in its path, as shown in the example below:
      -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

      1. ls -l /mnt
        total 0
        drwxr-xr-x. 2 root root 6 Aug 30 11:52 mountpoint
        drwxr-xr-x. 2 root root 6 Aug 30 11:52 othermountpoint
        lrwxrwxrwx. 1 root root 1 Aug 30 11:58 symlinktoself -> .
      1. tail -2 /etc/fstab
        /root/test_fs1 /mnt/symlinktoself/mountpoint xfs defaults 0 0
        /root/test_fs2 /mnt/othermountpoint xfs defaults 0 0
                    • 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

      Here above, /etc/fstab uses a symlink component in first mount "/mnt/symlinktoself/mountpoint". The real path is "/mnt/mountpoint" but that path is not used on purpose.

      When restarting mnt-othermountpoint.mount unit, system enters Emergency target.

      Version-Release number of selected component (if applicable):

      systemd-219-78.el7_9.7.x86_64

      How reproducible:

      Always

      Steps to Reproduce:
      1. Create the above setup

      -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

      1. truncate -s1G /root/test_fs1 && mkfs.xfs /root/test_fs1
      2. truncate -s1G /root/test_fs2 && mkfs.xfs /root/test_fs2
      3. cat >> /etc/fstab << EOF
        /root/test_fs1 /mnt/symlinktoself/mountpoint xfs defaults 0 0
        /root/test_fs2 /mnt/othermountpoint xfs defaults 0 0
        EOF
      1. mkdir /mnt/othermountpoint /mnt/mountpoint
      2. ln -s . /mnt/symlinktoself
                    • 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

      2. Reboot the system and confirm mounts are up

      -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

      1. mount | grep test_
        /root/test_fs1 on /mnt/mountpoint type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
        /root/test_fs2 on /mnt/othermountpoint type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
                    • 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

      3. Restart mnt-othermountpoint.mount unit

      -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

      1. systemctl restart mnt-othermountpoint.mount unit
                    • 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

      Actual results:

      Emergency target being entered due to "mnt-symlinktoself-mountpoint.mount" failing because it's already mounted.

      Expected results:

      No issue

      Additional info:

      Digging into this, it appears the issue happens because of systemd not considering "mnt-symlinktoself-mountpoint.mount" and "mnt-mountpoint.mount" units being the same. systemd indeed sees 2 different units:

      1. the "real" mount, based on /proc/self/mountinfo view, which is some "transient mount"

      -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

      1. systemctl status mnt-mountpoint.mount
        ● mnt-mountpoint.mount - /mnt/mountpoint
        Loaded: loaded (/proc/self/mountinfo)
        Active: active (mounted) since Wed 2023-08-30 12:02:34 CEST; 51min ago
        Where: /mnt/mountpoint
        What: /dev/loop0

      Aug 30 12:02:34 vm-systemd7 systemd[1]: mnt-mountpoint.mount changed dead -> mounted

      1. systemctl cat mnt-mountpoint.mount
        No files found for mnt-mountpoint.mount.
                    • 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

      2. the "static" mount (generated by fstab-generator), which got mounted at boot but is now considered as unmounted

      -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

      1. systemctl status mnt-symlinktoself-mountpoint.mount
        ● mnt-symlinktoself-mountpoint.mount - /mnt/symlinktoself/mountpoint
        Loaded: loaded (/etc/fstab; bad; vendor preset: disabled)
        Active: inactive (dead) since Wed 2023-08-30 12:02:34 CEST; 52min ago
        Where: /mnt/symlinktoself/mountpoint
        What: /root/test_fs1
        Docs: man:fstab(5)
        man:systemd-fstab-generator(8)
        Process: 575 ExecMount=/bin/mount /root/test_fs1 /mnt/symlinktoself/mountpoint -t xfs (code=exited, status=0/SUCCESS)

      Aug 30 12:02:34 vm-systemd7 systemd[1]: mnt-symlinktoself-mountpoint.mount changed dead -> mounting
      Aug 30 12:02:34 vm-systemd7 systemd[1]: Mounting /mnt/symlinktoself/mountpoint...
      Aug 30 12:02:34 vm-systemd7 systemd[575]: Executing: /bin/mount /root/test_fs1 /mnt/symlinktoself/mountpoint -t xfs
      Aug 30 12:02:34 vm-systemd7 systemd[1]: Child 575 belongs to mnt-symlinktoself-mountpoint.mount
      Aug 30 12:02:34 vm-systemd7 systemd[1]: Unwatching 575.
      Aug 30 12:02:34 vm-systemd7 systemd[1]: mnt-symlinktoself-mountpoint.mount mount process exited, code=exited status=0
      Aug 30 12:02:34 vm-systemd7 systemd[1]: mnt-symlinktoself-mountpoint.mount changed mounting -> mounted
      Aug 30 12:02:34 vm-systemd7 systemd[1]: Job mnt-symlinktoself-mountpoint.mount/start finished, result=done
      Aug 30 12:02:34 vm-systemd7 systemd[1]: Mounted /mnt/symlinktoself/mountpoint.
      Aug 30 12:02:34 vm-systemd7 systemd[1]: mnt-symlinktoself-mountpoint.mount changed mounted -> dead

      1. systemctl cat mnt-symlinktoself-mountpoint.mount
      2. /run/systemd/generator/mnt-symlinktoself-mountpoint.mount
      3. Automatically generated by systemd-fstab-generator

      [Unit]
      SourcePath=/etc/fstab
      Documentation=man:fstab(5) man:systemd-fstab-generator(8)
      Before=local-fs.target

      [Mount]
      What=/root/test_fs1
      Where=/mnt/symlinktoself/mountpoint
      Type=xfs
      -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

      Due to this, when remounting some unrelated mount, systemd enqueues "local-fs.target/restart" job, which makes it realize "mnt-symlinktoself-mountpoint.mount" is not mounted, causing a mount to be done, which fails in "already mounted" because the mount is actually "mnt-mountpoint.mount".

      The issue may apparently also happen during a "daemon-reload" but I wasn't able to trigger it.

              rhn-support-sbroz Stepan Broz
              rhn-support-rmetrich Renaud Métrich
              Stepan Broz Stepan Broz
              Bryan Mason Bryan Mason
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Created:
                Updated:
                Resolved: