Uploaded image for project: 'RHEL'
  1. RHEL
  2. RHEL-19235

readonly-root.service starts too early causing a race with /var mounting

XMLWordPrintable

    • None
    • Moderate
    • rhel-sst-cs-plumbers
    • ssg_core_services
    • 8
    • Dev ack
    • False
    • Hide

      None

      Show
      None
    • None
    • Red Hat Enterprise Linux
    • None
    • None
    • None
    • All
    • None

      What were you trying to do that didn't work?

      A customer followed the RHEL8 documentation related to readonly-root .
      On his system having a dedicated /var, this ends up breaking:

      [...] readonly-root[574]: mount: /var/lib/stateless/writable: mount point does not exist.

      The issue happens because of a race with mounting /var, as shown by the journal excerpt below:

      # journalctl -b -u readonly-root.service -u var.mount
-- Logs begin at Tue 2023-12-12 13:16:02 CET, end at Tue 2023-12-12 13:16:48 CET. --
Dec 12 13:16:03 vm-ro89 systemd[1]: Starting Configure read-only root support...
Dec 12 13:16:03 vm-ro89 systemd[1]: Mounting /var...
Dec 12 13:16:03 vm-ro89 systemd[1]: Mounted /var.
Dec 12 13:16:04 vm-ro89 systemd[1]: Started Configure read-only root support.

      Here above, we see both units run simultaneously.
      I cannot reproduce as such because it's a race, but I can force reproducing by intentionally delaying the mount of /var in /etc/fstab:

      UUID=9f4f1b75-6434-405d-911a-7dc79cce9dcd /var                    ext4    defaults,x-systemd.after=readonly-root.service        1 2

      With such setting, I then reproduce constantly:

      # journalctl -b -u readonly-root.service
-- Logs begin at Tue 2023-12-12 13:21:23 CET, end at Tue 2023-12-12 13:23:38 CET. --
Dec 12 13:21:24 vm-ro89 systemd[1]: Starting Configure read-only root support...
Dec 12 13:21:24 vm-ro89 readonly-root[589]: mount: /var/lib/stateless/writable: mount point does not exist.
Dec 12 13:21:24 vm-ro89 readonly-root[598]: cp: with --parents, the destination must be a directory
Dec 12 13:21:24 vm-ro89 readonly-root[598]: Try 'cp --help' for more information.
[...]

      The root cause for the issue is a bad ordering: readonly-root.service needs to execute after local-fs.target, not before.

      • Current order: 
          Before=shutdown.target emergency.service emergency.target systemd-tmpfiles-setup.service local-fs.target systemd-random-seed.service
  After=systemd-remount-fs.service
      • Proper order: 
          Before=shutdown.target emergency.service emergency.target systemd-tmpfiles setup.service systemd-random-seed.service
  After=systemd-remount-fs.service local-fs.target

      Please provide the package NVR for which bug is seen:

      readonly-root-10.00.18-1.el8.noarch

      How reproducible:

      Always, see above.

              jamacku@redhat.com Jan Macku
              rhn-support-rmetrich Renaud Métrich
              Jan Macku Jan Macku
              Daniel Rusek Daniel Rusek
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated: