Description of problem:

When the system is booting with the NFS dracut modules, e.g. when using the generic initramfs (due to presence of dracut-config-generic package), the following ordering cycle is seen at shutdown:
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
Sep 26 13:44:54 vm-rhel8 systemd[1]: local-fs.target: Found ordering cycle on var-lib-nfs-rpc_pipefs.mount/stop
Sep 26 13:44:54 vm-rhel8 systemd[1]: local-fs.target: Found dependency on systemd-tmpfiles-setup.service/stop
Sep 26 13:44:54 vm-rhel8 systemd[1]: local-fs.target: Found dependency on local-fs.target/stop
Sep 26 13:44:54 vm-rhel8 systemd[1]: local-fs.target: Job var-lib-nfs-rpc_pipefs.mount/stop deleted to break ordering cycle starting with local-fs.target/stop
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

After much digging, I was able to find out that the reason was having an issue with /var/lib/nfs/rpc_pipefs mount:
the unit has the following ordering dependencies (as seen by "systemd-analyze dump"):
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
Before: local-fs.target (origin-mountinfo-implicit)
Before: rpc_pipefs.target (destination-file)
Before: umount.target (origin-mountinfo-default)
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

whereas it should not and just have those:
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
Before: rpc_pipefs.target (destination-file)
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

Additionally (but unrelated), in such case, we can see the mount is not correct, it is mounted as:
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

whereas it should be mounted as (see What):
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

The root cause is the NFS dracut module:

1. firstly mounting with What == rpc_pipefs instead of What == sunrpc (this causes the discrepancy in mount parameters):

-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

1. diff -u /usr/lib/dracut/modules.d/95nfs/nfs-start-rpc.sh.orig /usr/lib/dracut/modules.d/95nfs/nfs-start-rpc.sh
   • • /usr/lib/dracut/modules.d/95nfs/nfs-start-rpc.sh.orig 2022-09-26 13:34:11.416891907 +0200
       +++ /usr/lib/dracut/modules.d/95nfs/nfs-start-rpc.sh 2022-09-26 13:34:19.152877777 +0200
       @@ -4,7 +4,7 @@
       if modprobe sunrpc || strstr "$(cat /proc/filesystems)" rpc_pipefs; then

[ ! -d /var/lib/nfs/rpc_pipefs/nfs ] && \

• mount -t rpc_pipefs rpc_pipefs /var/lib/nfs/rpc_pipefs
  + mount -t rpc_pipefs sunrpc /var/lib/nfs/rpc_pipefs

1. Start rpcbind or rpcbind
2. FIXME occasionally saw 'rpcbind: fork failed: No such device' – why?
   • • • • • • • 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

2. secondly the cleanup doing a bind-mount of /var/lib/nfs/rpc_pipefs to /sysroot/var/lib/nfs/rpc_pipefs, which leads to having the unexpected "Before=local-fs.target umount.target" dependencies and "origin-mountinfo-implicit/origin-mountinfo-default"

-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

1. diff -u /usr/lib/dracut/modules.d/95nfs/nfsroot-cleanup.sh.orig /usr/lib/dracut/modules.d/95nfs/nfsroot-cleanup.sh
   • • /usr/lib/dracut/modules.d/95nfs/nfsroot-cleanup.sh.orig 2022-09-26 13:43:19.441194953 +0200
       +++ /usr/lib/dracut/modules.d/95nfs/nfsroot-cleanup.sh 2022-09-26 13:43:59.199659403 +0200
       @@ -21,7 +21,7 @@

if [ -d $NEWROOT/$rpcpipefspath ]; then

1. mount --move does not seem to work???

• mount --bind /var/lib/nfs/rpc_pipefs $NEWROOT/$rpcpipefspath
  + #mount --bind /var/lib/nfs/rpc_pipefs $NEWROOT/$rpcpipefspath
  umount /var/lib/nfs/rpc_pipefs 2>/dev/null
  else
  umount /var/lib/nfs/rpc_pipefs 2>/dev/null
  • • • • • • • 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

Note: here above, not remounting to /sysroot is probably NOT correct when having a NFS root; this hack is just to show what fixes the issue on regulard systems actually.

Version-Release number of selected component (if applicable):

dracut-network-049-202.git20220511.el8_6.x86_64 and before

How reproducible:

Always

Steps to Reproduce:
1. Install dracut-config-generic package

1. yum -y install dracut-config-generic

2. Rebuild the initramfs

1. dracut -f

3. Setup persistent journal and boot twice

1. mkdir -p /var/log/journal
2. reboot
   ...
3. reboot

4. Check ordering cycles at shutdown

1. journalctl -b -1 | grep local-fs.target

Actual results:

-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
Sep 26 13:44:54 vm-rhel8 systemd[1]: local-fs.target: Found ordering cycle on var-lib-nfs-rpc_pipefs.mount/stop
Sep 26 13:44:54 vm-rhel8 systemd[1]: local-fs.target: Found dependency on systemd-tmpfiles-setup.service/stop
Sep 26 13:44:54 vm-rhel8 systemd[1]: local-fs.target: Found dependency on local-fs.target/stop
Sep 26 13:44:54 vm-rhel8 systemd[1]: local-fs.target: Job var-lib-nfs-rpc_pipefs.mount/stop deleted to break ordering cycle starting with local-fs.target/stop
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

Expected results:

No ordering cycle