Description of problem:

/usr/lib/systemd/systemd-cryptsetup relies on the libcryptsetup.so library, which itself makes use of the libdevmapper.so library.

Inside libdevmapper, there is code to detect the presence/usage of udev.
In such case, the code makes sure to rely on udev to create the symlinks /dev/mapper/luks-XXX back to "../dm-Y" devices.

It appears that during a small window, the code can believe udev is not in use, because /run/udev/control is not present "yet".
This causes the libdevmapper library to create the /dev/mapper/luks-XXX node, causing systemd-udevd rule in charge of creating the symlink to fail, e.g.:
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
systemd-udevd[XXX]: conflicting device node '/dev/mapper/luks-7b4618b9-6a49-4ee1-838a-7691839dde78' found, link to '/dev/dm-13' will not be created
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

The reason for not detecting udev is systemd-udevd and systemd-cryptsetup@ instances start concurrently, systemd-cryptsetup@ can even start a few clock ticks before systemd-udevd (column 22 is start time in clock ticks):
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

1. cat /proc/$(pgrep systemd-udevd)/stat | awk ' { print $22 }'
   795
   
   # ps -eaf | grep cryptsetup
   root 1576 1 4 15:40 ? 00:00:01 /usr/lib/systemd/systemd-cryptsetup attach luksvdb8 /dev/disk/by-uuid/7b4618b9-6a49-4ee1-838a-7691839dde78 none discard
   ...
   
   # cat /proc/1576/stat | awk '{ print $22 }

   '
   763

   • • • • • • • 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

To close this window, we need to generate a "After=systemd-udevd" stanza in systemd-cryptsetup@ excerpts.

Version-Release number of selected component (if applicable):

systemd-239

How reproducible:

Often

Steps to Reproduce:
1. Install a system, then add a disk with 8 Luks devices, no need to create a file system on those
2. Make sure to unlock them automatically (using a key file or Clevis) and add those to /etc/crypttab for automatic decryption

Adding _netdev facilitates troubleshooting because we can then log in using sshd while unlocking happens.
In such case, don't forget to enable "remote-cryptsetup.target"...

3. Boot until the issue occurs or delay startup of systemd-udevd as a hack

-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

1. mkdir -p /etc/systemd/system/systemd-udevd.service.d
2. cat > /etc/systemd/system/systemd-udevd.service.d/delay.conf << EOF
   [Service]
   ExecStartPre=/bin/sleep 30
   • • • • • • • 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

Actual results:

Journal shows "conflicting device node" and some /dev/mapper/luks-XXX devices are not symlinks:
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

1. ll /dev/mapper/
   total 0
   ...
   lrwxrwxrwx 1 root root 8 Jan 9 16:53 luksvdb1 -> ../dm-10
   lrwxrwxrwx 1 root root 7 Jan 9 16:53 luksvdb2 -> ../dm-7
   brw-rw---- 1 root disk 253, 6 Jan 9 16:52 luksvdb3
   lrwxrwxrwx 1 root root 7 Jan 9 16:53 luksvdb5 -> ../dm-3
   lrwxrwxrwx 1 root root 7 Jan 9 16:53 luksvdb6 -> ../dm-8
   brw-rw---- 1 root disk 253, 4 Jan 9 16:52 luksvdb7
   lrwxrwxrwx 1 root root 7 Jan 9 16:53 luksvdb8 -> ../dm-5
   ...

1. systemctl status systemd-udevd
   ...
   Jan 09 16:53:06 vm-clevis86 systemd-udevd[6542]: conflicting device node '/dev/mapper/luksvdb3' found, link to '/dev/>
   Jan 09 16:53:06 vm-clevis86 systemd-udevd[6530]: conflicting device node '/dev/mapper/luksvdb7' found, link to '/dev/>
   ...
   • • • • • • • 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

Expected results:

Always getting symlinks created by udevd