Description of problem:

A customer can see sometimes the following message in the journal:
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
systemd[1]: Starting User Manager for UID XXX...
systemd[78237]: pam_unix(systemd-user:session): session opened for user XXXUSERNAME by (uid=0)
systemd[78237]: Failed to fully start up daemon: Permission denied
systemd[1]: user@XXX.service: Failed with result 'protocol'.
systemd[1]: Failed to start User Manager for UID XXX.
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

This leads to tagging the user@.service for that user as Failed, causing a report to be shown in their monitoring tool.

James was able to reproduce this as well in real conditions.
I wasn't, however I was able to reproduce by creating the directory /run/user/UID manually, as root, similar to what /usr/lib/systemd/systemd-user-runtime-dir does when a session gets created.

Normally, /usr/lib/systemd/systemd-user-runtime-dir creates the directory, then mounts a TMPFS filesystem on top of it, which permissions set for the user.

For some (still unknown) reasons, we suspect that /run/user/UID doesn't get deleted at the time a new session comes in, causing the issue above, since /run/user/UID is owned by root, always.
This is because /usr/lib/systemd/systemd-user-runtime-dir doesn't do anything when it detects that /run/user/UID is already a mount point, in particular doesn't mount the TMPFS filesystem, hence the permissions are not correct for "systemd --user" to be able to create its directories later.

Digging into the code, I can see that systemd-user-runtime-dir relies on path_is_mount_point() to find out if it's a mount point:
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
33 static int user_mkdir_runtime_path(const char *runtime_path, uid_t uid, gid_t gid, size_t runtime_dir_size) {
34 int r;
:
45 if (path_is_mount_point(runtime_path, NULL, 0) >= 0)
46 log_debug("%s is already a mount point", runtime_path);
47 else {
48 char options[sizeof("mode=0700,uid=,gid=,size=,smackfsroot=*")
:
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

But that code doesn't work, it's indeed enough to have a top directory (e.g. "/run") be a mount point for condition on line 45 to be true and falling into line 46 and doing nothing.
On RHEL systems, "/run" is a mount point, so it's always true.
In that exact scenario, we would need to check that "/run/user/UID" *is* a mount point itself, and not a parent would be a mount point.

Comparing with Upstream code, I don't see a difference, the code is similarly buggy. However the failing condition on "systemd --user" is not hit because there is a test made in pam_systemd, which then avoids spawning the systemd user instance:
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
systemd[20313]: pam_systemd(systemd-user:session): Runtime directory '/run/user/101166' is not owned by UID 101166, as it should.
systemd[20313]: pam_systemd(systemd-user:session): Not setting $XDG_RUNTIME_DIR, as the directory is not in order.
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

Version-Release number of selected component (if applicable):

systemd-239-41.el8_3.1.x86_64

How reproducible:

Always

Steps to Reproduce:
1. create directory as root

1. mkdir -p /run/user/$(id -u rmetrich)

2. ssh to the system

1. ssh rmetrich@localhost true

Actual results:

systemd --user fails

Expected results:

no failure, and /run/user/rmetrich a TMPFS mountpoint