This is a continuation of RHEL-73701 - nfsd cannot open shared libraries by default.

I added a rule to the default policy to allow nfsd to open files, whatever their type is, as long as the files are hosted on the exported shares:

# echo "allow perm=open ppid=2 exe=nfsd : dir=/var/export/" > /etc/fapolicyd/rules.d/20-nfsd.rules
# systemctl restart fapolicyd

With such setup, 15 rules are enabled, all are default ones except rule 3:

# fapolicyd-cli --list
[...]
1. allow perm=any uid=0 : dir=/var/tmp/
2. allow perm=any uid=0 trust=1 : all
3. allow perm=open ppid=2 exe=nfsd : dir=/var/export/
4. allow perm=open exe=/usr/bin/rpm : all
5. allow perm=open exe=/usr/bin/python3.9 comm=dnf : all
6. deny_audit perm=any pattern=ld_so : all
7. deny_audit perm=any all : ftype=application/x-bad-elf
8. allow perm=open all : ftype=application/x-sharedlib trust=1
9. deny_audit perm=open all : ftype=application/x-sharedlib
10. allow perm=execute all : trust=1
11. allow perm=open all : ftype=%languages trust=1
12. deny_audit perm=any all : ftype=%languages
13. allow perm=any all : ftype=text/x-shellscript
14. deny_audit perm=execute all : all
15. allow perm=open all : all

What were you trying to do that didn't work?

Initially the NFS client mounts NFS share /var/export onto /mnt and the NFS share is configured to hold a library file (/var/export/libssl.so.3.2.2).

On reboot of the NFS server, the NFS client, which already had the mount to NFS server active, cannot open library files stored on the export (e.g. /mnt/libssl.so.3.2.2), because rule 9 hits:

rule=9 dec=deny_audit perm=open auid=-1 pid=1077 exe=nfsd : path=/ ftype=application/x-sharedlib trust=0

Digging further, it appears that when fapolicyd checks the file descriptor "submitted" by nfsd (file descriptor 8), the symlink resolution is broken: the target is slash (/) instead of /var/export/libssl.so.3.2.2, explaining why the rule hit, since the custom rule 3 only allows files under /var/export.
This can be confirmed using strace or a stap script. With strace, we can see that file descriptor 8 resolves badly, knowing that strace and fapolicyd resolve symlinks similarly, through performing a readlink(/proc/self/fd/<fdnumber>):

1450  07:26:14.951284 fstat(8</>, {st_dev=makedev(0xfc, 0x10), st_ino=131, st_mode=S_IFREG|0755, st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=1872, st_size=957480, ...) = 0 <0.000043>

Note that file descriptor 8 is the proper file /var/export/libssl.so.3.2.2, it's only the name that is incorrect.

It hence seems that there is a bug in procfs or the kernel in general, however I'm filing against fapolicyd component for now to get another pair of eyes.

What is the impact of this issue to you?

Cannot open files from a NFS client.

Please provide the package NVR for which the bug is seen:

RHEL9:
fapolicyd-1.3.3-100.el9
kernel-core-5.14.0-503.21.1.el9_5

RHEL8:
fapolicyd-1.3.2-1.el8
kernel-core-4.18.0-553.30.1.el8_10 and later

How reproducible is this bug?:

Always

Steps to reproduce

1. Configure a NFS server with fapolicyd, sharing a dedicated filesystem (/var/export) and hosting a library

   # cat /etc/exports
   /var/export	*(rw,sync,no_root_squash,no_subtree_check)
   
   # grep /var/export /etc/fstab 
   /dev/vdb		/var/export		xfs	defaults,nodev	0 0
   
   # cp /lib64/libssl.so.3.2.2 /var/export/
   
   # cat /etc/fapolicyd/rules.d/20-nfsd.rules 
   allow perm=open ppid=2 exe=nfsd : dir=/var/export/
   

2. Reboot the NFS server to be operational under fapolicyd
3. On the NFS client, mount the share from NFS server and confirm accessing the library works fine

   # mount vm-fapolicyd9:/var/export /mnt
   # hexdump -C -n 32 /mnt/libssl.so.3.2.2 
   00000000  7f 45 4c 46 02 01 01 00  00 00 00 00 00 00 00 00  |.ELF............|
   00000010  03 00 3e 00 01 00 00 00  d0 66 02 00 00 00 00 00  |..>......f......|
   00000020
   

4. Reboot the NFS server
5. On the NFS client, try accessing the library again (failure)

   # hexdump -C -n 32 /mnt/libssl.so.3.2.2 
   hexdump: /mnt/libssl.so.3.2.2: Operation not permitted
   hexdump: all input file arguments failed
   

Expected results

Can access the file

Actual results

Rule 9 hitting because resolved path is not correct:

rule=9 dec=deny_audit perm=open auid=-1 pid=1076 exe=nfsd : path=/ ftype=application/x-sharedlib trust=0

Additional information

The following stap script will shows the readlink resolution and confirm the issue:

# cat fapolicyd.stp
global traced

probe syscall.readlink {
	if (execname() != "fapolicyd") next
	if (path_unquoted !~ "^/proc/self/fd/") next
	traced[tid()] = buf_uaddr
	printf("readlink(%s) -> ", path_unquoted)
}

probe syscall.readlink.return { 
	if (! traced[tid()]) next
	printf("%s\n", user_string(traced[tid()]))
	delete traced[tid()]
}

Run it, then restart fapolicyd (to clear caches):

# stap -v ./fapolicyd.stp &
# systemctl restart fapolicyd

Then access the file from the NFS client, you will see this below (fd number may vary of course):

readlink(/proc/self/fd/9) -> /