Description of problem:

Tuned sysctl plugin wrongly processes /etc/sysctl.conf twice therefore undoing settings previously applied in /etc/sysctl.d/99-tripleo.conf.

Code:

def _apply_system_sysctl():
files = {}
for d in SYSCTL_CONFIG_DIRS:
try:
flist = os.listdir(d)
except OSError:
continue
for fname in flist:
if not fname.endswith(".conf"):
continue
if fname not in files:
files[fname] = d

for fname in sorted(files.keys()):
d = files[fname]
path = "%s/%s" % (d, fname)
_apply_sysctl_config_file(path) <----- SYSCTL_CONFIG_DIRS = [ "/run/sysctl.d","/etc/sysctl.d" ]
_apply_sysctl_config_file("/etc/sysctl.conf") <----- And last /etc/sysctl.conf which was already applied via symlink /etc/sysctl.d/99-sysctl.conf

Sample: /etc/sysctl.d from a RHEL 8.4 host
$ ls -al /etc/sysctl.d/
total 28
drwxrwxrwx. 2 yank yank 4096 Jun 1 07:54 .
drwxrwxrwx. 72 yank yank 4096 Jun 8 03:47 ..
rw-rw-rw. 1 yank yank 525 Jan 14 2022 50-libreswan.conf
rw-rw-rw. 1 yank yank 50 Jun 1 07:54 60-kernel.core_pattern.conf
rw-rw-rw. 1 yank yank 143 Jun 1 07:54 60-tf-node-init.conf
lrwxrwxrwx. 1 yank yank 14 Jun 7 2021 99-sysctl.conf -> ../sysctl.conf
rw-rw-rw. 1 yank yank 1378 Jun 1 07:54 99-tripleo.conf <----- Settings are overridden when sysctl.conf is wrongly applied the second time.

A review of the systemd-sysctl.service handling and changelogs reveals what should likely be the correct approach:

From: systemd-239/NEWS

• The systemd-sysctl tool no longer natively reads the file
  /etc/sysctl.conf. If desired, the file should be symlinked
  from /etc/sysctl.d/99-sysctl.conf. Apart from providing
  legacy support by a symlink rather than built-in code, it
  also makes the otherwise hidden order of application of the
  different files visible. (Note that this partly reverts to a
  pre-198 application order of sysctl knobs!)

Which matches the code:

r = conf_files_list_strv(&files, ".conf", NULL, 0, (const char**) CONF_PATHS_STRV("sysctl.d"));
if (r < 0)

{ log_error_errno(r, "Failed to enumerate sysctl.d files: %m"); goto finish; }

Version-Release number of selected component (if applicable):

• tuned-2.18.0-3.1.20220714git70732a57.el8fdp.noarch

How reproducible:

• Everytime tuned is run with reapply_sysctl = 1

Steps to Reproduce:
1. set vm.nr_hugepages=1024 in /etc/sysctl.conf
2. set vm.nr_hugepages=0 in /etc/sysctl.d/99-tripleo.conf
3. apply with restart of systemd-sysctl.service
4. check sysctl -a | grep hugepages to confirm correctly set to 0
5. restart tuned service
4. check sysctl -a | grep hugepages to confirm incorrectly set to 1024

Actual results:

• Settings that are in 99-tripleo.conf get incorrectly overridden

Expected results:

• Settings match what would be applied by systemd-sysctl.service.

Additional info:

• This could be impacting production workloads.
• This is a hot topic for some RHOSP consulting work and needs to be addressed.