What were you trying to do that didn't work?

In rsyslog, we are using the <systemd/sd-journal.h> lib to fetch data from journal. On RHEL-8.10, the journal API does not always return all the data resulting in message loss.

This can be reliably reproduced when the following options are set:

# cat /etc/systemd/journald.conf
[Journal]
MaxRetentionSec=10s
MaxFileSec=10s

I took the code from rsyslog and created a minimal reproducer (repro.sh and main.c). Unzip it and execute repro.sh. It will always pass on RHEL-{9/10} but will almost always fail on RHEL-8.10.

The C program opens the journal and tries to retrieve logs from the tail of the database. In the meantime, the script will generate 10000 logs via logger. At the end the script will check if there are missing logs.

reproducer.zip

What is the impact of this issue to you?

By default, rsyslog configuration uses the journal API to retrieve logs, it affects all systems using the default logging config.

Please provide the package NVR for which the bug is seen:

systemd-239-82.el8_10.2.x86_64

How reproducible is this bug?:

I attached a pretty reliable reproducer (~90%).

Expected results

Same behavior as on RHEL-9 and RHEL-10, no message loss when retrieving logs from journal.

Actual results

Message loss on busy systems especially when MaxRetentionSec and MaxFileSec are altered.