Description of problem:

We were using "journalctl -f" in the OpenShift MCO operator to watch for login events and we noticed that we don't get any (I suspect because we don't ever reach 50+ logins to a RHCOS box)

If you:

• run "journalctl -f" with a MESSAGE_ID= AND
• there are no matching log messages already in the journal at the time the command is run

Then:

• once messages with that MESSAGE_ID do exist, you will not receive them until some number/amount/size of them accumulate

Version-Release number of selected component (if applicable):

systemd 239 (239-45.el8_4.2) on RHCOS
systemd 248 (v248.7-1.fc34) on FCOS

How reproducible:

Every time.

Steps to Reproduce:

1. Create a unit that logs filtered logs to a text file:

sudo cat << EOF > /etc/systemd/system/early_listen.service
[Unit]
Description=Hang journalctl before anyone logs in by filtering both unit and message id
Before=network-online.target

[Service]
User=root
ExecStart=/bin/sh -c "journalctl -b -f -o cat -u systemd-logind.service MESSAGE_ID=8d45620c1a4348dbb17410da57c60c66 | tee /tmp/journal_unit_messageid.txt"

[Install]
WantedBy=multi-user.target
EOF

2.) Enable that unit
systemctl enable early_listen

3.) Clear your journal:
sudo journalctl --flush
sudo journalctl --vacuum-time=1s

4.) Restart your server (may be able to get by without rebooting, I just always did)
sudo systemctl reboot

5.) When the server comes back up, login/ssh into the server
ssh core@192.168.122.16

6.) observe that /tmp/journal_unit_messageid.txt is still empty
cat /tmp/journal_unit_messageid.txt

7.) if you repeatedly login in a loop, after some number of logins (usually > 50) it will snap out of it and events start coming out immediately

Actual results:

Messages do not come out until some unseen/unknown threshold is met

Expected results:

Messages come out as they occur

Additional info:

• It feels like buffering, and at first I thought it was standard output buffering and I was doing something wrong, but after testing I don't think so. I also ran it with "stdbuf" for good measure and it didn't help.

• If MESSAGE_ID is omitted, journalctl works as expected and messages are returned immediately

• Any "journalctl -f" process started after a login message is present in the journal does return (and continue to return) events immediately

• It does not appear to be time based – the messages do not seem to come out after days or hours, the only time they start coming out again is when there seem to be "enough of them"