Undefined A customer configured the journal to rotate daily but also have its file up to 100GB:
$ cat etc/systemd/journald.conf.d/99-journald.conf [Journal] Storage=persistent MaxFileSec=1day SystemMaxFileSize=100M SystemMaxFiles=100 MaxRetentionSec=2month
Despite having these settings, the system journal rotates while still having 8MB size, when some Ansible tasks sends a message.
The reason for rotation is shown below, seen when putting systemd-journald in debug mode:
Jul 24 09:11:40 ... systemd-journald[2506665]: Field hash table of /var/log/journal/.../system.journal has a fill level at 76.0 (253 of 333 items), suggesting rotation. Jul 24 09:11:40 ... systemd-journald[2506665]: /var/log/journal/.../system.journal: Journal header limits reached or header out-of-date, rotating.
It appears indeed that the journal contains many fields, see attached file:
# journalctl --fields | wc -l 351
On many of his systems, he has many fields, so it seems the hardcoded max value (333) is not adequate for some RHEL customers.
Reproducer
- Setup the persistent journal and parameters
# cat > etc/systemd/journald.conf.d/99-journald.conf << EOF [Journal] Storage=persistent SystemMaxFileSize=100M SystemMaxFiles=100 MaxRetentionSec=2month EOF # systemctl restart systemd-journald
- Vacuum the journal and check number of files and number of fields
# journalctl --vacuum-time 10m # ll /var/log/journal/b5af1fc93c0a4c7eae36e167138931d5/ total 8192 -rw-r-----+ 1 root systemd-journal 8388608 Jul 26 08:26 system.journal # journalctl --fields | wc -l 74 - Send messages to the journal using logger facility
# for i in $(seq 1 200); do logger --journald << EOF MESSAGE=$(date) FIELD$i=$i EOF done
Expected result
Still one file but many fields:
# ll /var/log/journal/b5af1fc93c0a4c7eae36e167138931d5/
total 8192
-rw-r-----+ 1 root systemd-journal 8388608 Jul 26 08:26 system.journal
# journalctl --fields | wc -l
275
Actual result
System journal got rotated:
# ll /var/log/journal/b5af1fc93c0a4c7eae36e167138931d5/
total 11388
-rw-r-----+ 1 root systemd-journal 8388608 Jul 26 08:28 system@9ad57c2b2c7f47d9ba4c7c1161aa6b93-00000000000098f7-00063acf1c44b74c.journal
-rw-r-----+ 1 root systemd-journal 8388608 Jul 26 08:28 system.journal
Journal in debug mode shows:
[...] systemd-journald[624]: Field hash table of /var/log/journal/b5af1fc93c0a4c7eae36e167138931d5/system.journal has a fill level at 75.1 (250 of 333 items), suggesting rotation. [...] systemd-journald[624]: /var/log/journal/b5af1fc93c0a4c7eae36e167138931d5/system.journal: Journal header limits reached or header out-of-date, rotating. [...] systemd-journald[624]: Reserving 333 entries in field hash table. [...] systemd-journald[624]: Reserving 182044 entries in data hash table. [...] systemd-journald[624]: Vacuuming... [...] systemd-journald[624]: Vacuuming done, freed 0B of archived journals from /var/log/journal/b5af1fc93c0a4c7eae36e167138931d5.
