Uploaded image for project: 'RHEL'
  1. RHEL
  2. RHEL-99932

RHEL8+ rsyslog do not record 'signal 15' in /var/log/messages once system normal shutdown/reboot

Linking RHIVOS CVEs to...Migration: Automation ...RHELPRIO AssignedTeam ...SWIFT: POC ConversionSync from "Extern...XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Can't Do
    • Icon: Normal Normal
    • None
    • rhel-8.10, rhel-9.6
    • rsyslog
    • No
    • None
    • rhel-security-special-projects
    • None
    • False
    • False
    • Hide

      None

      Show
      None
    • None
    • None
    • None
    • None
    • Unspecified
    • Unspecified
    • Unspecified
    • x86_64
    • None

      • On RHEL6 and RHEL7 systems, if the system is directed shutdown/reboot by user/applicaiton, the messages log often records 'Stoped xxxx' or 'signal 15' information. For example, see the following messages:
        ~~~
        rhel7 systemd: Stopped Dump dmesg to /var/log/dmesg.
        rhel7 systemd: Stopped target RPC Port Mapper.
        rhel7 systemd: Stopped target Timers.
        rhel7 systemd: Stopping RPC bind service...
        rhel7 systemd: Stopped target Multi-User System.   <--------
        rhel7 systemd: Stopping OpenSSH server daemon...
        rhel7 systemd: Stopped Resets System Activity Logs.
        rhel7 systemd: Stopping Command Scheduler...
        rhel7 systemd: Unmounting RPC Pipe File System...
        rhel7 systemd: Stopping Authorization Manager...
        rhel7 systemd: Stopped Daily Cleanup of Temporary Directories.
        rhel7 systemd: Stopping LSB: Starts the Spacewalk Daemon... <--------
        rhel7 systemd: Removed slice system-selinux\x2dpolicy\x2dmigrate\x2dlocal\x2dchanges.slice.
        rhel7 systemd: Stopping Enable periodic update of entitlement certificates....
        rhel7 systemd: Stopping irqbalance daemon...  <--------
        ~~~

      Through the above log, customers easy judge that that OS is directed reboot/shutdown. This function helps customers quickly determine the reasons for the system reboot. See the following KB
      ~~~
      Why does Red Hat Enterprise Linux Server reboot with "signal 15" messages in /var/log/messages?
      https://access.redhat.com/solutions/31411
      ~~~

      • However, On RHEL8 & RHEL9 system, if the system is directed shutdown/reboot, the messages log will not exist 'Stoped xxxx' or 'signal 15' information any more, For example, see the following messages:
        ~~~
        RHEL8 systemd[1]: Starting system activity accounting tool...
        RHEL8 systemd[1]: sysstat-collect.service: Succeeded.
        RHEL8 systemd[1]: Started system activity accounting tool. <———— initial reboot command
        RHEL8  kernel: Command line: BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-193.13.2.el8_2.x86_64
        ~~~

      According to the following explanation from Attila++ in RHEL-5198,  the RHEL8+(default configuration) can no longer record any 'Stoped xxxx' or 'signal 15' information in messages.
      ~~~
      on RHEL8+, the syslog daemon behaves like a normal journal client, and reads messages from the journal files, similarly to journalct.
      With this, messages do not have to be read immediately, which allows a logging daemon which is only started late in boot to access all messages since the start of the system.
      Because the default value of ForwardToSyslog= has changed between the two major versions of RHEL, in rsyslog we need to use the journal API (similarly to journalctl) to read the journal. That's why you might experience 'message loss' in rsyslog when shutting down a system. However, this is not message loss, rather it's expected to work like this. If you want to store journal logs during shutdown/reboot, please set the Storage="persistent" option.
      ~~~

      Therefore, on RHEL8+ & RHEL9 system, if the system is directed shutdown/reboot, then customers cannot confirm it intuitively from the messages any more, which has caused trouble for many customers. More and more customers strongly require that the messages on RHEL8+ & RHEL9 can record the log the 'Stoped xxxx' or 'signal 15' which similar with RHEL7。
      If we optimize our logging system,It will improve the cost of customer surveying the root cause of machine reboot.

      BTW,as a workaround on RHEL8+ and RHEL9, we can configure journal to store the log in disk, so that the messages log will record specific 'signal 15' or 'shutdown' or 'Stoped xxxx'. But this workaround is not the default configuration of the system journald.conf. it is impossible to request all customer to change the default configuration。
      ~~~
      mkdir -p /var/log/journal
      systemctl restart systemd-journald.service
      ~~~

      Although I understand that It is the default behavior of RHEL8+ and RHEL9, If we can make some improvements that will greatly improve the customer experience.

              rh-ee-alakatos Attila Lakatos
              yruo1@redhat.com Yu Ruo
              Attila Lakatos Attila Lakatos
              SSG Security QE SSG Security QE
              Votes:
              1 Vote for this issue
              Watchers:
              7 Start watching this issue

                Created:
                Updated:
                Resolved: