How do I display log messages from previous boots under CentOS 7?

  • Executing journalctl under a CentOS 7 system just prints messages generated after the last boot.

    The command

    # journalctl --boot=-1
    

    prints

    Failed to look up boot -1: Cannot assign requested address
    

    and exits with status 1.

    Comparing it to a current Fedora system I notice that the CentOS 7 does not have /var/log/journal (and journalctl does not provide --list-boots).

    Thus my question how to display log messages which were written before the last boot date.

    Or, perhaps this functionality has to be enabled on CentOS 7?

    (The journalctl man page lists 'systemd 208' as version number.)

  • tl;dr

    On CentOS 7, you have to enable the persistent storage of log messages:

    # mkdir /var/log/journal
    # systemd-tmpfiles --create --prefix /var/log/journal
    # systemctl restart systemd-journald
    

    Otherwise, the journal log messages are not retained between boots.

    Details

    Whether journald retains log messages from previous boots is configured via /etc/systemd/journald.conf. The default setting under CentOS 7 is:

    [Journal]
    Storage=auto
    

    Where the journald.conf man page explains auto as:

    One of "volatile", "persistent", "auto" and "none". If "volatile", journal log data will be stored only in memory, i.e. below the /run/log/journal hierarchy (which is created if needed). If "persistent", data will be stored preferably on disk, i.e. below the /var/log/journal hierarchy (which is created if needed), with a fallback to /run/log/journal (which is created if needed), during early boot and if the disk is not writable. "auto" is similar to "persistent" but the directory /var/log/journal is not created if needed, so that its existence controls where log data goes.

    (emphasize mine)

    The systemd-journald.service man page thus states that:

    By default, the journal stores log data in /run/log/journal/. Since /run/ is volatile, log data is lost at reboot. To make the data persistent, it is sufficient to create /var/log/journal/ where systemd-journald will then store the data.

    Apparently, the default was changed in Fedora 19 (to persitent storage) and since CentOS 7 is derived from Fedora 18 - it is still non-persisent there, by default. Persistency is implemented by default outside of journald via /var/log/messages and the rotated versions /var/log/messages-YYYYMMDD which are written by rsyslogd (which runs by default and gets its input from journald).

    Thus, to enable persistent logging with journald under RHEL/CentOS 7 one has to

    # mkdir /var/log/journal
    

    and then fix permissions and restart journald, e.g. via

    # systemd-tmpfiles --create --prefix /var/log/journal
    # systemctl restart systemd-journald
    

    A `systemctl restart systemd-journald` should do it. So no reboot required.

    @xx4h, updated the answer

    In debian this is documented at `/usr/share/doc/systemd/README.Debian`: `install -d -g systemd-journal /var/log/journal`.

    @pevik, I've looked at a CentOS 7 system (where I just used mkdir) and the current permissions are `drwxr-sr-x. 3 root systemd-journal` - perhaps journald fixes the permissions/ownership during initialization.

    @maxschlepzig: Debian docs says: _systemd will add an ACL for read permissions for users in the "adm" group_.

    After you enabled the persistent storage of log messages, you can use `journalctl -b -N` where `N` is the Number of any previous boot, eg `-1`, `-2` and so on. I use this to easily compare log between 2 boots after update for example (`journalctl -b >$(hostname)-journalctl-0 && journalctl -b -1 >$(hostname)-journalctl-1 && vimdiff -journalctl-*`

    Learned a neat trick: if you send a `USR1` signal instead of restarting, you don't lose the current journald contents. `killall -USR1 systemd-journald`

    Same on Debian 9 and 10. /var/log/journal is not created automatically and you need to follow these instructions.

    A simplification on Centos 7: you don't need to run `systemd-tmpfiles --create --prefix /var/log/journal`; simply creating the `/var/log/journal` directory and then setting the group correctly on the directory with `chgrp systemd-journal /var/log/journal/`, before restarting the service, is sufficient.

    @paulmdavies, I don't think so. I've checked a CentOS 7 system and `systemd-tmpfiles` also sets some ACLs on that directory.

    @maxschlepzig well, it definitely worked for me. Maybe the ACLs are set by `journald` intialization, like you suggested in a previous comment?

    @paulmdavies cf. `grep journal /usr/lib/tmpfiles.d/systemd.conf` - also, the `ls` output in my previous comment shows that the resulting directory in that experiment didn't have any ACLs set. Thus, just doing a `mkdir` and possibly setting the group may work in some scenarios/for some journald features, but it's certainly not sufficient. Note that `systemd-tmpfiles` also runs during system boot.

    Pairs well with hibernate bugs and frothing rage.

License under CC-BY-SA with attribution


Content dated before 6/26/2020 9:53 AM