Skip to content

Understanding logs

Logs are the memory of a server: what services did, which errors occurred, who logged in. Linux has two places for them: the systemd journal and classic text files in /var/log.

Journal or text file?

Source Where Read with
systemd journal Binary files in /var/log/journal/ journalctl
rsyslog Text files like /var/log/syslog or /var/log/messages less, tail, grep
Applications Own files, e.g. /var/log/nginx/, /var/log/postgresql/ less, tail, grep

Almost all system services log to the journal. Whether rsyslog also writes text files depends on the distribution:

Debian Ubuntu RHEL Arch
rsyslog installed no yes yes no
General system log journal only /var/log/syslog /var/log/messages journal only
Logins and sudo journal only /var/log/auth.log /var/log/secure journal only
Package manager /var/log/apt/history.log, /var/log/dpkg.log /var/log/apt/history.log, /var/log/dpkg.log /var/log/dnf.log /var/log/pacman.log
Kernel journal, dmesg /var/log/kern.log, dmesg journal, dmesg journal, dmesg

Log levels

Every message has a priority. journalctl -p and many tools filter by it:

Level Name Meaning
0 emerg The system is unusable
1 alert Action must be taken immediately
2 crit Critical condition, e.g. hardware failure
3 err Error
4 warning Warning
5 notice Normal, but significant
6 info Information
7 debug Debug messages

journalctl -p warning shows levels 0 to 4, everything from warnings upwards.

Log rotation

Log files grow endlessly. logrotate runs once a day and, according to its rules in /etc/logrotate.d/, renames the current file (access.log becomes access.log.1), compresses older ones (access.log.2.gz) and deletes the oldest. The journal limits its size itself, see SystemMaxUse= in /etc/systemd/journald.conf.

Failed logins

Without /var/log/auth.log and without lastb (no longer available on Debian 13), the journal shows failed SSH logins:

$ journalctl -u ssh --since today | grep "Failed password"
Oct 26 09:58:02 tuxdex-srv sshd-session[2187]: Failed password for nox from 192.0.2.45 port 51720 ssh2
Oct 26 10:31:02 tuxdex-srv sshd-session[3301]: Failed password for invalid user admin from 198.51.100.66 port 40218 ssh2
Oct 26 10:31:05 tuxdex-srv sshd-session[3301]: Failed password for invalid user admin from 198.51.100.66 port 40218 ssh2

invalid user means: the account doesn't exist, typical for automated attacks. Tools like fail2ban-client and faillock react to such attempts.

Look at the time first

When something went wrong, note the time and look at all logs around it: journalctl --since "10:25" --until "10:35" shows every service in that window.

Logs contain personal data

IP addresses, user names and URLs in logs are personal data. Keep them only as long as you need them, and protect access: on Debian and Ubuntu, only the group adm may read most of /var/log.