Linux Syslog and Journal Analysis

Syslog is a standard protocol (RFC 5424) for message logging on Unix-like systems. It separates log generation (by processes) from log storage (by a daemon like rsyslog or syslog-ng). Syslog messages have a facility (e.g., auth, cron, daemon, kern) and a severity level (0-7, with 0=emergency, 7=debug). The syslog daemon reads messages from /dev/log, filters them based on rules in /etc/rsyslog.conf or /etc/syslog-ng/syslog-ng.conf, and writes them to files (typically in /var/log/).

Journald is part of systemd, the init system used by most modern Linux distributions. It collects logs from the kernel, systemd services, and applications via structured binary logs stored in /var/log/journal/ (persistent) or /run/log/journal/ (volatile). Journald captures not only the message text but also metadata like PID, UID, GID, SELinux context, kernel subsystem, and more. It supports indexing, compression, and integrity checking (via forward-secure sealing).

Syslog Flow: 1. An application (e.g., sshd) generates a log message using syslog() or a library like libc. 2. The message is sent to the syslog daemon (rsyslogd) via a Unix socket (/dev/log) or UDP/TCP port 514. 3. rsyslogd checks its configuration rules (e.g., "if facility=mail and severity=info, write to /var/log/maillog"). 4. The daemon writes the message to the appropriate file, optionally adding a timestamp and hostname.

Journald Flow: 1. An application logs via the systemd journal API (sd-journal), or syslog messages are forwarded to journald via a socket. 2. journald stores the message in a binary journal file with metadata fields (e.g., _PID, _UID, _COMM, SYSLOG_FACILITY). 3. Logs can be queried using journalctl, which reads the journal files and outputs filtered results. 4. Journald can optionally forward logs to traditional syslog daemons for centralization.

Attackers often clear or manipulate logs to cover tracks. Common techniques:

Defenders use syslog and journald for: - Forensic Analysis: Querying journal logs for specific timeframes, users, or commands. Example: journalctl _UID=0 --since "2024-01-01 00:00:00" --until "2024-01-02 00:00:00" shows all root actions. - Centralized Logging: Forwarding syslog to a SIEM (e.g., Splunk, ELK) via TCP/514 or TLS/6514. - Integrity Verification: Journald's forward-secure sealing allows detecting if logs were modified after creation. - Alerting: Using rsyslog rules to trigger email alerts on specific patterns (e.g., multiple failed logins).

Syslog configuration snippet (/etc/rsyslog.conf):

# Log all kernel messages to /var/log/kern.log
kern.* /var/log/kern.log

# Log authentication messages to /var/log/auth.log
auth,authpriv.* /var/log/auth.log

# Send all messages to remote log server via TCP
*.* @@logserver.example.com:514

Journalctl query examples:

# Show all logs for the ssh service
journalctl -u ssh.service

# Show logs from last boot
journalctl -b

# Show logs with priority 3 (err) or higher
journalctl -p err

# Show logs for a specific executable
journalctl /usr/sbin/sshd

# Follow new logs in real time (like tail -f)
journalctl -f

Checking journal integrity:

# Verify journal integrity (requires --verify)
journalctl --verify

Using logger to test syslog:

logger -p authpriv.info "Test message from Courseiva"

Viewing syslog files:

# Common log files
/var/log/messages (general)
/var/log/secure (authentication)
/var/log/maillog (mail server)
/var/log/cron (cron jobs)
/var/log/boot.log (boot messages)

Scenario 1: SSH Brute-Force Detection A SOC analyst notices a spike in failed SSH logins on a Linux server. Using journalctl: journalctl -u ssh.service -p info --since "1 hour ago" | grep "Failed password". They see hundreds of entries from IP 10.0.0.5. The correct response: block the IP via firewall (iptables -A INPUT -s 10.0.0.5 -j DROP) and check for successful logins from that IP: journalctl -u ssh.service | grep "Accepted.*10.0.0.5". Common mistake: only checking auth.log and missing journald's richer metadata (e.g., SSH protocol version, key fingerprints).

Scenario 2: Privilege Escalation via sudo After a breach, an investigator uses journalctl to find all sudo commands: journalctl -u sudo.service --since "2024-03-10". They see a user 'tempadmin' ran sudo -i and then sudo visudo. The investigator checks if 'tempadmin' was added to sudoers: grep tempadmin /etc/sudoers. They also verify the user's creation time: journalctl -u useradd.service. The correct response: disable the account and review all commands executed with sudo. Common mistake: assuming sudo logs are only in /var/log/secure; on systemd systems, journald captures more detail including the working directory.

Scenario 3: Malware Persistence via Cron A Linux server exhibits unusual outbound connections. The analyst checks cron logs: journalctl -u cron.service. They find a root cron job running /tmp/.malware.sh every minute. The analyst also checks the syslog file /var/log/cron for redundancy. Correct response: remove the cron job, delete the malware, and check for other persistence mechanisms (systemd services, .bashrc). Common mistake: only looking at syslog and missing journald's metadata (e.g., the exact command line arguments).