How do you find out why a server crashed at 3 AM when no one was watching? Or prove to a manager that a specific error happened last week? This is the problem of system logging and monitoring. For the EX200 exam, you must know how to configure the two main logging systems on Red Hat Enterprise Linux – journald and rsyslog – so you can collect, store, and search log messages to troubleshoot problems and prove compliance.
Jump to a section
A simple way to picture Configuring Logging and System Monitoring
Have you ever walked into a shared kitchen and found the sink full of cold, greasy water, a half-eaten pizza on the counter, and no idea who left the hob on? You need a way to know what happened, when, and who was responsible. That is exactly the problem system logging solves.
Imagine you and three flatmates share a kitchen. To keep track, you hang a whiteboard on the wall called the 'Kitchen Diary'. Every time someone uses the kettle, burns toast, or spills milk, they must write an entry: the time, what they did, and how serious it was ('Info': made tea. 'Warning': almost set fire to the toaster. 'Error': flooded the floor). This diary is your journald – it captures every event in a structured, searchable format. But the whiteboard has limited space. After a week, old entries get wiped. For anything important – like a gas leak report – you need a permanent record. So, once a day, you copy the most critical entries from the whiteboard into a physical 'Permanent Log Book' stored in a drawer. This copying process is your rsyslog service. It reads the fast, temporary journal and writes selected messages to slow, permanent text files on disk (like /var/log/messages). You can also configure the system: tell the log book to record every single 'Error' from the oven, but ignore 'Info' messages from the microwave. That is the 'configuration' part – deciding what gets saved forever and where it goes. Without this diary and log book, when the landlord asks who broke the dishwasher last Tuesday, you would have no answer. Your server is the same – without logging, when a service crashes at 3 AM, you are clueless.
Every action on a Linux server generates a 'log'. A log is a timestamped record of an event: a user logged in, a service started, a hard drive failed, a firewall blocked a connection. These logs are the server's memory. Without them, you are blind. When something breaks, your only path to fixing it is reading the logs.
Red Hat Enterprise Linux (RHEL) uses two primary tools for logging: journald and rsyslog. They work together, but they are not the same thing.
journald is the newer, default logging system. It is part of systemd, the system and service manager that controls how programs start and run. journald collects logs from the kernel, from services, and from applications. It stores them in a structured, binary format (not plain text you can read with a normal text editor). This binary format is fast and compact. You view journald logs with the command 'journalctl'. For example, typing 'journalctl -xe' shows you the most recent log entries with explanations. journald can also add useful metadata to each log entry, like the exact process ID that generated the message, the user ID, and the priority level (0 for emergencies, 7 for debug). However, journald's log storage is designed to be temporary. By default, logs are stored only in memory (RAM) or in a small file on disk, and they are rotated (deleted) when they get too old or too large. This means if your server crashes and loses power, any logs stored only in memory are gone forever.
This is where rsyslog comes in. rsyslog is the traditional, file-based logging system. Its job is to receive log messages (from journald or directly from applications) and write them to permanent text files on the hard drive. These files live in the /var/log/ directory. Common files include:
/var/log/messages: Most general system messages (non-critical, non-debug).
/var/log/secure: Authentication and security-related messages (logins, sudo usage).
/var/log/maillog: Logs from the mail server.
/var/log/cron: Logs from the cron job scheduler.
The key configuration file for rsyslog is /etc/rsyslog.conf and files inside /etc/rsyslog.d/. In that configuration, you define 'rules' that say: when a message of a certain type arrives, write it to a specific file. The type is defined by two things: a 'facility' (which part of the system generated the message, like 'kern' for kernel or 'auth' for authentication) and a 'severity' (how bad it is, like 'emerg' for emergency or 'info' for information). A typical rule looks like: 'kern.* /var/log/kernel.log' – meaning: take all kernel messages of any severity, and append them to /var/log/kernel.log.
The two systems work together seamlessly. By default, journald is the first stop for all logs. Then rsyslog reads from journald and writes the files to /var/log/. This gives you the best of both worlds: the speed and structure of journald for recent, detailed queries, and the persistence of rsyslog for long-term storage and simple text-file viewing.
Why does this matter for the EX200 exam? You must know how to start, stop, and check the status of both services ('systemctl status rsyslog', 'systemctl status systemd-journald'). You must know how to view logs with journalctl (especially 'journalctl -u [service-name]' to see logs for a specific service, 'journalctl -p err' to see only error-level messages, and 'journalctl --since "1 hour ago"'). You must know how to edit the rsyslog configuration to send specific types of logs to a particular file. And you must understand the difference between persistent and volatile logs, and how to make journald store logs permanently on disk by creating an empty directory '/var/log/journal'.
View recent logs for a specific service
Run 'journalctl -u sshd.service --since "1 hour ago"'. This shows all log entries from the SSHD (secure shell daemon) service from the last hour. It is your first step in troubleshooting why SSH logins are failing.
Filter logs by priority level
Run 'journalctl -p err'. This shows only messages with a priority of 'err' (error) or higher (crit, alert, emerg). It reduces noise when you know something is broken but not which service is causing it.
Follow new log messages in real time
Run 'journalctl -f'. This continuously displays new log entries as they occur, similar to 'tail -f'. Use this when you are about to reproduce a problem (like starting a service) and want to see the log messages appear live.
Configure rsyslog to write a specific log to a new file
Edit /etc/rsyslog.d/custom.conf and add a line like 'if $programname == "crond" then /var/log/cron-events.log'. Save the file. Then run 'systemctl restart rsyslog'. Now, all cron-related messages will be written to a dedicated file, making it easy to audit scheduled tasks.
Test your logging configuration with the logger command
Run 'logger -p local0.info "This is a test from the administrator"'. This sends a message to the logging system. Then run 'journalctl -p info --since "1 minute ago"' (or check your configured rsyslog file) to confirm the message arrived. This proves your configuration works before you rely on it in production.
Make journald logs persistent across reboots
Run 'mkdir -p /var/log/journal' and then 'systemctl restart systemd-journald'. After this, journald will write its journal files to disk instead of only RAM. If the server crashes or reboots, the logs from before the crash are preserved.
Send logs to a remote rsyslog server
Edit /etc/rsyslog.conf and add a line like '*.* @192.168.1.100:514' (UDP) or '*.* @@192.168.1.100:514' (TCP). Then restart rsyslog. This sends all logs to a centralised log server, which is crucial for auditing and monitoring a fleet of servers.
You are the junior system administrator for a small e-commerce company. The CEO calls you, panicked. 'The website went down last night for about 20 minutes. Our monitoring tool said the web server process died, but nobody knows why. We need to know what happened so it does not happen again.'
Your first task is to investigate the logs. You log into the server via SSH. You do not have a GUI – only the command line. Your best friend is 'journalctl'. You start broad:
Type 'journalctl -u httpd.service --since "2024-10-26 22:00:00" --until "2024-10-27 02:00:00"'. This tells journald: 'Show me all log entries for the Apache web server service (httpd), from 10 PM last night to 2 AM this morning.' The output fills the screen. You see a series of normal 'GET' requests during the evening. Then, at 1:47 AM, you see a line: 'httpd[1234]: segfault at 0 ip 00007f... error 4 in libphp.so'. That is the smoking gun. The web server crashed because of a segmentation fault in the PHP library.
But why? You need more context. You run 'journalctl -u httpd.service -p err --since "1 day ago"'. The '-p err' flag filters to show only messages with a priority of 'err' or higher (more severe). This shows you only the crash-related lines, reducing noise. You see that the crash happened immediately after a specific PHP process accessed a corrupted memory block.
Now, the CEO asks for a permanent record for compliance. 'I need a report of all errors on the web server from the last month.' You check /var/log/messages. But you realise that /var/log/messages only contains the last few days of logs because rsyslog's default configuration rotates logs weekly. You need to check the rotated files: /var/log/messages-20241021, /var/log/messages-20241014, etc. You use 'zgrep' or 'less' to search through these compressed, old log files. You find the exact series of events leading up to the crash.
To prevent future blind spots, you decide to configure rsyslog to send all 'httpd' errors to a dedicated file. You edit /etc/rsyslog.d/web.conf and add the line: 'if $programname == "httpd" and $severity <= 3 then /var/log/httpd-error.log'. This tells rsyslog: 'If the program name is exactly httpd, and the severity is err or worse, write the message to this separate file.' You restart rsyslog with 'systemctl restart rsyslog'. Now, future httpd errors will be neatly collected in one file, making future investigations much faster.
Finally, you want to ensure that if the server reboots, you do not lose the crash logs. You check if /var/log/journal exists. It does not, meaning journald is storing logs only in memory (volatile). You create the directory with 'mkdir -p /var/log/journal' and restart journald. Now, journald will write its log data to disk, surviving a reboot. You have just implemented a robust logging strategy.
The EX200 exam will test your hands-on ability to command your way through logging scenarios. You will not be asked to write an essay about logging theory. Instead, you will be given a task: 'Configure rsyslog so that all messages from the kernel facility are logged to the file /var/log/kernel.log.' You must type the correct configuration line into the correct file and restart the service.
Here are the exact concepts they love to test:
The 'logger' command: They will ask you to generate a test log message. You must know: 'logger -p user.err "This is a test error message"'. This sends a message with facility 'user' and priority 'err' to the logging system.
journalctl filters: You must know 'journalctl -u [unit]' for a specific service, 'journalctl -p [priority]' to filter by priority level (emerg, alert, crit, err, warning, notice, info, debug), and 'journalctl -f' to follow new messages in real time (like 'tail -f').
Rsyslog configuration syntax: The classic format 'facility.severity action'. For example, 'authpriv.* /var/log/secure' means all authentication messages go to /var/log/secure. They might test the '=' modifier for exact severity, e.g., 'authpriv.=info' only matches info, not higher severity.
The difference between facility and severity. Facility defines the source of the message (kern, user, mail, auth, authpriv, cron, daemon, syslog, local0 through local7). Severity defines the importance. A common exam trap: they ask 'log all mail messages except debug' – answer: 'mail.none' is not valid; you must use 'mail.info' to exclude debug.
Persistent journald: They will ask 'ensure journald logs survive a reboot'. The answer is to create /var/log/journal. They might ask to set 'Storage=persistent' in /etc/systemd/journald.conf, but the direct, simple method of creating the directory is the typical exam solution.
Log rotation: They rarely ask about logrotate configuration creation, but you must know that old logs are compressed and named with a date suffix.
Traps to watch for:
They might give you a file path that does not exist. You must create the directory first (mkdir -p) before configuring rsyslog to write to it, or rsyslog will silently fail to start.
They may ask you to 'send all logs from the local7 facility to a remote server'. You need to know the remote log syntax: 'local7.* @remote-server-ip:514' (using '@' for UDP, '@@' for TCP).
You must know how to verify your configuration worked. After restarting rsyslog, use 'logger' to send a test message, then check the target file with 'cat' or 'tail' to confirm the message arrived.
They may ask you to 'disable' logging for a specific facility. The correct answer is to set the rule to drop the messages: 'local3.none /var/log/dummy.log' is wrong; you write 'local3.* ~' (the tilde means discard).
journald stores logs in a binary format and is viewed with the 'journalctl' command.
rsyslog writes persistent, human-readable logs to files in the /var/log/ directory based on rules in /etc/rsyslog.conf.
The two systems work together: journald collects all logs, and rsyslog reads from it to write permanent files.
Use 'journalctl -u [service-name]' to view logs for a specific service, and 'journalctl -p err' to filter by priority.
Log a custom test message using 'logger -p user.err "test message"' to verify logging configuration.
To make journald logs survive a reboot, create the directory /var/log/journal and restart the journald service.
The rsyslog configuration syntax is 'facility.severity action', for example 'kern.err /var/log/kernel_errors.log'.
After changing rsyslog configuration, you must restart it with 'systemctl restart rsyslog'.
Logs are a primary troubleshooting tool – when a service fails, always check the logs first.
The priority levels from most to least severe are: emerg, alert, crit, err, warning, notice, info, debug.
These come up on the exam all the time. Here's how to tell them apart.
journald
Stores logs in binary format, not plain text.
Best for recent, fast, highly detailed queries via journalctl.
Default system logger on RHEL; part of systemd.
Logs may be volatile in memory unless /var/log/journal exists.
rsyslog
Stores logs in plain text files in /var/log/.
Best for long-term persistent storage and simple text-based searching.
Traditional syslog daemon; reads from journald or directly from applications.
Logs are written to disk by default; survives reboots.
Facility
Identifies the source of the message (e.g., kern, mail, auth, local0).
For example, 'kern' means the message came from the Linux kernel.
Used in rsyslog rules to filter which messages to log where.
Severity (Priority)
Identifies how important or urgent the message is (e.g., emerg, err, info).
For example, 'err' means something went wrong but the system can still function.
Used with facility to define a rule, e.g., 'kern.err' is kernel errors.
journalctl -u service
Filters log output to show only entries from a specified systemd unit or service.
Helps you focus on a single application's logs (e.g., sshd, httpd).
The syntax is 'journalctl -u sshd.service'.
journalctl -p err
Filters log output to show only entries with a specific priority level or higher.
Helps you focus on only errors and more severe issues across all services.
The syntax is 'journalctl -p err' to show only error, crit, alert, and emerg.
Persistent journald storage
Stores the journal in /var/log/journal on disk.
Logs survive a system reboot or crash.
Requires creating the directory /var/log/journal and restarting journald.
Volatile journald storage
Stores the journal in /run/log/journal in RAM.
Logs are lost when the system reboots or loses power.
This is the default configuration on many RHEL installations.
Mistake
journald and rsyslog are competing systems and you must choose to use only one of them.
Correct
They are designed to work together. journald collects and stores logs in a structured, fast binary format. rsyslog reads from journald and writes persistent, human-readable text files to /var/log/. Using both is the default and best practice.
Beginners see two tools that do a similar thing (handle logs) and assume they are rivals. The exam emphasises they are complementary, but many people study them in isolation and miss the integration.
Mistake
Log files in /var/log/ are always up to date and contain the very latest message immediately.
Correct
Log files written by rsyslog are buffered. There can be a slight delay (a few seconds) between an event happening and the message appearing in the file. Also, rsyslog may not write messages in real time if it is busy.
When running a command like 'tail -f /var/log/messages' and then generating a log, beginners expect instant output. If it does not appear immediately, they think their configuration is broken. This leads to unnecessary troubleshooting.
Mistake
You can read journald logs by looking at a file in /var/log/.
Correct
journald stores logs in a binary journal file (typically /run/log/journal or /var/log/journal) which is not human-readable with cat or less. You must use the 'journalctl' command to view them.
People are used to reading text files in Linux. The binary format feels alien. They might try to 'cat /var/log/journal/...' and get garbage, then assume the system is corrupt.
Mistake
Changing the rsyslog configuration automatically takes effect immediately.
Correct
After editing /etc/rsyslog.conf or files in /etc/rsyslog.d/, you must restart the rsyslog service ('systemctl restart rsyslog') or send it a SIGHUP signal ('kill -1 <pid>') for the changes to load.
In many Linux configuration scenarios, changes to text-based config files are read on the fly. Beginners forget that many daemons (including rsyslog) only read their configuration at startup. They edit the file, test it, and nothing changes, causing confusion.
Mistake
The 'logger' command is only for testing and is not a real tool used in production.
Correct
The 'logger' command is used in production shell scripts and cron jobs to send custom messages into the system log. For example, a backup script might run 'logger Backups completed successfully' so the admin can track it in the logs.
Beginners assume 'logger' is just a toy for practicing for the exam. They do not appreciate that it is a standard utility for injecting messages from scripts, which is a critical concept for real-world automation and monitoring.
Reveal each answer, then mark whether you got it right. Score 60%+ to unlock the next chapter.
journald is the default, modern logging system that stores logs in a fast, structured binary format. rsyslog is the traditional system that reads from journald and writes logs to permanent, human-readable text files in /var/log/. They work together; you use journalctl for journald and check /var/log/ files for rsyslog.
Use the command 'journalctl -u [service-name]'. For Apache, it is typically 'journalctl -u httpd.service'. For Nginx, it is 'journalctl -u nginx.service'. You can add '--since "10 minutes ago"' to narrow down the time frame.
Check if the rsyslog service is running with 'systemctl status rsyslog'. Also, verify that /var/log/messages exists and is not empty. On some minimal installations, rsyslog may not be enabled or configured to log to that file. Also check that the messages are not being sent to a different file by a rule in /etc/rsyslog.d/.
The 'logger' command sends a message with a facility and severity. To send it to a specific file, you first configure rsyslog to catch that facility/severity combination and write it to a file. For example, 'logger -p local7.info "test"' and then in rsyslog.conf have 'local7.info /var/log/my-test.log'.
These are standard syslog priority levels. 'emerg' (0) is a panic condition, the system is unusable. 'alert' (1) means action must be taken immediately. 'crit' (2) is a critical condition. 'err' (3) is an error. 'warning' (4), 'notice' (5), 'info' (6), and 'debug' (7) are lower. The higher the number, the less severe.
Edit /etc/rsyslog.conf and add a line like '*.* @central-server-ip:514' for UDP or '*.* @@central-server-ip:514' for TCP. Replace 'central-server-ip' with the IP address of your logging server. Then restart rsyslog. The remote server must be configured to receive logs on that port.
logrotate is a system utility that manages log files: it can rotate them (archive the current log and start a new one), compress old ones, and delete very old ones to save disk space. Its configuration files are in /etc/logrotate.conf and /etc/logrotate.d/. You typically do not need to create logrotate configs for the EX200 exam, but you should know that they are responsible for the rotated log files you see (like messages-20241021).
You've finished Configuring Logging and System Monitoring. Continue through the EX200 study guide to build a complete picture of the exam.
Done with this chapter?