System troubleshooting and performance tuning are the methods used to find why a computer is running slowly or breaking, and then to fix it so it runs optimally. For the XK0-006 exam, you need to understand the systematic process of diagnosing problems and the common tools used to measure system health, because employers expect Linux administrators to keep systems stable and responsive.
Jump to a section
A simple way to picture System Troubleshooting and Performance Tuning
Your server has a normal operating temperature of 72°C. If it reaches 85°C, the CPU (the brain of the computer) will automatically slow down to protect itself, just like a overheating oven automatically turns off its heating elements. If it hits 100°C, the server will shut down completely to prevent permanent damage. This is called thermal throttling - the system is intentionally performing worse to avoid breaking itself.
Now imagine you are a pastry chef. Every morning, you check the temperature of your main oven with a laser thermometer. If it reads 72°C, you know everything is normal. If it reads 80°C, you check if the door seal is worn, or if someone left the oven on 'high' instead of 'bake'. You have a set of standard checks: you look at the exhaust fan speed, you feel the air vents, you listen for unusual sounds. This is exactly what an IT professional does with monitoring tools - they measure temperature (CPU usage), fan speed (disk I/O), and airflow (network throughput). When one metric is too high, they methodically check the 'kitchen' until they find the root cause. Just like you would not replace the whole oven because the door seal is loose, a troubleshooter does not replace a whole server because a single misconfigured log file is filling up the hard drive.
System troubleshooting is a structured way of solving a problem with a computer system. It is not guessing or randomly pressing buttons. It follows a logical order: identify the problem, establish a theory of probable cause, test that theory, establish a plan of action, implement the fix, verify full system functionality, and document the findings. This process is called the 'troubleshooting methodology'.
Performance tuning is the practice of making a system run faster or handle more work without adding more hardware. This involves adjusting software settings, cleaning up unnecessary files, or changing how the operating system allocates resources like memory and CPU time.
Let us examine the key components that affect a Linux system's performance:
CPU (Central Processing Unit): The 'brain' of the computer that executes instructions. If the CPU usage is constantly at 100%, processes queue up and the system feels slow. Tools like 'top' or 'htop' show real-time CPU usage per process.
Memory (RAM): Random Access Memory stores data the CPU needs to access quickly. When RAM is full, the system uses a part of the hard drive called 'swap space' as a substitute. Swap is much slower than RAM, so heavy swapping causes performance to drop significantly.
Disk I/O (Input/Output): The speed at which the hard drive reads and writes data. If a process is writing huge log files, the disk can become the bottleneck. The 'iostat' command shows disk activity and can help find the problem.
Network: The connection between this server and other systems. High network latency or packet loss can make services feel slow. Tools like 'ping', 'traceroute', and 'netstat' help diagnose network issues.
Common performance issues include memory leaks, where a program uses more and more RAM over time and never releases it; runaway processes, where a single program consumes all CPU cycles; and disk space exhaustion, where the system cannot write new files because the drive is 100% full.
Logs are critical for troubleshooting. Linux stores logs in the /var/log directory. The most important log file is 'messages' or 'syslog', which records system events like hardware errors, service crashes, and login attempts. The 'dmesg' command shows kernel ring buffer messages, which are useful for diagnosing hardware problems like failing disk drives or faulty RAM.
Performance tuning involves monitoring baseline metrics first - what is 'normal' for this system? If a web server normally uses 30% CPU and 4GB of RAM, but today it uses 80% CPU and 6GB of RAM, something changed. You look at recent configuration changes, new software installations, or increased user traffic. Tuning can involve changing 'kernel parameters' through /etc/sysctl.conf - for example, adjusting 'vm.swappiness' controls how aggressively the system uses swap memory. Lower swappiness means the system keeps more data in RAM, which can improve performance for some workloads.
Another key area is managing 'systemd' services. You can check the status of a service with 'systemctl status servicename', and if a service is misbehaving, you restart it or look at its journal with 'journalctl -u servicename'. The 'journalctl' command shows logs from systemd's journal, which is much easier to search than plain text files.
Finally, 'load average' is a metric that represents the number of processes waiting to use the CPU. The three numbers shown by 'uptime' or 'top' represent the 1-minute, 5-minute, and 15-minute averages. If the 1-minute average is much higher than the 15-minute average, the load is increasing. A load average of 4.0 on a 4-core CPU means the system is fully utilised. Higher than that means processes are waiting, and performance will degrade.
Identify the Problem
Gather information from users or monitoring alerts. Ask specific questions: 'What exactly is slow?', 'When did it start?', 'Has anything changed since then?'. Determine the scope: is it one service, one user, or the whole system? This step defines the exact symptom.
Establish a Theory of Probable Cause
Use your knowledge of common issues to form a hypothesis. For example, if the web server is slow during peak hours, the theory could be insufficient memory. Look at recent system changes, configuration updates, or new software installs. Start with the most likely cause.
Test the Theory
Run diagnostic commands to confirm or disprove your theory. Check CPU with 'top', memory with 'free -m', disk with 'df -h', and logs with 'journalctl'. If the theory is incorrect, revise it and test again. This step eliminates guesses and narrows down the real problem.
Establish a Plan of Action
Decide exactly what steps to take to fix the problem. Consider the impact: will the fix require downtime? Can it be done during business hours? Write down the plan, including backup steps in case the initial fix fails. For example, plan to increase swap space if memory is the bottleneck.
Implement the Fix
Execute the planned actions. If possible, make a backup of configuration files before changing them. Apply changes step by step, verifying each action does not cause new issues. For a memory issue, you might add a swap file using 'dd if=/dev/zero of=/swapfile bs=1G count=2' and 'mkswap /swapfile'.
Verify Full System Functionality
Confirm that the original symptom is resolved. Re-run the commands from the testing step to see if metrics are back to normal. Also check related services: if you fixed a web server, verify that the database it connects to still works. Ask the user who reported the problem to test the system.
Document Findings
Write down the symptom, root cause, fix applied, and any configuration changes made. This documentation helps other administrators and prevents the same issue from wasting time in the future. Include commands used and any error messages seen.
An IT professional, often called a 'sysadmin', uses these skills daily. Consider a scenario: a medium-sized e-commerce company runs its website on a single Linux server. At 10am, the sales team reports that the website is loading very slowly, and some customers are getting error messages. The sysadmin follows the troubleshooting methodology. - Step 1: Identify the problem. The sysadmin accesses the server via SSH (secure shell) and runs 'uptime' to see the load average: 12.5, 8.3, 4.1. The 1-minute average is critically high compared to the 15-minute average, confirming a recent spike. - Step 2: Establish a theory. The sysadmin checks 'top' and sees that the Apache web server process (httpd) is using 90% of CPU and 8GB of RAM. The theory is that the web server is overwhelmed by traffic or a misconfigured script. - Step 3: Test the theory. The sysadmin checks disk space with 'df -h' (disk free) and sees the root partition is 98% full. They run 'du -sh /var/log' and discover the Apache access log is 50GB - it was never rotated. - Step 4: Establish a plan of action. The plan is to archive the log file, configure log rotation manually, and restart the web server. - Step 5: Implement the fix. The sysadmin compresses the log with 'gzip /var/log/httpd/access_log', creates a new blank log file, and sets up 'logrotate' configuration to rotate logs daily. - Step 6: Verify full functionality. They run 'systemctl restart httpd', then check 'top' again. The load average drops to 2.1. They visit the website from their browser, and it loads normally. - Step 7: Document findings. The sysadmin writes a ticket in their IT service management system explaining the cause and the fix, so future incidents can be resolved faster.
A performance tuning scenario: The same company expects a holiday sales surge. The sysadmin pre-emptively adjusts kernel parameters to handle more concurrent connections. They edit /etc/sysctl.conf and add 'net.ipv4.tcp_max_syn_backlog = 4096' to increase the number of connection requests the server can queue, and 'net.core.somaxconn = 1024' to increase the socket listen backlog. After applying with 'sysctl -p', the server can handle the increased traffic without dropping connections. They also set up monitoring with 'prometheus' and 'grafana' to watch real-time metrics. If CPU usage exceeds 80% for more than 5 minutes, an alert is emailed to the team. This proactive tuning prevents outages before they happen.
The XK0-006 exam tests your ability to recognise troubleshooting symptoms and choose the correct diagnosis tool or fix. You will see scenario-based multiple-choice questions, performance-based items where you execute commands in a simulated terminal, and matching questions linking tools to their purpose.
Key concepts the exam loves to test:
Troubleshooting methodology: Must know the seven steps in order. A question may describe a step out of sequence and ask you to identify the missing step. Memorise: identify, establish theory, test, plan, implement, verify, document.
Log file locations: Know that /var/log/messages or /var/log/syslog contain general system messages, /var/log/secure or /var/log/auth.log contain authentication attempts, and /var/log/dmesg contains hardware messages. The exam often shows a log entry and asks where it came from or what tool generated it.
Process management: The 'ps' command (process status) with options like 'ps aux' shows all processes. 'kill' sends signals to processes - SIGTERM (graceful shutdown) vs SIGKILL (forced termination) is a common trap. Questions may ask which signal allows cleanup before termination.
Memory and CPU metrics: 'free' shows RAM and swap usage. 'vmstat' shows virtual memory stats. 'top' shows real-time processes. Be prepared to interpret the output - for example, if swap usage (si/so) is non-zero, the system is swapping, which indicates insufficient RAM.
Disk troubleshooting: 'df' shows disk usage by partition. 'du' shows usage by directory. 'iostat' shows per-disk I/O. 'fsck' (file system check) is used to repair a corrupted file system, but it requires the partition to be unmounted first. A common trick question involves trying to run 'fsck' on a mounted partition - the correct answer is 'unmount first'.
Network troubleshooting: 'ping' checks connectivity. 'traceroute' shows the path packets take. 'netstat' or 'ss' display open sockets and listening ports. 'tcpdump' captures packets for deep analysis. The exam may ask which tool shows which network connections are established.
Systemd journal: 'journalctl' is the primary way to view logs on modern systems. Know that 'journalctl -u service-name' shows logs for a specific service, 'journalctl -p err' shows only error-level messages, and 'journalctl --since '1 hour ago'' narrows the time window.
Performance bottlenecks: Questions often describe a symptom (e.g., system is slow, disk activity light, CPU idle, swap used heavily) and ask you to identify the bottleneck - in this case, insufficient RAM. Another common pattern: high load average but low CPU usage indicates many processes waiting for I/O, typically disk.
Tuning parameters: 'sysctl' applies kernel parameter changes. Know that changes are temporary unless written to /etc/sysctl.conf. The exam may ask which file makes kernel changes permanent.
Common traps:
Confusing 'swap' with 'swap partition' versus 'swap file'. Both work, but the exam expects you to know how to add swap files using 'dd' (disk duplicator) and 'mkswap'.
Assuming a service restart is always needed. Some configuration changes take effect immediately (like 'sysctl -w'), others require a restart. The exam tests this distinction.
Mixing up 'dmesg' (kernel hardware messages) with 'journalctl' (user-space logs). If the question involves a hard drive error, dmesg is the correct first place to look.
Memorise the command set: df, du, ps, top, free, vmstat, iostat, netstat, ss, ping, traceroute, tcpdump, journalctl, dmesg, sysctl, kill, systemctl. Know what each measures and its most common option.
The seven-step troubleshooting methodology must be followed in order: identify, establish theory, test, plan, implement, verify, document.
The 'journalctl -u servicename' command is the fastest way to see logs for a specific systemd service.
Swap space usage (si/so in 'vmstat') being non-zero indicates that the system has run out of physical RAM and is using slower disk memory.
The 'df' command shows disk usage by partition; the 'du' command shows usage by directory; combined they help find what fills the disk.
A high load average with low CPU usage typically means the bottleneck is disk I/O, not the processor.
Kernel parameter changes made with 'sysctl -w' are temporary; add them to /etc/sysctl.conf to make them permanent across reboots.
Use SIGTERM (signal 15) before SIGKILL (signal 9) to stop a process gracefully and avoid data corruption.
The 'dmesg' command is the first place to check for hardware-related errors like failing drives or faulty memory modules.
Performance tuning is about establishing a baseline of normal metrics so that you can detect when something changes.
System logs in /var/log are rotated automatically by 'logrotate' to prevent disks from filling up; configuration is in /etc/logrotate.conf.
These come up on the exam all the time. Here's how to tell them apart.
top
Pre-installed on most Linux distributions
Minimal interface without colours
Sorting done via keyboard commands like 'M' for memory
htop
Often needs to be installed via package manager
Colour-coded interface with mouse support
Easier to scroll and click to kill processes
SIGTERM (signal 15)
Requests a process to terminate gracefully
Allows the process to clean up and save data
Can be ignored or handled by the process
SIGKILL (signal 9)
Forces immediate termination of a process
Does not allow any cleanup or data saving
Cannot be ignored; always kills the process
df command
Shows disk usage per mounted partition
Reports total space, used space, and available space per partition
Useful for checking if a disk is full
du command
Shows disk usage per directory or file
Requires a specific path to scan
Useful for identifying which folder is taking up space
/var/log/messages
Contains general system logs from running services
Written by the syslog daemon
Includes authentication attempts and cron job output
dmesg command
Contains kernel ring buffer messages
Shows hardware-related events like drive errors and hardware detection
Does not show user-space logs like SSH login attempts
Mistake
If the load average is high, the CPU must be overworked.
Correct
A high load average can also mean processes are waiting for disk I/O or memory resources, not just CPU. Always check multiple metrics (CPU, memory, disk) together.
Beginners see 'load' and think CPU, but the load counts processes waiting for any resource, including a busy hard drive or full swap.
Mistake
You should reboot the server first when it is running slowly.
Correct
Troubleshooting should begin with checking system metrics and logs to identify the root cause. Rebooting might fix the current symptom but prevents you from finding the underlying issue, and it causes downtime for users.
In home computing, rebooting often fixes problems because user-mode software is restarted. In a production server, the goal is uptime, not quick fixes.
Mistake
The 'kill -9' command is the safest way to stop a misbehaving program.
Correct
SIGKILL (-9) does not allow the program to clean up resources or save data. The preferred signal is SIGTERM (-15), which requests a graceful shutdown. Use SIGKILL only if the process ignores SIGTERM.
Movies and online forums often present 'kill -9' as a universal solution, but it can cause data corruption or orphaned files in real databases.
Mistake
If a disk is full, you can simply delete any log file to fix it.
Correct
Deleting a log file that is currently open by a process does not free up space until the process releases the file handle. You must truncate the file (e.g., '> /var/log/messages') or stop the process first.
On Linux, deleting an open file removes the directory entry but the file remains on disk until the process closes it. Beginners learn this the hard way when space is not recovered.
Mistake
Running 'fsck' on a mounted partition is safe if you do not write to it.
Correct
Never run 'fsck' on a mounted partition. It can cause severe file system corruption because fsck expects exclusive write access. Always unmount the partition first or boot into rescue mode.
The myth persists because some read-only checks appear to work, but any repair attempt on a mounted file system risks data loss.
Reveal each answer, then mark whether you got it right. Score 60%+ to unlock the next chapter.
Both show running processes and resource usage, but 'htop' provides a more user-friendly interface with colour coding and mouse support. 'top' is pre-installed on nearly all Linux distributions, while 'htop' may need to be installed via the package manager.
Run the 'free -m' command. If the 'swap used' line shows a non-zero number, the system is using swap. For more detail, run 'vmstat 1 5' and look at the 'si' (swap in) and 'so' (swap out) columns - any non-zero values indicate active swapping.
If the log file was open by a running process (like a web server), deleting it removes the directory entry but the file remains until the process closes it. Use 'truncate -s 0 /path/to/logfile' to empty the file without restarting the process, or restart the service to release the file handle.
The load average is the number of processes that are either running or waiting to run, averaged over 1, 5, and 15 minutes. If the value exceeds the number of CPU cores, the system is overloaded. For example, a load of 4.0 on a 4-core CPU means the CPU is fully utilised.
Use the 'ps aux --sort=-%mem' command. This lists all processes sorted by memory usage from highest to lowest. The process with the highest %MEM value is the largest memory consumer. You can also use 'top' and press 'M' to sort by memory.
The 'kill' command terminates a process by its process ID (PID), while 'killall' terminates all processes matching a given name. For example, 'kill 1234' kills the process with PID 1234; 'killall httpd' kills all Apache web server processes.
No, you cannot run 'fsck' on a mounted partition, including root. 'fsck' requires exclusive write access to the file system. To repair the root partition, you must boot into rescue mode or a live USB environment where the root partition is not mounted.
You've finished System Troubleshooting and Performance Tuning. Continue through the XK0-006 study guide to build a complete picture of the exam.
Done with this chapter?