# Linux Host Analysis

> Chapter 6 of the Courseiva CISCO-CYBEROPS-ASSOCIATE curriculum — https://courseiva.com/learn/cisco-cyberops-associate/linux-host-analysis

**Official objective:** 3.2 — Explain how to analyze Linux operating system artifacts

## Introduction

Linux Host Analysis is the systematic examination of a Linux computer to determine its security state, identify signs of an attack, and gather evidence of malicious activity. For the 200-201 exam, you must understand how to inspect the core components of a Linux system — its processes, users, logs, file system, and network connections — because attackers often target Linux servers and leave forensic artefacts behind that only a trained analyst can find.

## The Flat Inspection Analogy

When you first walk into a flat you are considering renting, you don't just glance at the sofa and say 'good enough'. You open the fridge, you check under the sink for leaks, you flush the toilet, you test the light switches, and you look at the corners of the rooms for mould. You do this because the surface appearance can hide serious problems that will only cause you grief later.

Linux Host Analysis is exactly that same process, but for a computer that uses the Linux operating system. The first thing that happens is an alert fires or an investigation starts (the flat viewing begins). That leads directly to the analyst needing to check every critical part of the Linux system — the running processes (think of these as the rooms that are currently occupied), the authorised users (the tenants on the lease), the login history (the guestbook), the installed software packages (the furniture inventory), and the configuration files (the building plans).

Just like a flat can look perfect but have a secret back door or a window that never locks, a Linux host can appear healthy but hide an attacker's foothold. The analyst inspects logs (the CCTV footage of the flat), checks crontab entries (the automated routines like a timed sprinkler system), and examines network connections (the pipes and cables running in and out). With every file you open and every command you type, you are building a clear picture of whether the system is safe or has been compromised. You are the careful flat inspector, and the Linux host is the flat you must not let any detail slip past.

## Core explanation

Linux Host Analysis is the process of examining a Linux operating system to find evidence of unauthorised access, malware, data theft, or other security incidents. It is a core skill for a CyberOps Associate because Linux powers most of the world's servers, cloud infrastructure, and security appliances. When something goes wrong on a Linux machine, the first thing an analyst does is look at the host itself — not the network traffic or the firewall logs — because the host contains the most direct evidence of what happened.

Let's start with the fundamental building blocks. A Linux operating system is a software layer that manages a computer's hardware and provides services for applications to run. There are many versions, called distributions (or 'distros'), such as Ubuntu, CentOS, Debian, and Red Hat Enterprise Linux. The 200-201 exam focuses on the core concepts that are common to all distributions.

The first thing an analyst checks is the running processes. A process is a program that is currently executing on the system. Imagine your computer as a busy kitchen. Each process is a chef working on a specific dish. You use the 'ps' command (short for 'process status') to list them. The 'top' command shows them in real time, sorted by how much CPU or memory they are using — like watching which chef is working hardest. Attackers often run malicious processes disguised with innocent names, so you must learn how to spot the odd ones out.

The next layer is user accounts and authentication. Every person or service that interacts with the Linux system has a username and a unique user ID (UID). The file '/etc/passwd' lists all users. The file '/etc/shadow' stores encrypted passwords. Analysts check these files for unexpected accounts — for example, a new user called 'backup' that someone created without authorisation. If an attacker gains access, they often create a backdoor account to return later.

Log files are the system's diary. Linux logs almost everything: who logged in, when, from where, what commands they ran, and what went wrong. The main logging system is called 'syslog' or 'rsyslog', and logs are stored in the '/var/log' directory. The file '/var/log/auth.log' records authentication attempts, '/var/log/syslog' records general system messages, and '/var/log/dpkg.log' records software installations. Analysts search these logs for failed logins (a sign of brute-force attacks), logins at odd hours, or commands that should never have been run.

The file system is the tree of directories and files. A critical artefact is the 'crontab' — a configuration file that tells the system to run certain commands automatically at scheduled times. Attackers love to plant malicious cron jobs because they run silently without anyone noticing. An analyst checks a user's crontab with 'crontab -l' and the system-wide crontab files in '/etc/cron.*'.

Network connections are also vital. Every time a program on the Linux host connects to the internet or another computer, it creates a network socket. The 'netstat' or 'ss' command shows all active connections. Analysts look for connections to known malicious IP addresses, or for processes listening on unusual ports (the logical doorways into the computer). For example, if you see a process called 'sshd' listening on port 4444 instead of the standard port 22, that is a huge red flag.

Another key area is installed software. Attackers are lazy — they often use existing tools on the system instead of downloading new ones. The 'dpkg' command on Debian-based systems lists all installed packages. Analysts look for unexpected tools like a web server on a database machine, or a compiler on a system that never needs to compile code.

Finally, memory analysis is sometimes necessary. The 'lsof' command (list open files) shows which files are currently open by which processes. This is useful for finding a malicious file that an attacker is actively reading or writing.

Linux Host Analysis exists because it replaces guesswork with evidence. Before these techniques, administrators would have to restart systems or reinstall software without knowing exactly what went wrong. Now, with a systematic approach, an analyst can reconstruct the entire timeline of an attack, identify the attacker's entry point, and recommend specific fixes.

## Real-world context

A large e-commerce company called ShopFast receives an alert from its security monitoring system: a Linux web server has been sending outbound connections to an IP address known to be a command-and-control server for a botnet. The Security Operations Center (SOC) analyst, Priya, is tasked with performing a Linux Host Analysis to determine the scope of the compromise.

Priya opens a secure shell (SSH) connection to the affected server — but she does not log in using the root account directly. Instead, she uses her personal user account (which has 'sudo' privileges) to ensure an audit trail. Her first action is to check running processes using 'ps aux'. She immediately spots a process called 'httpd' running as root — that is suspicious because the legitimate web server runs under a less-privileged 'www-data' user. The process is using an unexpected amount of CPU. She makes a note of its process ID (PID).

Next, she runs 'netstat -tulpn' to see which ports the system is listening on and which connections are active. She sees that the malicious 'httpd' process has an established connection to that blacklisted IP address on port 8080. Now she knows the system is actively communicating with the attacker.

Priya moves to user accounts. She inspects '/etc/passwd' and sees a user called 'sysadmin' that was created yesterday. Her team never created that account. She checks the file modification timestamps on '/etc/shadow' to confirm the timing. The attacker left a backdoor.

She then checks the logs. Looking at '/var/log/auth.log', she finds thousands of failed SSH login attempts from a Russian IP address, followed by a successful login at 3:17 AM. That is the moment the attacker broke in. She also searches for 'sudo' commands in the log and sees that the attacker used 'sudo useradd sysadmin' to create the backdoor account.

Finally, she checks the cron jobs. Running 'crontab -l' for the 'www-data' user reveals a line that downloads a script every hour from the same malicious IP and executes it. That script is the persistence mechanism — even if the malicious process is killed, it comes back.

Priya documents everything: the malicious process, the backdoor user, the log evidence, and the cron job. She then scrubs the system by isolating it from the network, taking a forensic image of the hard drive, and deploying a clean replacement server. Her analysis directly stopped a data breach and informed the legal team about the attacker's methods.

Common tools used in real-world Linux Host Analysis include:
- 'ps' for process listing
- 'netstat' or 'ss' for network connections
- 'lsof' for open files
- 'journalctl' for viewing systemd logs
- 'grep' and 'awk' for searching logs
- 'md5sum' and 'sha256sum' for hashing files to check integrity

## Exam focus

The 200-201 exam tests Linux Host Analysis in objective 3.2, and it is a popular topic. Expect 4 to 8 questions. The exam does not ask you to run commands — you are not in a lab. Instead, you have to interpret command output and know what each piece of evidence means.

The traps on the exam are subtle. A common trick is to give you a piece of output from 'ps aux' and ask you to identify the suspicious process. Many beginners look only at the process name and miss that the user column shows 'www-data' for a process that should be root, or vice versa. Always check the user column and the CPU column together.

Another trap involves log files. The exam might show a snippet from '/var/log/auth.log' and ask you to determine the attacker's IP address. The trick is that there may be many failed login attempts from different IPs, but only one successful login — you must find that. The exam loves to test whether you can distinguish between an authentication failure event and a successful login event.

Key concepts you must memorise:
- The meaning of common log entries: 'Failed password' means authentication failed; 'Accepted password' means success.
- The default ports: SSH is port 22, HTTP is 80, HTTPS is 443. Deviations are suspicious.
- The difference between a user account and a service account. 'nobody', 'www-data', 'daemon' are typically service accounts, not interactive users.
- What 'sudo' does: elevates privileges. Log entries showing sudo indicate a user ran a command with root rights.
- The purpose of cron: automated task scheduling. Malicious cron jobs often run scripts from user-writable directories like '/tmp'.

Exam topics you will see:
- Process analysis using 'ps' and 'top' output
- Log analysis from '/var/log/auth.log', '/var/log/syslog', and '/var/log/dpkg.log'
- User account anomalies (unexpected accounts, modified UIDs, empty password fields)
- Network connection analysis ('netstat', 'ss', 'lsof -i')
- File system artefacts (hidden files, suspicious SUID/SGID permissions)
- Crontab entries and startup scripts

To pass this section, you must be able to:
1. Read a command output and identify the single anomaly.
2. Understand what each column in 'ps aux' means (USER, PID, %CPU, %MEM, VSZ, RSS, TTY, STAT, START, TIME, COMMAND).
3. Track the timeline of an attack through log timestamps.
4. Distinguish between a legitimate system process and a malicious one.

The exam also tests your understanding of file integrity. You may be asked what a change in a hash value means — it means the file has been modified. Attackers often replace legitimate binaries (like 'ls' or 'ps') with trojanised versions that hide their activity. Knowing how to verify file hashes against known good values is critical.

## Step by step

1. **Gather Initial Information** — Start by recording the hostname, IP address, current date/time, and uptime. Use 'hostname', 'ip addr', 'date', and 'uptime'. This establishes the context for the investigation and ensures that any evidence you collect later can be correlated with this baseline.
2. **Capture Running Processes** — Run 'ps aux' to list every process with its user, PID, CPU, memory usage, and the full command. Also run 'top' or 'htop' to see processes sorted by resource usage. Look for processes that should not be running (e.g., a compiler on a database server) or processes using excessive resources.
3. **Inspect Network Connections** — Use 'netstat -tulpn' or 'ss -tulpn' to list all listening ports and active connections. Note each foreign address and port. Compare against known good IPs. Any connection to a known malicious IP, or a listening service on an unusual port, is a critical finding.
4. **Examine User Accounts and Authentication Logs** — Check '/etc/passwd' and '/etc/shadow' for unexpected users. Look for accounts with UID 0 (superuser) that are not root. Then read '/var/log/auth.log' to find the last successful login, failed logins, and sudo commands. Timestamps help you build a timeline.
5. **Analyse Scheduled Tasks and Startup Scripts** — Check cron jobs with 'crontab -l' for each user and examine the system-wide cron directories in '/etc/cron.d', '/etc/cron.daily', etc. Check systemd services with 'systemctl list-units'. Look for tasks that download or execute scripts from external sources, as these are common persistence mechanisms.
6. **Review Installed Software and File Integrity** — List installed packages with 'dpkg -l' (Debian/Ubuntu) or 'rpm -qa' (Red Hat/CentOS). Look for unexpected packages. Then check critical binaries like 'ls', 'ps', 'netstat' by computing their hashes with 'md5sum' and comparing against known good values from official repositories or backup hashes.

## Comparisons

### Process (ps aux) vs Process (top)

**Process (ps aux):**
- Shows a static snapshot of all processes at the moment the command runs
- Columns include USER, PID, %CPU, %MEM, VSZ, RSS, TTY, STAT, START, TIME, COMMAND
- Best for seeing every process quickly, but does not update in real time

**Process (top):**
- Shows a dynamic, real-time view of processes, updated every few seconds
- Columns include PID, USER, PR, NI, VIRT, RES, SHR, S, %CPU, %MEM, TIME+, COMMAND
- Best for spotting processes that are currently consuming high CPU or memory

### /var/log/auth.log vs /var/log/syslog

**/var/log/auth.log:**
- Records authentication-related events: logins, logouts, sudo usage, SSH sessions
- Primarily used to identify brute-force attacks, backdoor logins, and privilege escalation
- Entries include 'Failed password', 'Accepted password', and 'sudo' commands

**/var/log/syslog:**
- Records general system messages: kernel messages, service starts/stops, hardware errors
- Used to find system crashes, driver issues, and service failures
- Entries include timestamps, hostname, service name, and a log message

### Crontab (user-specific) vs Systemd Service

**Crontab (user-specific):**
- Configured by individual users with 'crontab -e'
- Runs as the user who owns the crontab, with that user's permissions
- Schedules tasks by minute, hour, day, month, weekday pattern

**Systemd Service:**
- Configured by system administrator using unit files in /etc/systemd/system
- Runs as root or a specified service user
- Manages long-running processes, dependencies, and startup order

### netstat -tulpn vs lsof -i

**netstat -tulpn:**
- Shows listening ports (services waiting for connections) and active connections
- Output includes Proto, Local Address, Foreign Address, State, PID/Program name
- Best for a quick overview of all network services on the host

**lsof -i:**
- Lists open network connections for all processes, with file descriptor information
- Output includes COMMAND, PID, USER, FD, TYPE, DEVICE, SIZE/OFF, NODE, NAME
- Best for finding exactly which process has which specific port or connection open

## Diagram

_Flowchart showing the logical steps of a Linux Host Analysis investigation, from alert to identification and remediation._

```mermaid
graph LR
    A[Alert or Suspicious Event] --> B[Linux Host Analysis]
    B --> C[Process Analysis - ps aux]
    B --> D[Network Analysis - netstat]
    B --> E[User/Log Analysis - auth.log]
    B --> F[Persistence Check - crontab/systemd]
    C --> G[Identify Malicious Process]
    D --> H[Find C2 Connection or Backdoor Service]
    E --> I[Discover Attacker Account or Intrusion Time]
    F --> J[Remove Scheduled Malicious Task]
```

## Common misconceptions

- **Misconception:** If a process has the same name as a legitimate system process, it must be safe. **Reality:** Attackers frequently name their malware with common process names like 'sshd', 'httpd', or 'crond'. You must check the user running the process, its resource usage, its network connections, and its command path, not just its name. (This mistake comes from assuming that malware is always obviously different. In reality, attackers want to blend in, so they use familiar names to avoid detection.)
- **Misconception:** The /var/log directory is only for system logs and is rarely useful for an investigation. **Reality:** The /var/log directory is the single most important source of forensic evidence on a Linux system. It contains authentication logs, system messages, kernel logs, and application-specific logs that together can reconstruct the entire attack timeline. (Many beginners think logs are just for troubleshooting errors. They do not realise that logs capture every login attempt, every sudo command, and every service start, which is gold for an investigator.)
- **Misconception:** Checking the root user account is enough; you do not need to examine other user accounts. **Reality:** Attackers rarely break into the root account. They typically compromise a low-privilege user account first, then escalate privileges later. Inspecting all user accounts — especially ones with UID 0 (superuser) or accounts that are not supposed to exist — is essential. (This misconception comes from thinking only root matters. In reality, attackers often create new user accounts with names like 'oracle' or 'nagios' that look legitimate but are backdoors.)
- **Misconception:** A network connection to an external IP address is always suspicious and should be blocked immediately. **Reality:** Many legitimate services (software updates, DNS queries, API calls) connect to external IPs. The suspiciousness depends on whether the IP is known malicious, whether the connection is to an unexpected port, and whether the process making the connection is unexpected. (Beginners see any outbound connection as a threat. They need to learn the difference between normal traffic (e.g., an apt update) and malicious traffic (e.g., a reverse shell).)
- **Misconception:** Linux does not have a registry, so there are no persistence mechanisms. **Reality:** Linux has many persistence mechanisms: cron jobs, systemd services, init.d scripts, .bashrc files, SSH authorized_keys modifications, and kernel modules. They are just stored in files, not a single database like Windows Registry. (This mistake arises from comparing Linux to Windows. The file-based approach is actually easier for attackers to modify because many files are world-writable or owned by users.)
- **Misconception:** If the system has no malware signatures, it is clean. **Reality:** Signature-based detection only catches known malware. Attackers use custom tools, scripts, and legitimate system binaries (a technique called 'living off the land') to avoid leaving signature traces. Behavioural analysis — looking at what processes are doing — is often more important. (People trust antivirus tools too much. In a security investigation, you cannot rely on signatures; you must look at the system's behaviour.)

## Key takeaways

- Linux Host Analysis examines processes, users, logs, network connections, and file system artefacts to find evidence of compromise.
- Always check the 'ps aux' output for processes running as unexpected users or consuming abnormal CPU.
- The /var/log/auth.log file is the first place to look for authentication events — both failed and successful logins.
- A suspicious network connection involves a process connecting to an unknown IP on a non-standard port, especially if the process itself looks unusual.
- Cron jobs and systemd services are the primary persistence mechanisms on Linux; always inspect them.
- Hidden files (names starting with a dot) and files in /tmp are common hiding places for attacker tools.
- User accounts that appear legitimate but are not in your official directory are often backdoors created by attackers.
- File hashes (MD5, SHA-256) allow you to verify if a critical binary has been tampered with.

## FAQ

**What is the difference between 'ps aux' and 'ps -ef'?**

Both show running processes, but 'ps aux' is the BSD-style syntax and is more common on modern Linux systems, while 'ps -ef' is the standard UNIX syntax. They display the same information but in slightly different column order. The 200-201 exam expects you to recognise both.

**How do I find the IP address of a suspicious process?**

Use 'netstat -tulpn' or 'ss -tulpn' to see the foreign address (the IP and port the connection is going to). The 'p' flag shows the PID and process name. Cross-reference that PID with the 'ps aux' output to confirm the process.

**What does 'lsof' do and when would I use it?**

'lsof' stands for 'list open files'. It shows every file currently open by every process, including regular files, network sockets, pipes, and devices. You use it to find out which process has a specific file open, which is useful when you suspect a malicious file is being accessed.

**I see a process named 'sshd' listening on port 2222. Is that suspicious?**

Yes, it is highly suspicious. The standard SSH port is 22. A process named 'sshd' listening on 2222 could be an attacker's copy of the SSH server (a backdoor) listening on a non-standard port to avoid detection and allow remote access. Always investigate.

**What is the 'shadow' file and why is it important?**

'/etc/shadow' stores encrypted user passwords. It is readable only by the root user. An analyst checks this file for accounts with empty password fields (meaning no password is required) or for password hashes that match known weak or compromised passwords.

**How do I check if a file on Linux has been modified by an attacker?**

Compute a cryptographic hash of the file using 'md5sum' or 'sha256sum' and compare it against the known good hash. If they differ, the file has been modified. Many organisations use tools like AIDE or Tripwire to automate this process.

## Check your understanding

1. **You run 'ps aux' and see a process named 'httpd' owned by user 'nobody' consuming 95% CPU. What is the most likely explanation?**

   Answer: This is suspicious because a legitimate web server typically runs under the 'www-data' user. The high CPU usage and unexpected user suggest this is a malicious process masquerading as the real web server.

2. **What is the significance of finding a crontab entry that executes a script from /tmp every hour?**

   Answer: This is a common persistence mechanism. Attackers often place scripts in /tmp (a world-writable directory) and use cron to reinstall their malware if it is deleted.

3. **In a Linux system, which file is the primary source for authentication logs?**

   Answer: The file '/var/log/auth.log' (or on some distributions, '/var/log/secure') contains all authentication-related events including login attempts and sudo usage.

4. **During an investigation, you find an established TCP connection from the Linux host to an IP address on port 4444. The process is 'bash'. What does this typically indicate?**

   Answer: This strongly indicates a reverse shell attack. An attacker has used a bash process to create an outbound network connection, allowing them to remotely control the system.

5. **You see an entry in /var/log/auth.log: 'Mar 10 03:17:12 server sshd[1234]: Accepted publickey for root from 10.0.0.5 port 54321 ssh2: RSA SHA256:abc123'. What does this tell you?**

   Answer: This tells you that a user successfully authenticated as root via SSH using a public key, originating from IP 10.0.0.5. If that IP is not expected, this is evidence of a successful intrusion.

---

Interactive version with quiz and diagrams: https://courseiva.com/learn/cisco-cyberops-associate/linux-host-analysis