Courseiva
312-50Chapter 5 of 18Objective 4.1

System Hacking

System hacking is the process attackers use to break into a computer system and take control of it. For the CEH 312-50 exam, understanding system hacking is critical because it covers the core tactics you will need to know to both attack and defend networks. This chapter breaks down the three main phases—password cracking, privilege escalation, and executing applications—so you can see exactly how an attacker moves from the outside to full control.

12 min read
Intermediate
Updated Jul 23, 2026
Reviewed by Johnson Ajibi· Senior Network & Security Engineer · MSc IT Security

A simple way to picture System Hacking

The Fake Landlord Inspection Analogy

First, you notice your neighbour's spare key hidden under the doormat, and that leads to you being able to unlock their front door when they're not home.

In this analogy, you are the ethical hacker (playing the role of an attacker), and the neighbour's house represents a target computer system on a network. The spare key under the doormat is like a weak or default password—easily discovered by anyone who looks. Once you have that key, you enter the house. Inside, you find a locked safe in the bedroom. That safe is like an administrative account or a higher-privilege user account. You cannot open the safe with the front-door key. So you look around and find a sticky note on the computer monitor with the safe combination written on it. That sticky note is like a privilege escalation vulnerability—a piece of information or a flaw that lets you move from a normal user to an administrator. You open the safe and find a USB drive containing a list of all the neighbour's bank passwords. That USB drive is like a sensitive file (e.g., the SAM database) that you now have permission to access. You copy it onto your own device. Finally, before you leave, you hide a tiny camera in the living room, so you can watch the neighbour later. That hidden camera is like a backdoor or a remote-access Trojan (RAT) that you install to maintain persistent access to the system. The whole process—finding the spare key, escalating to the safe, stealing the data, and planting the camera—maps exactly to the system hacking methodology: gaining access, escalating privileges, extracting information, and maintaining access.

How It Actually Works

System hacking is the phase in ethical hacking where you move from the network perimeter to the target computer itself. It is what happens after you have already discovered a target (reconnaissance) and identified a weakness (scanning and enumeration). Now you are actively trying to break in.

The first major technique is password cracking. A password is a string of characters used to verify a user's identity. Attackers do not need to guess the password from nothing—they often use lists of common passwords or leaked databases. Password cracking tools, like John the Ripper or Hashcat, work by taking a list of candidate passwords, turning each one into a hash (a fixed-length string output from a mathematical function), and comparing that hash to the stolen hash from the target system. If they match, you have found the password. This is called an offline attack because you are not trying to log in repeatedly; you are just comparing hashes on your own machine. The most common types are dictionary attacks (using a wordlist) and brute-force attacks (trying every possible combination). Salting is a defence where a unique random value is added to each password before hashing, which makes pre-computed hash lists (rainbow tables) useless.

The second technique is privilege escalation. Once you have a normal user account (like a standard employee login), you need to gain higher privileges—typically Administrator or SYSTEM level on Windows, or root on Linux. This is because many critical files and system settings are only accessible to high-privilege accounts. There are two types: vertical escalation (moving from user to admin) and horizontal escalation (accessing another user's account at the same privilege level). Attackers do this by exploiting bugs in the operating system or installed software—for example, a buffer overflow vulnerability that lets them run code with elevated rights. Another common method is token impersonation, where an attacker steals an access token (a temporary pass) from a higher-privilege process. On Windows, tools like Mimikatz can extract plaintext passwords or hash values from memory, and if the system has weak service permissions, you can use that to run commands as SYSTEM.

The third technique is executing applications. After you have gained access and escalated privileges, you need to do something useful—like install a backdoor, a keylogger, or a remote access tool (RAT). A backdoor is a piece of software that gives you a secret way back into the system, bypassing normal authentication. A keylogger records every keystroke the user types, capturing passwords and sensitive data. A RAT lets you control the computer remotely. Attackers often use a technique called DLL injection to hide malicious code inside a legitimate running process, making it harder for antivirus software to detect. Another method is to modify the Windows registry to run your malicious program every time the system starts, which is called maintaining persistence.

The entire system hacking process follows a methodology: gaining access, escalating privileges, maintaining access, and clearing logs. Clearing logs is the final step—attackers delete or alter event logs to cover their tracks. For the CEH exam, you must know the tools for each phase: password cracking (John the Ripper, Hashcat, Cain & Abel), privilege escalation (Mimikatz, Windows Exploit Suggester), and executing applications (Netcat, Metasploit, RATs). You also need to understand how defences work: strong password policies, account lockout mechanisms, least privilege principle, and application whitelisting.

Flowchart showing the system hacking methodology from initial access through privilege escalation, executing applications, maintaining access, and clearing logs.

Walk-Through

1

Gaining Access

The attacker obtains initial entry into the system. This is often done by cracking a weak password using a dictionary attack, exploiting a service vulnerability, or using social engineering to get login credentials.

2

Escalating Privileges

Once inside with a limited account, the attacker exploits a vulnerability or misconfiguration to gain higher privileges (e.g., Administrator or SYSTEM). This allows them to access sensitive files and execute commands with full control.

3

Executing Applications

With elevated privileges, the attacker installs malicious software like a backdoor, keylogger, or RAT. This step is about achieving the attack objective—data theft, surveillance, or persistent access.

4

Maintaining Access

The attacker ensures they can return to the system even if the initial entry method is closed. They install a backdoor, create hidden user accounts, or modify system startup files to auto-launch their malicious code.

5

Clearing Tracks

The attacker deletes or alters event logs, removes evidence of file modifications, and cleans up any trace of their tools. This makes forensic investigation difficult and delays detection.

What This Looks Like on the Job

An IT security professional, let us call her Priya, works for a mid-sized financial services company. One morning, she receives an alert from the company's intrusion detection system (IDS) about unusual outbound traffic from an employee's workstation. That employee works in finance and handles sensitive client data. Priya's job is to investigate whether this is a real attack or a false alarm.

First, Priya isolates the workstation from the network to prevent any further data exfiltration. She then takes a forensic image of the hard drive so she can analyse it without altering evidence. Using a tool like FTK Imager, she creates a bit-for-bit copy. Next, she runs a password-cracking analysis. She finds a suspicious process running in memory. She uses Mimikatz to dump the hashes of all local accounts. One hash matches the administrator account, even though the employee does not have admin rights. This suggests the attacker already performed privilege escalation. She then checks the Windows Event Logs for logon events and finds hundreds of failed logon attempts followed by a successful one—this looks like a dictionary attack against the employee's password.

Priya now reconstructs the attack timeline. She uses John the Ripper to crack the employee's password from the captured hash, and it turns out to be 'Spring2024!'—a common password based on a seasonal pattern. The attacker used that low-privilege account, then exploited a known vulnerability in an outdated version of a PDF reader installed on the workstation. That exploit gave the attacker SYSTEM-level privileges. Once SYSTEM, the attacker used a RAT to exfiltrate client contact lists in small chunks over several days to avoid detection. Priya finds the RAT's executable hidden inside a legitimate Windows system folder under a misleading name like 'svchost.exe.backup'.

Priya's response includes: resetting the compromised password, applying the latest security patches to all workstations, disabling the old PDF reader version company-wide, implementing account lockout after 5 failed attempts, and deploying endpoint detection and response (EDR) software to catch similar RAT installations in real time. She also writes a report for management detailing the attack path and recommends mandatory security awareness training to discourage weak passwords like 'Spring2024!'.

How 312-50 Actually Tests This

The CEH 312-50 exam tests system hacking thoroughly in Domain 4. Expect questions that ask you to identify the correct order of steps in the system hacking methodology, to match tools to their function, and to recognise attack types from brief descriptions.

Key concepts the exam loves to test:

The system hacking cycle: Gaining Access -> Escalating Privileges -> Maintaining Access -> Clearing Logs. They will present a scenario and ask which phase a specific action belongs to.

The difference between passive and active online attacks, offline attacks, and non-electronic attacks (like shoulder surfing or social engineering).

The use of LM and NTLM hashes. LM hashes are old, weak, and do not use salting. NTLM is newer but still vulnerable to pass-the-hash attacks. You need to know that Windows stores password hashes in the SAM file, which is located at C:\Windows\System32\config\SAM and is locked while the OS is running.

Password cracking tools: John the Ripper (supports many hash types), Hashcat (GPU-accelerated, fast), Cain & Abel (also does sniffing), Ophcrack (uses rainbow tables).

Pass-the-hash attacks: an attacker uses a captured hash (not the plaintext password) to authenticate to other systems. This bypasses the need to crack the password. The tool Mimikatz is the defacto choice.

Privilege escalation methods: exploiting unquoted service paths, weak service permissions, missing patches (like EternalBlue), and token manipulation.

Executing applications: they will ask about backdoors (netcat, Netbus), keyloggers (hardware vs software), and RATs. They also test DLL injection and process hollowing.

Traps they set:

They will give a long description that sounds like privilege escalation but is actually lateral movement (moving between systems using the same account).

They will list a series of tools and ask which one is NOT used for password cracking. For example, Nmap is not a password cracker—it is a scanning tool.

They will describe an attack where an attacker uses a flash drive to infect a computer, and they will call it a 'trojan'—but the correct term is a 'logic bomb' only if it detonates based on a condition. Be precise.

They will use the term 'brute force' when the description clearly indicates a dictionary attack (using a wordlist). Read carefully.

Key definitions to memorise:

Pass-the-hash: using a captured hash to authenticate without cracking.

Kerberos ticket: a ticket-granting ticket (TGT) can be stolen for pass-the-ticket attacks.

SAM (Security Account Manager): file that stores local user hashes.

LSASS (Local Security Authority Subsystem Service): process that stores credentials in memory, target for Mimikatz.

Backdoor: program that bypasses normal authentication.

Keylogger: records keystrokes.

Rootkit: hides malicious files from the operating system.

DLL injection: runs code inside a legitimate process.

Key Takeaways

System hacking follows a four-step cycle: Gaining Access, Escalating Privileges, Maintaining Access, and Clearing Logs.

Password cracking is typically done offline using stolen hashes, not by guessing at a login screen.

A standard user account cannot perform administrative actions; you must escalate privileges to modify system files or install software.

Mimikatz is the primary tool for extracting passwords and hashes from Windows memory (LSASS process).

Pass-the-hash attacks allow an attacker to authenticate using a captured hash without ever knowing the plaintext password.

Maintaining access often involves installing a backdoor or RAT and configuring it to launch at startup via the registry or scheduled tasks.

Clearing logs is the final step in system hacking to avoid detection and forensic investigation.

Salting makes rainbow table attacks ineffective because each password has a unique hash even if the password is identical.

Easy to Mix Up

These come up on the exam all the time. Here's how to tell them apart.

Dictionary Attack

Uses a wordlist of common passwords

Much faster than brute-force

Ineffective if password is not in the list

Brute-Force Attack

Tries every possible character combination

Very slow for long passwords

Will eventually find the password given enough time

Vertical Escalation

Moving from a lower privilege to a higher one (user to admin)

Grants access to system-level resources

Requires exploiting a vulnerability or misconfiguration

Horizontal Escalation

Moving from one user account to another at the same privilege level

Grants access to another user's files or sessions

Often done via credential theft or token stealing

Offline Attack

Performed on stolen data (e.g., hashes) away from the target

No account lockout risk

Common tools: John the Ripper, Hashcat

Online Attack

Performed live against the target's login service

Can trigger account lockout

Common tools: Hydra, Medusa

Backdoor

Provides remote access to a system

Often uses a network listener

Can be detected by scanning for open ports

Rootkit

Hides the attacker's presence from the OS

Modifies kernel or system files

Difficult to detect with standard antivirus

Watch Out for These

Mistake

Password cracking always means trying every possible combination until you guess correctly.

Correct

Most real-world password cracking uses lists of leaked passwords (dictionary attacks) or pre-computed hash tables (rainbow tables). Brute-force attacks are only used when the password is short or when there is no other option, because they are very time-consuming.

Beginners think of 'hacking' as someone typing random guesses at a login screen, based on movies. They do not realise attackers steal the hash file first and crack it offline.

Mistake

If you have a user password, you automatically have full access to the system.

Correct

A standard user account has limited privileges. You cannot install software, change system settings, or access other users' files. You need privilege escalation to become administrator or SYSTEM.

People assume all accounts are equal because on their personal computer they have admin rights. In corporate environments, least privilege is enforced.

Mistake

You cannot get a Windows password hash because the SAM file is encrypted and locked.

Correct

The SAM file is locked while Windows is running, but you can dump hashes from memory (LSASS process) using tools like Mimikatz, or you can boot from a Live USB to copy the SAM file offline.

The term 'locked' sounds absolute. Beginners do not understand that memory can be read and that offline access bypasses the lock.

Mistake

Hashes are useless once the password is cracked because the plaintext password is what matters.

Correct

Hashes themselves can be used directly in pass-the-hash attacks. You do not always need the plaintext password. On Windows networks, you can authenticate to remote services using just the hash.

This misconception stems from thinking authentication always requires a password entry. In reality, NTLM authentication accepts the hash directly.

Mistake

Privilege escalation only happens through software vulnerabilities that give you admin rights instantly.

Correct

Privilege escalation can also happen through misconfigurations like weak service permissions, unquoted service paths, or credential theft from memory (Mimikatz). These do not involve a vulnerability in the OS kernel.

The word 'escalation' sounds like a single exploit. Most beginners do not consider that an attacker might steal an admin's password that is cached in memory.

Do You Actually Know This?

Reveal each answer, then mark whether you got it right. Score 60%+ to unlock the next chapter.

Frequently Asked Questions

What is the difference between a dictionary attack and a brute-force attack?

A dictionary attack uses a wordlist of common passwords, while a brute-force attack tries every possible combination of characters (e.g., aaaa, aaab, aaac...). Dictionary attacks are much faster but only work if the password is in the list.

Can I crack a Windows password without the SAM file?

Yes, you can extract password hashes from the LSASS process memory using Mimikatz while the system is running. You can also capture hashes over the network using a tool like Responder.

What is a pass-the-hash attack?

It is an attack where the attacker uses a stolen NTLM hash to authenticate to a remote Windows service without needing the plaintext password. Tools like Mimikatz can perform this attack.

What is the SAM file and why is it important?

The SAM (Security Account Manager) file stores local user password hashes on Windows. It is located at C:\Windows\System32\config\SAM. Attackers target it because cracking the hashes reveals user passwords.

How do I defend against privilege escalation?

Apply the principle of least privilege (users only have the rights they need), keep software patched, disable unnecessary services, use application whitelisting, and enable Windows Defender Credential Guard to protect against Mimikatz.

What is a backdoor?

A backdoor is a piece of software that gives an attacker a secret way to access a system, bypassing normal authentication. It often listens on a network port for incoming commands.

Terms Worth Knowing

Keep going

You've finished System Hacking. Continue through the 312-50 study guide to build a complete picture of the exam.

Done with this chapter?