Malware forensics: reverse engineering and analysis. This is the field of IT security where investigators take malicious software — viruses, worms, trojans, ransomware — and figure out exactly what it does, how it entered a system, and what damage it caused. For the CHFI exam, mastering this concept means you can explain how evidence is extracted from malware itself, turning a piece of code into a courtroom-ready report.
Jump to a section
A simple way to picture Malware Forensics: Reverse Engineering and Analysis
Have you ever found a small, unexpected package in your mailroom with no return address and a strange ticking sound? That is exactly how a malware sample appears to a forensic investigator. You do not just throw it in the bin — you need to understand what it is, how it ticks, and what it was designed to do before it caused damage.
Let us say you take that package to a secure room. First, you look at the outside without touching the contents. You read the postmark, the handwriting, and the type of tape used. This is like static analysis of malware — examining the file structure, strings, and metadata without running it. Then, you carefully open the package and watch what happens inside a sealed transparent box. This is like dynamic analysis — executing the malware in a safe, isolated environment (called a sandbox) to observe its behaviour, like what files it creates or what internet addresses it connects to. Finally, you might take the device apart piece by piece to understand its internal wiring. This is reverse engineering — using a disassembler to turn the program’s machine code back into human-readable instructions so you can understand its logic. Just as you would never open that suspicious package on your desk without precautions, a forensic investigator never runs malware on a live system. Every step is controlled, documented, and reversible.
Malware forensics is a specialised branch of digital forensics that focuses on malicious software. The goal is to understand the malware’s behaviour, origins, and purpose without allowing it to harm real systems. This field is critical because modern cyber attacks often use custom or modified malware that signature-based antivirus tools cannot detect. Forensics analysts must dig deeper.
Let us start with the basics. Malware is any software intentionally designed to cause damage, steal data, or disrupt operations. Common types include viruses (which attach to clean files), worms (which spread without a host file), Trojan horses (which disguise themselves as legitimate software), ransomware (which encrypts files and demands payment), and rootkits (which hide deep in the operating system). When a forensic investigator gets a malware sample, they follow a structured process with three main stages: static analysis, dynamic analysis, and reverse engineering.
Static analysis means examining the malware without running it. The investigator looks at the file’s properties: its name, size, file type, and hash value (a unique digital fingerprint). A hash is a fixed-length string of characters generated by an algorithm like MD5 or SHA-1. Two identical files produce the same hash. If the malware’s hash matches a known threat database, you immediately know what you are dealing with. The investigator also extracts strings — sequences of readable text hidden in the binary code. Strings can reveal IP addresses, URLs, registry keys, or error messages. Another static technique is to examine the file’s binary structure using a hex editor, which shows the raw bytes of data in hexadecimal format. This can reveal embedded files, encryption keys, or suspicious section headers.
Dynamic analysis is the opposite: you execute the malware — but only inside a sandbox, which is an isolated virtual machine (a simulated computer) that has no network access to the real world. The sandbox records every action the malware takes. For example, does it create a new file in the Windows system folder? Does it modify the registry to run automatically at startup? Does it attempt to connect to a remote server on the internet (a callout to a command-and-control or C2 server)? Dynamic analysis tools include process monitors, file system monitors, and network traffic analyzers. This stage gives you a live behaviour log of the malware.
Reverse engineering is the deepest level of analysis. Here, the investigator uses a disassembler (like IDA Pro or Ghidra) to convert the malware’s machine code — the raw binary instructions a computer’s CPU understands — back into assembly language, which is a low-level human-readable programming language. A disassembler translates bytes like 55 8B EC into instructions like push ebp; mov ebp, esp. An investigator might also use a debugger, which lets them step through the code one instruction at a time, watching how the CPU registers and memory change. This is how they discover hidden encryption algorithms, backdoor passwords, or anti-analysis techniques (like code that detects if it is running in a sandbox and stops executing).
The field exists because organisations need to respond to incidents. If a company’s server gets infected with ransomware, the forensic analyst must determine how the ransomware got in (the infection vector), what data was encrypted, and whether a decryptor can be built. Without reverse engineering, you cannot create signatures for antivirus tools or write a proper incident report for law enforcement. The CHFI exam focuses on the methodology: the order of steps, the purpose of each tool, and how to preserve evidence (like keeping the original malware file in a write-protected state to avoid altering its hash). You need to know the difference between static and dynamic analysis, be able to name common tools used in each phase, and understand the legal chain of custody requirements for malware samples.
Evidence Acquisition and Preservation
The investigator creates a bit-for-bit forensic copy of the malware sample using a write-blocker to prevent any modification. The original storage medium is kept in a secure evidence locker. The hash of the original file is recorded. This step is critical because any change to the evidence makes it inadmissible in court.
Static Analysis (First Look)
The investigator examines the malware file without executing it. This involves viewing the file header, extracting human-readable strings, checking for embedded resources, and identifying the packer or obfuscation tool. A hash lookup in online databases like VirusTotal can instantly reveal if the malware is known. This step provides initial clues about the malware's type and capabilities.
Dynamic Analysis (Behavioural Observation)
The malware is executed inside a controlled sandbox environment — a virtual machine with no production network access. Specialised monitoring tools log every system call, file creation, registry change, process start, and network connection. The investigator watches for actions like persistence mechanisms, C2 communication, and data exfiltration. This step reveals the real-world behaviour of the malware.
Reverse Engineering (Code-Level Understanding)
Using a disassembler or debugger, the investigator converts the malware's machine code back into assembly language. They identify key functions like the encryption routine, the command-and-control protocol, and any hidden payloads. This step is essential for creating custom detection signatures, building a decryptor, or understanding how the malware evades detection.
Reporting and IOC Extraction
All findings are compiled into a formal forensic report that includes the original hash, analysis methodology, behavioural observations, and a list of IOCs. The report is shared with the incident response team so they can search for the same IOCs across the network and apply countermeasures like firewall rules or endpoint detection rules. This step translates technical findings into actionable security outcomes.
Imagine you are a junior forensic analyst at a mid-sized financial services company. One morning, the IT helpdesk gets a flood of calls: employees cannot open their files, and a ransom note has appeared on every screen demanding payment in Bitcoin. The company’s incident response team isolates the affected computers from the network and hands you a copy of a suspicious executable file found on the first infected machine. Your job is to perform malware forensics.
Here is your step-by-step real-world scenario:
First, you create a forensic copy of the file using a write-blocker — a hardware device that prevents any data from being written to the original storage medium. This preserves the chain of custody. You record the original file’s hash in your evidence log.
Second, you perform static analysis. You open the file in a tool called PEview, which shows the Portable Executable (PE) structure of a Windows binary. You note that the file imports functions like CreateRemoteThread and VirtualAllocEx — these are common in process injection malware. You extract all human-readable strings and find an IP address: 192.168.1.50. A reverse DNS lookup (done safely offline) shows it is not a legitimate server owned by your company. You also notice the file is packed, meaning its code is compressed or obfuscated to avoid detection. You use a tool like PEiD to identify the packer.
Third, you move to dynamic analysis. You load the malware into a sandbox virtual machine that has no network access except to a simulated internet service (like INetSim). You execute the file. The sandbox reports:
- The malware created a file in C:\Users\Public\svchost.exe.
- It added a registry run key to HKLM\Software\Microsoft\Windows\CurrentVersion\Run.
- It attempted to contact 192.168.1.50 on port 443 (HTTPS).
- It deleted its own original executable after 30 seconds.
This confirms the malware is a trojan that establishes persistence and tries to phone home for further instructions.
Fourth, you proceed to reverse engineering. You load the unpacked malware into IDA Pro. You look for the function that contains the IP address you found earlier. You trace the code and discover it decrypts a second-stage payload using a simple XOR key. You extract that key and the encrypted blob. You now know exactly how the malware communicates and what it downloads next.
Finally, you write your report. You include the file hash, the static and dynamic analysis findings, the decrypted communication details, and indicators of compromise (IOCs) like the IP address and registry key. The security team uses your IOCs to update firewall rules and run a scan across all endpoints. The company avoids paying the ransom because you proved that the encryption is standard and a backup restore is safe. Without your malware forensics work, the company might have paid the ransom or remained infected.
The CHFI exam (312-49) tests malware forensics in several specific ways. First, you will be asked to identify the correct order of analysis: static first, then dynamic, then reverse engineering if needed. A common trap is a question that suggests running the malware on a live production system — the correct answer always involves an isolated sandbox or virtual machine. Another trap is confusing static and dynamic analysis: static analysis does not execute the file, dynamic does.
Key concepts you must memorise for the exam:
The definition of a hash and its role in file identification and integrity verification. Expect a question asking which hashing algorithm (MD5, SHA-1, SHA-256) is most common for malware sample identification. The answer is MD5 for speed, though SHA-256 is more secure.
The difference between a disassembler and a debugger. A disassembler (like IDA Pro) converts machine code to assembly without running the program. A debugger (like OllyDbg) lets you run the code step by step.
Anti-analysis techniques: packers, obfuscation, anti-debugging checks (like IsDebuggerPresent API), and timing checks that detect if the malware is running in a sandbox. The exam may ask what a packer does: compresses or encrypts the original code to avoid static detection.
Common tools: PEview, Hex Workshop, IDA Pro, OllyDbg, Ghidra (open-source), Wireshark for network analysis, and sandbox solutions like Cuckoo Sandbox.
The chain of custody for malware samples: always use a write-blocker, keep the original storage write-protected, and document every step in a forensic log.
Indicators of compromise (IOCs): file hashes, IP addresses, domain names, registry keys, file paths, and mutexes (named objects malware creates to ensure only one copy runs). Expect a multiple-choice question listing several IOCs and asking which one is a network-based IOC.
Static analysis techniques: examining file headers, strings, imports/exports, embedded resources, and packer detection. Dynamic analysis techniques: monitoring file system changes, registry changes, process creation, and network connections.
The exam loves to test the purpose of the Windows Registry and how malware uses Run keys or scheduled tasks to achieve persistence. You need to know the exact registry path: HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Run.
You will also see questions about portable executable (PE) file structure: the DOS header, NT header, section table, and the difference between .text (code), .data (initialised data), and .rdata (read-only data) sections.
Exam question styles include: “Which of the following is a static analysis technique?” (answer: examining strings), “What is the first step in analysing a malware sample?” (answer: make a bit-for-bit forensic copy), and “Which tool is used to view the PE structure of a Windows executable?” (answer: PEview). Traps often use plausible but wrong tool names like “Malwarebytes” (an antivirus, not a forensic tool) or suggest dynamic analysis before static analysis.
To pass, memorise the taxonomy of malware (virus, worm, trojan, ransomware, rootkit, spyware, adware) and know which type is characterised by self-replication without a host file (worm). You must also understand the concept of a sandbox escape — malware that detects it is in a virtual machine and behaves differently. This is an advanced anti-analysis technique the exam may reference.
Static analysis examines malware without executing it; dynamic analysis runs it in an isolated sandbox to observe behaviour.
A hash (MD5, SHA-1, SHA-256) is the unique digital fingerprint of a file and is used to identify known malware and verify evidence integrity.
Reverse engineering uses a disassembler (like IDA Pro or Ghidra) to convert machine code into human-readable assembly language to understand malware logic.
Anti-analysis techniques include packing, obfuscation, anti-sandbox checks, and anti-debugging calls that malware uses to evade forensic tools.
Indicators of Compromise (IOCs) include file hashes, IP addresses, domain names, registry keys, and file paths that help detect and contain the same malware across an organisation.
The chain of custody requires a write-blocker, forensic copy, hash verification, and detailed logging from the moment of evidence seizure through analysis and reporting.
These come up on the exam all the time. Here's how to tell them apart.
Static Analysis
Examines malware without execution.
Uses hex editors, PE viewers, and string extractors.
Cannot observe runtime behaviour like network connections.
Dynamic Analysis
Executes malware in a monitored sandbox.
Uses process monitors, file monitors, and network traffic tools.
Reveals real-time actions such as file creation and C2 communication.
Disassembler
Converts machine code to assembly in a single pass.
Does not execute the program.
Output is a static listing of all instructions.
Debugger
Executes the program step by step.
Allows inspection of registers, memory, and stack at each step.
Can modify the program flow during analysis.
Malware File Hash
A mathematical fingerprint of the exact file content.
Unique to that file; even a single byte change alters the hash.
Used for identification and integrity verification.
Antivirus Signature
A pattern or string that antivirus software uses to detect malware.
Can be updated by vendors to catch variants.
Prone to evasion via code obfuscation and packing.
Portable Executable (PE) Format
File format for Windows executables and DLLs.
Includes DOS header, NT header, and section table.
Common sections: .text (code), .data, .rdata.
ELF Format (Executable and Linkable Format)
File format for Linux and Unix executables.
Uses ELF header, program header table, and section header table.
Common sections: .text, .data, .bss.
Trojan Horse
Disguises itself as legitimate software to trick users.
Requires user action (e.g., opening an attachment) to infect.
Does not self-replicate; spreads via social engineering.
Worm
Self-replicating malware that spreads across networks without user action.
Exploits vulnerabilities or network shares to propagate.
Often consumes network bandwidth and system resources.
Command-and-Control (C2) Communication
The remote server that sends commands to infected systems.
Uses protocols like HTTP, HTTPS, DNS, or IRC.
Primary goal: issue instructions to the botnet or malware.
Exfiltration Channel
The path used to steal data from the infected system.
Often uses encrypted tunnels or covert timing channels.
Primary goal: transmit stolen data (e.g., credit card numbers, documents).
Mistake
You can safely run malware on a normal computer as long as you have antivirus software installed.
Correct
Malware must only be executed in an isolated sandbox or virtual machine with no network access to the real environment. Antivirus cannot protect against unknown malware or prevent unintended spread.
Beginners often think that antivirus is a safety net. In reality, malware analysis is about observing behaviour, not prevention, and antivirus may fail to detect new or polymorphic malware.
Mistake
Static analysis means just looking at the malware's name and file size.
Correct
Static analysis includes examining the file's binary structure, strings, imports, exports, packer detection, and hash calculation using tools like hex editors and PE viewers.
Newcomers assume static means superficial. They do not realise it involves deep inspection of the binary code without execution, which reveals valuable hidden data.
Mistake
All malware analysis requires deep programming knowledge in assembly or C.
Correct
Basic malware forensics can be done with static analysis and dynamic analysis using automated tools. Full reverse engineering requires assembly knowledge, but the CHFI exam focuses on methodology, not low-level coding.
People fear the topic because they think they need to be expert programmers. Actually, many forensic roles involve using prepackaged tools and interpreting logs.
Mistake
A malware sample's hash should change after analysis because the file gets modified.
Correct
A properly preserved forensic copy is never altered. The original file's hash is recorded before analysis and must remain identical to prove evidence integrity.
Beginners confuse evidence handling with regular file use. In forensics, any modification is contamination that breaks the chain of custody.
Mistake
Dynamic analysis can be done by just double-clicking the malware on a test machine.
Correct
Dynamic analysis requires a monitored sandbox with tools that log every system call, registry change, and network connection. Simply running it provides no useful forensic data.
People think just executing malware will show what happens. Without monitoring tools, you miss most behavioural indicators.
Reveal each answer, then mark whether you got it right. Score 60%+ to unlock the next chapter.
Static analysis examines the malware without running it (looking at strings, headers, packers). Dynamic analysis executes the malware inside an isolated sandbox and observes its runtime behaviour (file changes, network connections, registry modifications).
No, the CHFI exam does not require you to read assembly code. You just need to understand what a disassembler does and the purpose of reverse engineering. The focus is on methodology, tools, and chain of custody.
A packer is a tool that compresses or encrypts the original executable code to reduce its size and obfuscate its real content. Malware authors use packers to evade static signature detection by antivirus software.
Yes, but with caution. Uploading a sample to VirusTotal sends it to a third party, which may break the chain of custody for legal evidence. Always hash the file first and only upload if permitted by your organisation's policy. For the exam, hash lookup is considered a valid static analysis step.
A sandbox is an isolated virtual machine environment where malware can be executed safely without risking the real network or host system. It has monitoring tools that record all file, registry, process, and network activity of the malware.
IOCs are pieces of forensic data — such as file hashes, IP addresses, domain names, registry keys, or file paths — that uniquely identify a specific malware infection. They allow security teams to detect and contain the same malware across multiple systems.
You've finished Malware Forensics: Reverse Engineering and Analysis. Continue through the 312-49 study guide to build a complete picture of the exam.
Done with this chapter?