This chapter covers network traffic analysis, a core skill for the CompTIA CySA+ CS0-003 exam, falling under Domain 3: Security Operations, Objective 3.1 (Given a scenario, analyze data as part of security monitoring activities). Approximately 15–20% of exam questions touch on traffic analysis, packet inspection, and protocol analysis. Mastering this topic enables you to detect intrusions, identify malicious patterns, and validate security controls—skills essential for any cybersecurity analyst.
Jump to a section
Think of network traffic analysis as reviewing security footage from an airport. The airport has hundreds of cameras (sensors) capturing every movement (packet). Each passenger (packet) has a ticket (header) showing origin (source IP), destination (destination IP), flight number (port), and class (protocol). The footage is stored in a DVR (packet capture file). An analyst can fast-forward (aggregate statistics), rewind (retrospective analysis), or zoom in on a specific passenger (follow TCP stream). Just as security reviews footage to find a suspicious person who entered a restricted area (intrusion), a network analyst examines packets to detect an attacker who sent malformed data to a vulnerable service. The analyst sets filters—like looking only at passengers from a specific flight (filter by IP)—to reduce hours of footage to minutes of relevant clips. They can also correlate timestamps across cameras (time-based analysis) to reconstruct a suspect's path through the terminal (attack chain). The key is knowing what normal looks like: typical passenger flow (baseline traffic patterns) versus a sudden crowd at an exit (DoS attack). Without this analogy, you might think of packets as abstract data; with it, you see them as evidence in a physical investigation.
What is Network Traffic Analysis?
Network traffic analysis is the process of intercepting, recording, and examining network packets to understand what is happening on a network. It is used for troubleshooting, performance monitoring, and, most critically for CySA+, security incident detection and response. Unlike log analysis which looks at system or application logs, traffic analysis provides raw, unadulterated visibility into every packet that traverses the network, including data that may not be logged elsewhere.
Why It Exists
Networks are the backbone of modern IT. Attackers often use network-borne attacks—scanning, exploitation, command-and-control (C2), data exfiltration—that leave traces in packets. Firewalls, IDS/IPS, and endpoint protection generate alerts, but they can miss or misinterpret traffic. Manual traffic analysis lets an analyst confirm or refute those alerts, discover zero-day attacks, and perform forensic reconstruction.
How Traffic Analysis Works Internally
Traffic analysis relies on packet capture (pcap) data. The process involves:
- Capture: Using tools like tcpdump, Wireshark, or network taps to copy packets from the wire.
- Filtering: Applying BPF (Berkeley Packet Filter) syntax to isolate relevant traffic (e.g., tcp port 80).
- Decoding: The capture tool interprets raw bits into protocol headers (Ethernet, IP, TCP/UDP, application).
- Analysis: Examining headers, payloads, timing, and flows to identify anomalies.
Key Components, Values, Defaults, and Timers
Packet Structure: Ethernet frame (14 bytes header), IP header (20–60 bytes), TCP header (20–60 bytes), payload. MTU is typically 1500 bytes for Ethernet.
TCP Flags: SYN, ACK, FIN, RST, PSH, URG. Attackers often manipulate flags (e.g., SYN scan uses SYN alone; FIN scan uses FIN).
Default Capture Size: tcpdump default snaplen is 262144 bytes; Wireshark default is 65535 bytes. For performance, set snaplen to 68–100 bytes to capture only headers.
Timeouts: TCP connection timeout is typically 60–120 seconds (varies by OS). Wireshark's default TCP stream timeout is 2 minutes.
Port Numbers: Well-known (0–1023), registered (1024–49151), dynamic/private (49152–65535). Exam expects knowledge of common ports: 21 (FTP), 22 (SSH), 23 (Telnet), 25 (SMTP), 53 (DNS), 80 (HTTP), 110 (POP3), 143 (IMAP), 443 (HTTPS), 445 (SMB), 3389 (RDP).
Configuration and Verification Commands
tcpdump (Linux):
- tcpdump -i eth0 -w capture.pcap — capture to file.
- tcpdump -r capture.pcap -X — read and hex dump.
- tcpdump -n -i eth0 tcp port 22 — capture SSH traffic without DNS resolution.
- tcpdump -c 100 — capture only 100 packets.
Wireshark (GUI):
Filters: ip.addr == 192.168.1.1, tcp.port == 80, http.request.
Statistics: IO Graph, Flow Graph, Conversations, Endpoints.
Follow TCP Stream: Right-click a TCP packet → Follow → TCP Stream.
tshark (CLI version of Wireshark):
- tshark -i eth0 -w output.pcap
- tshark -r capture.pcap -Y "http.request" — display filter.
- tshark -r capture.pcap -T fields -e ip.src -e ip.dst -e tcp.port — extract specific fields.
Capsa (Windows):
Professional tool with expert diagnosis; used in enterprise environments.
How It Interacts with Related Technologies
IDS/IPS: Snort and Suricata generate alerts based on signatures. Analysts use traffic analysis to validate alerts (e.g., confirm a SQL injection attempt by examining HTTP payload).
SIEM: SIEMs like Splunk or ELK ingest netflow or packet metadata. Full packet capture (FPC) is often stored separately due to volume.
NetFlow/sFlow: Flow data provides summaries (IPs, ports, bytes) without full packets. Traffic analysis complements flow by providing payload details.
Endpoint Detection and Response (EDR): EDR logs system calls and network connections. Traffic analysis can verify if a connection actually contained malicious data.
Advanced Analysis Techniques
Protocol Anomaly Detection: Looking for non-standard protocol behavior, e.g., HTTP on port 443 (should be HTTPS), or malformed headers.
Timing Analysis: Unusually short or long delays can indicate scanning (rapfire) or data exfiltration (slow drip).
Statistical Analysis: Baseline normal traffic volume, then detect spikes (DoS) or periodic beacons (C2). Tools like Wireshark's IO Graph help.
Signatures vs. Heuristics: Signatures detect known patterns (e.g., specific byte sequence in a malware download). Heuristics detect anomalies (e.g., large outbound DNS queries).
Common Traffic Analysis Use Cases for CySA+
Malware C2 Detection: Beaconing traffic—regular intervals, small payloads, same destination IP. Example: A host sends a 64-byte packet to 203.0.113.5 every 60 seconds.
Data Exfiltration: Large outbound transfers, especially to unusual ports or IPs, or using DNS tunneling (long subdomains with base64-encoded data).
Port Scanning: Sequential connection attempts to multiple ports on a single IP (vertical scan) or same port across multiple IPs (horizontal scan). SYN packets alone (no ACK) indicate SYN scan.
ARP Spoofing: Multiple ARP replies with different MAC addresses for the same IP. Wireshark filter: arp.duplicate-address-detected.
DNS Tunneling: Queries with long subdomain names, high frequency, or unusual record types (TXT, NULL).
Traffic Analysis in Cloud and Virtualized Environments
VPC Traffic Mirroring: AWS, Azure, GCP allow mirroring VM traffic to a collector. Mirrored traffic is encapsulated (e.g., VXLAN).
Virtual Switches: In VMware, promiscuous mode on vSwitch allows capturing VM traffic. Performance overhead can be significant.
Container Networks: Docker uses bridge networking; capturing requires running tcpdump inside the container or using CNI plugins.
Performance Considerations
Capture Rate: High-speed networks (10 Gbps+) require dedicated capture hardware or software like PF_RING or AF_PACKET to avoid packet loss.
Storage: Full packet capture at 1 Gbps generates ~450 GB per day. Retention policies are critical.
Filtering at Capture: Apply BPF filters to reduce volume (e.g., capture only TCP port 80, 443).
Legal and Ethical Considerations
Wiretap Laws: In many jurisdictions, capturing traffic without consent is illegal. Ensure authorization before capturing on production networks.
Privacy: Packet payloads may contain PII. Anonymize or restrict access to captures.
Verification of Analysis
Confirm with Multiple Tools: If Wireshark shows a suspicious packet, verify with tcpdump or another tool.
Cross-reference with Logs: Check firewall logs, IDS alerts, and endpoint logs to corroborate findings.
Time Synchronization: Ensure all devices use NTP; otherwise, correlating events becomes unreliable.
Capture traffic on the network
Use a packet capture tool like tcpdump or Wireshark to record packets from a network interface. On Linux, run `tcpdump -i eth0 -w capture.pcap`. Set an appropriate snaplen to capture full packets or just headers. For security analysis, capture full packets to see payloads. Ensure the interface is in promiscuous mode to capture all traffic, not just packets destined to the host. Monitor capture buffer to avoid packet loss—use `-B` to set buffer size (e.g., `-B 4096` for 4 MB).
Apply filters to isolate relevant traffic
Use BPF syntax to focus on specific protocols, IPs, or ports. For example, `tcp port 80` captures only HTTP traffic. In Wireshark, use display filters like `http.request` to see only HTTP requests. Filtering reduces noise and speeds up analysis. Common filters: `ip.addr == 10.0.0.1`, `tcp.port == 443`, `icmp`, `arp`. Remember that capture filters (BPF) are applied before capture, while display filters are applied after—use capture filters to limit volume on busy networks.
Analyze packet headers for anomalies
Examine IP and TCP/UDP headers. Look for unusual flag combinations (e.g., SYN-FIN packets indicate scanning), TTL values that don't match the OS (e.g., Windows default TTL 128 vs. Linux 64), or IP fragmentation (possible evasion). Check for source/destination IPs that are private (RFC 1918) when they should be public, or vice versa. Use Wireshark's 'Expert Info' to highlight malformed packets. For example, a packet with both SYN and RST flags set is invalid and suggests a crafted packet.
Inspect payloads for malicious content
Follow TCP streams to reassemble application-layer data. Look for encoded payloads (base64, hex) in HTTP requests, SQL injection patterns (`' OR 1=1--`), or shellcode (NOP sleds like `\x90\x90`). For encrypted traffic, analyze handshake metadata: TLS version, cipher suites, certificate details. Anomalies like self-signed certificates or unusual SNI (Server Name Indication) can indicate C2. Use Wireshark's 'Follow TCP Stream' or 'Export Objects' (HTTP, SMB) to extract files.
Correlate findings and document evidence
Cross-reference suspicious packets with other data sources: firewall logs, IDS alerts, endpoint logs. Use timestamps to sequence events. For example, if a port scan is detected, check if any subsequent connections to those ports occurred. Document the packet details: source/destination IPs, ports, protocol, payload snippet, and timestamps. Create a timeline of the attack. This step is crucial for incident response and reporting. Tools like Wireshark's 'IO Graph' can visualize traffic patterns, and 'Conversations' list all flows.
Enterprise Scenario 1: Malware C2 Detection
A large enterprise with 10,000 endpoints uses a SIEM that aggregates netflow data. The SOC notices a host communicating with an external IP on TCP 4444 every 60 seconds with 64-byte payloads. Netflow shows the flow but not the payload. The analyst initiates full packet capture on the host's switch port using a network TAP. After capturing 30 minutes of traffic, the analyst filters for ip.addr == <external IP> and tcp.port == 4444. Following the TCP stream reveals a base64-encoded string that decodes to a beacon message. The analyst confirms C2 activity and isolates the host. The capture is saved as evidence. Performance consideration: at 1 Gbps, the TAP must handle line rate without dropping packets. The capture server uses a 10 TB RAID array, retaining 7 days of full packet capture.
Enterprise Scenario 2: Data Exfiltration via DNS
A financial institution monitors outbound DNS queries. The anomaly detection system flags a workstation making thousands of TXT record queries to a rarely visited domain. Each query has a subdomain like 'aGVsbG8=.example.com' (base64 of 'hello'). The analyst uses Wireshark to capture DNS traffic from the workstation. The filter dns.qry.type == TXT shows queries with subdomains up to 255 characters. Decoding the base64 reveals exfiltrated credit card numbers. The analyst also notices the queries are spaced 5 seconds apart to avoid rate limiting. The DNS server logs confirm the queries. The analyst blocks the domain and quarantines the workstation. The capture is used to estimate data loss: 250 bytes per query × 2000 queries = 500 KB exfiltrated.
Enterprise Scenario 3: Lateral Movement Detection
A healthcare organization uses network segmentation. The IDS detects an SMB brute force from an internal workstation to a file server. The analyst captures traffic on the file server's segment. Filtering for smb2 and ntlmssp reveals multiple failed login attempts followed by a successful one. The analyst follows the SMB stream to see the exact commands executed after login, including a PowerShell script that downloads a payload. The capture shows the script content, which includes an IP address of a C2 server. The analyst uses this to block the IP at the firewall and initiate incident response. The capture also reveals the attacker's source IP, which is traced back to a compromised contractor's laptop.
What CS0-003 Tests on Network Traffic Analysis
The exam objective 3.1 (Given a scenario, analyze data as part of security monitoring activities) heavily emphasizes traffic analysis. You must be able to interpret packet captures, identify malicious patterns, and recommend actions. Specific sub-objectives include:
Analyze indicators of compromise (IoCs) from network traffic.
Identify common attack vectors (scanning, DoS, C2, exfiltration).
Use tools like tcpdump, Wireshark, and netstat.
Understand protocol headers and flag combinations.
Common Wrong Answers and Why Candidates Choose Them
Choosing 'NetFlow' as the best tool for payload analysis: NetFlow only provides metadata (IPs, ports, bytes). Candidates often confuse it with full packet capture. The correct answer is 'Wireshark' or 'tcpdump' for payload inspection.
Assuming a SYN-ACK packet indicates an established connection: SYN-ACK is part of the three-way handshake but does not mean the connection is complete. The connection is established only after the final ACK. Exam questions may show a SYN-ACK without a preceding SYN (spoofed) or without a following ACK (half-open).
Interpreting a high number of SYN packets as a DoS attack: Without seeing the source IPs and timing, it could be a port scan. A SYN flood typically has spoofed source IPs and no subsequent ACKs. The exam expects you to differentiate between scan and flood.
Believing encrypted traffic cannot be analyzed: While payload is hidden, metadata (TLS handshake, certificate, SNI, packet sizes, timing) can reveal malicious patterns. Exam may ask about detecting C2 over HTTPS.
Specific Numbers, Values, and Terms to Know
Default TCP window size: 65535 bytes (typical).
MTU: 1500 bytes (Ethernet).
TTL defaults: Windows 128, Linux 64, Cisco 255.
Common attack ports: 22 (SSH brute force), 445 (SMB exploits), 3389 (RDP brute force), 1433 (SQL injection), 8080 (proxy/HTTP alt).
TCP flag combinations: SYN scan (SYN only), FIN scan (FIN only), Xmas scan (FIN+PSH+URG), Null scan (no flags).
Wireshark display filter syntax: ip.src, tcp.dstport, http.request.uri.
Edge Cases and Exceptions
IPv6 traffic: Must use IPv6 filters (e.g., ipv6.addr). TTL replaced by Hop Limit.
VLAN tagging: 802.1Q adds 4 bytes; filter with vlan.id.
MPLS: Multiple labels; filter with mpls.label.
Encapsulated traffic: GRE, IPsec, VXLAN require decapsulation before analysis.
Fragmented packets: ID field and fragment offset; fragments may be reassembled by the tool but can be used for evasion.
How to Eliminate Wrong Answers
If a question asks for the 'best' tool for a specific task, eliminate options that don't provide the needed depth. For payload analysis, eliminate netflow and SIEM. For real-time capture, eliminate log analysis.
If a question shows a packet capture snippet, identify the protocol and flags. For example, a packet with only SYN flag is a scan; SYN+ACK is a response; RST indicates reset.
For scenario-based questions, map the symptoms to the attack: regular small packets = C2 beacon; large outbound = exfiltration; many connections to many ports = scanning.
Network traffic analysis involves capturing, filtering, and inspecting packets to detect security incidents.
The three-way TCP handshake is SYN, SYN-ACK, ACK; any deviation indicates scanning or spoofing.
Common C2 beacons are small, periodic packets to a fixed IP; use Wireshark IO Graph to visualize.
DNS tunneling uses TXT or NULL records with long subdomains; decode payload to reveal exfiltrated data.
Wireshark display filter syntax: ip.addr, tcp.port, http.request; capture filter syntax: 'tcp port 80'.
Always capture with promiscuous mode and consider SPAN/TAP for full visibility on switched networks.
Encrypted traffic still reveals metadata; analyze TLS handshake, SNI, and packet sizes.
tcpdump default buffer is 2 MB; increase with -B to prevent packet loss on busy links.
Port scans can be detected by multiple SYN packets without ACKs; use Wireshark's 'Conversations' to identify scanning hosts.
Legal authorization is required before capturing traffic; ensure proper consent and privacy safeguards.
These come up on the exam all the time. Here's how to tell them apart.
Full Packet Capture (FPC)
Captures entire packet including payload.
Provides complete forensic detail for incident response.
High storage requirements (~450 GB/day at 1 Gbps).
Requires dedicated capture infrastructure (TAPs, large storage).
Allows deep protocol analysis and payload inspection.
NetFlow/sFlow
Captures only metadata (IPs, ports, protocols, bytes, timestamps).
Low storage requirements (~1-5% of FPC).
Scalable for long-term retention and historical analysis.
Cannot inspect payload; cannot detect application-layer attacks.
Ideal for baselining and anomaly detection on volume.
Mistake
Wireshark can capture traffic on any network without special configuration.
Correct
Wireshark requires the capture interface to be in promiscuous mode to see all packets on a switched network. On a switched network, without port mirroring or a TAP, Wireshark only sees broadcast/multicast and packets destined to its own MAC. For full visibility, configure SPAN port or use a network TAP.
Mistake
A SYN packet without an ACK always means a port is open.
Correct
A SYN packet alone is a probe. The response determines port state: SYN+ACK = open; RST = closed; no response = filtered. The absence of ACK does not indicate open; it's the response that matters.
Mistake
Encrypted traffic (HTTPS) cannot be analyzed for security threats.
Correct
While payload is encrypted, metadata such as TLS version, cipher suites, certificate details, SNI, packet sizes, and timing can reveal malicious patterns. For example, a self-signed certificate or unusual SNI may indicate C2. Additionally, the handshake can be inspected for anomalies.
Mistake
tcpdump captures all packets by default without any loss.
Correct
tcpdump can drop packets if the capture buffer is too small or if the kernel cannot keep up. Use `-B` to increase buffer size (default 2 MB). On high-speed links, consider using PF_RING or dedicated hardware. Check for packet loss with `tcpdump -s 0 -i eth0 -w /dev/null &` and monitor 'packets dropped by kernel'.
Mistake
A high number of DNS queries from a single host is always malicious.
Correct
It could be legitimate due to misconfigured applications, browser prefetching, or a busy web server. Analysts must correlate with payload size, domain reputation, and query types. DNS tunneling typically uses TXT or NULL records with long subdomains and base64 encoding.
Reveal each answer, then mark whether you got it right. Score 60%+ to unlock the next chapter.
Use tcpdump in SSH session. Run `tcpdump -i eth0 -w capture.pcap -C 100 -W 10` to capture up to 10 files of 100 MB each. Then copy the pcap file to your local machine using `scp` and analyze with Wireshark. For streaming, use `tcpdump -i eth0 -w - | wireshark -k -i -` (requires X11 forwarding). Alternatively, use tshark with SSH.
A capture filter (BPF) is applied before packets are captured; it reduces the amount of data stored. Example: `tcp port 80`. A display filter is applied after capture; it hides packets from view but they remain in the file. Example: `http.request`. Use capture filters to limit volume on busy networks; use display filters for post-capture analysis.
Look for a large number of SYN packets to one or more ports from many different source IPs, with no corresponding SYN-ACK or ACK. The source IPs may be spoofed. Use Wireshark's IO Graph with filter `tcp.flags.syn==1 and tcp.flags.ack==0`. A sudden spike indicates a flood. Also check for retransmissions and RST packets.
A RST (reset) packet is used to abruptly terminate a TCP connection. It can be sent by either endpoint when an error occurs or when a connection is refused. In security analysis, a RST in response to a SYN indicates the port is closed. A RST after a SYN-ACK may indicate a port scan detection (e.g., by a firewall).
Even though payload is encrypted, you can examine the TLS handshake: look for unusual cipher suites (e.g., weak ciphers), self-signed certificates, or certificates issued to domains that don't match the SNI. Also analyze packet sizes and timing: C2 beacons often have consistent packet sizes and intervals. Use JA3/JA3S hashing to fingerprint TLS clients and servers.
Use dedicated capture hardware with specialized NICs (e.g., Intel XL710) and software like PF_RING or AF_PACKET. Apply capture filters to reduce volume. Use multiple capture threads or distributed capture. Consider using netflow for metadata and selective full packet capture based on triggers. Typical lossless capture at 10 Gbps requires tuned kernel parameters and large buffers.
Go to File → Export Objects → choose protocol (HTTP, SMB, etc.). Wireshark will list all files transferred. You can save them individually. Alternatively, use `foremost` or `binwalk` on the pcap file. For HTTP, use `wireshark -r capture.pcap --export-objects http,./extracted`.
You've just covered Network Traffic Analysis — now see how well it sticks with free CS0-003 practice questions. Full explanations included, no account needed.
Done with this chapter?