This chapter covers application whitelisting and control, a critical security control for preventing unauthorized software execution. For the SY0-701 exam, this maps to Domain 4.0: Security Operations, Objective 4.1: Given a scenario, apply common security techniques to computing resources. Application whitelisting is a foundational defense against malware, ransomware, and unauthorized software, and understanding its mechanisms, implementation, and limitations is essential for both the exam and real-world practice.
Jump to a section
Imagine a high-end nightclub that only allows access to a pre-approved VIP list. The bouncer at the door has a clipboard with the names of every person allowed inside. When someone arrives, the bouncer checks their ID against the list. If the person's name is on the list, they are granted entry; if not, they are turned away, even if they have a ticket from a friend or claim to be someone important. This is exactly how application whitelisting works: a system administrator creates a list of approved applications (the VIP list), and the whitelisting software acts as the bouncer, allowing only those applications to run. Any executable not on the list is blocked from executing, regardless of its source or apparent legitimacy. The bouncer does not rely on recognizing 'bad' people (like blacklisting), but instead only knows 'good' people. This approach is far more secure because it prevents unknown or malicious software from running by default. The bouncer also checks for tampering—if a VIP's name is slightly misspelled or the ID looks forged, entry is denied. Similarly, whitelisting often uses cryptographic hashes or digital signatures to ensure the application has not been modified. This mechanistic control eliminates entire classes of malware that rely on executing unauthorized code.
What Is Application Whitelisting?
Application whitelisting is a security technique that explicitly permits only pre-approved software to run on a system. This is the opposite of blacklisting, which blocks known malicious software. In a whitelisting model, everything is denied by default; only applications that have been explicitly authorized are allowed to execute. This approach is highly effective because it stops unknown malware—including zero-day exploits—from running, as they are not on the approved list.
How Application Whitelisting Works Mechanically
The process begins with the creation of a baseline of trusted applications. An administrator identifies all legitimate software that should be allowed on a system, such as the operating system components, business applications, and approved utilities. The whitelisting software then captures attributes of each allowed executable, which can include:
File path and name: The exact location and filename of the executable.
File hash: A cryptographic hash (e.g., SHA-256) of the executable file. This ensures that even if the file is renamed or moved, the hash remains the same. If the file is modified, the hash changes, and execution is blocked.
Digital signature: The publisher's certificate and signature. This allows the whitelist to trust all files signed by a specific publisher (e.g., Microsoft, Adobe) without needing individual hashes.
Publisher: The company that signed the executable.
Product name and version: Specific product and version strings embedded in the executable.
When a user attempts to run an executable, the whitelisting software intercepts the request (typically at the kernel level via a filter driver or security hook). It calculates the file's hash (or extracts its digital signature) and compares it against the whitelist. If a match is found, execution proceeds normally. If no match is found, the execution is blocked, and an event is logged. Some solutions allow temporary bypasses or require administrator approval for unknown software.
Key Components and Variants
Application whitelisting can be implemented through various tools and policies:
Windows AppLocker: Built into Windows Enterprise and Education editions. It can control executables, scripts, Windows Installer files, and packaged apps. Rules can be based on file path, publisher, or file hash.
Windows Defender Application Control (WDAC): A more advanced whitelisting solution for Windows 10/11 Enterprise and Server. It uses code integrity policies that can be enforced at the kernel level. WDAC can be configured using Group Policy or PowerShell.
Third-party solutions: Products like Carbon Black, CrowdStrike, McAfee Application Control, and Bit9 (now part of VMware Carbon Black) offer enterprise-grade whitelisting with centralized management.
Linux: Tools like fapolicyd (File Access Policy Daemon) provide similar functionality based on file hashes, paths, or digital signatures.
macOS: Gatekeeper and XProtect provide limited whitelisting by allowing only apps from the Mac App Store or identified developers.
How Attackers Bypass Application Whitelisting
Attackers continuously develop techniques to bypass whitelisting. Common methods include:
Living-off-the-land binaries (LOLBins): Using legitimate, whitelisted executables (like powershell.exe, wmic.exe, mshta.exe) to execute malicious code. Since the binary itself is whitelisted, it can run scripts or load malicious DLLs.
DLL side-loading: Placing a malicious DLL in the same directory as a whitelisted executable. The executable loads the malicious DLL because of its search order.
File path abuse: If whitelisting rules are based on file path (e.g., allow C:\Program Files\*), an attacker can write a malicious executable to that directory.
Renaming or moving files: If whitelisting is based on file name or path, an attacker can rename a malicious file to match a whitelisted name.
Using interpreted scripts: Scripts (PowerShell, VBScript, JavaScript) may not be checked if the whitelist only covers executables. Many whitelisting solutions now include script control.
Bypassing via kernel exploits: A privilege escalation exploit can disable or tamper with the whitelisting service.
Defensive Countermeasures
To counter these bypasses, administrators should:
Use publisher rules instead of path-based rules whenever possible. Publisher rules verify digital signatures and are harder to spoof.
Enable script control to restrict which scripts can run. AppLocker and WDAC can control PowerShell, VBScript, and other script hosts.
Implement DLL control to prevent untrusted DLLs from being loaded by whitelisted executables.
Use hash-based rules for critical executables to ensure integrity.
Regularly audit whitelist events to identify blocked attempts and adjust policies.
Combine whitelisting with other controls like least privilege, application sandboxing, and endpoint detection and response (EDR).
Real Commands and Tools
Creating an AppLocker rule via PowerShell:
New-AppLockerPolicy -RuleType Publisher -User Everyone -Path "C:\Windows\*" -Action AllowExporting AppLocker policy:
Get-AppLockerPolicy -Local | Export-AppLockerPolicy -Path C:\Policy.xmlEnabling WDAC via Group Policy: Navigate to Computer Configuration > Administrative Templates > System > Device Guard > Turn on Virtualization Based Security. Set to Enabled and select the code integrity policy.
Linux fapolicyd example rule:
# Allow all files signed by Red Hat
allow perm=execute trust=1Checking WDAC policy in Windows:
Get-CIPolicy -FilePath C:\Policy.xmlStandards and References
NIST SP 800-167: Guide to Application Whitelisting
CIS Controls: Control 2 (Inventory and Control of Software Assets)
Common Vulnerabilities and Exposures (CVEs): Bypass techniques are often documented; e.g., CVE-2019-0859 (Windows Defender Application Control bypass via PowerShell).
Identify Trusted Software Baseline
Begin by inventorying all legitimate applications required for business operations. This includes operating system files, productivity suites, security tools, and line-of-business applications. Use automated tools like SCCM, PDQ Inventory, or manual audits to capture file paths, hashes, and publishers. This baseline must be created from a known clean system to avoid including malware. Document the version numbers and update schedules. This step is critical because any omission will block necessary software, causing user disruption. For the exam, remember that whitelisting requires a thorough understanding of the environment's software needs.
Create Whitelisting Policy Rules
Based on the baseline, create rules using the whitelisting tool. For AppLocker, you can create rules by publisher, path, or hash. For example, allow all Microsoft-signed executables from `C:\Program Files\*`. Use the principle of least privilege: allow only what is necessary. Avoid overly permissive rules like allowing all files from `C:\*`. Test policies in audit mode first to identify potential blocks without affecting users. Audit mode logs what would be blocked. Review logs and adjust rules accordingly. This step is where many mistakes occur—overly broad rules reduce security, overly narrow rules break functionality.
Deploy Policy to Systems
Use Group Policy, Microsoft Intune, or a third-party management tool to deploy the whitelisting policy to target systems. For AppLocker, the policy is stored as an XML file and applied via Group Policy Object (GPO). For WDAC, policies can be deployed via GPO or PowerShell. Ensure the policy is applied to all relevant systems, including servers, workstations, and kiosks. Test on a small pilot group before full rollout. Monitor for errors using Event Viewer (Event ID 8000-8008 for AppLocker). This step requires careful scheduling to avoid disrupting business operations.
Enable Enforcement Mode
After successful testing in audit mode, switch the policy from Audit to Enforce. This will block any unapproved executables. Communicate the change to users and IT support staff. Prepare a process for handling false positives—users should have a method to request approval for new software. This step is often the most disruptive; common mistakes include forgetting to update the policy for new software updates (e.g., Windows patches) which can break essential functionality. Ensure there is a rollback plan if critical applications are blocked.
Monitor and Maintain the Whitelist
Continuously monitor whitelisting logs for blocked attempts, which may indicate malicious activity or legitimate software that needs approval. Regularly review and update the whitelist to accommodate software updates, new applications, and decommissioned software. Use automation where possible, such as allowing all files signed by a trusted publisher. Conduct periodic audits to ensure the whitelist remains accurate and no unauthorized applications have been inadvertently allowed. This ongoing process is essential for maintaining security and usability.
Scenario 1: Ransomware Outbreak at a Healthcare Organization A hospital's IT team receives alerts from their EDR about multiple systems attempting to execute a new unknown binary. The binary is a strain of ransomware that has not been seen before. Because the hospital has implemented application whitelisting using WDAC, the binary is blocked from executing on all endpoints. The security analyst reviews the WDAC audit logs and sees hundreds of blocked attempts from the same file hash. The analyst isolates the affected systems and uses the logs to identify the initial entry point (a phishing email). The correct response is to contain the infection, block the file hash globally, and update the whitelist if needed. A common mistake would be to assume the whitelist is sufficient and ignore the blocked attempts, but the logs should be investigated to prevent future breaches.
Scenario 2: Software Installation Request in a Financial Firm
A financial analyst needs to install a specialized data analysis tool. The firm uses AppLocker with strict publisher rules. The analyst submits a request to IT, who verifies the software's digital signature and adds a publisher rule for the vendor. The analyst then installs the software without issue. A common mistake is for IT to create a path-based rule allowing C:\Users\* to avoid frequent requests, which would weaken security. The correct approach is to maintain strict rules and use a formal approval process.
Scenario 3: Living-off-the-Land Attack Detection An attacker gains access to a workstation and attempts to use PowerShell to download and execute a malicious payload. The whitelisting policy allows PowerShell.exe (as it is a Microsoft-signed binary), but the script control feature of AppLocker is enabled. The script is blocked because it is unsigned and not in an allowed path. The security team receives an event ID 8004 (AppLocker blocked script). The analyst investigates the source of the script and identifies the compromised user account. The correct response is to reset the account credentials, scan for persistence, and review the script control policy. A common mistake is to disable script control because it blocks legitimate scripts, but this would leave the organization vulnerable to such attacks.
Common Mistake Across Scenarios: Relying solely on file path rules without verifying integrity. Attackers can easily write malicious executables to allowed paths. The correct practice is to use publisher or hash rules whenever possible.
What SY0-701 Tests on Application Whitelisting
The exam focuses on understanding the purpose, implementation, and limitations of application whitelisting. Specific sub-objectives include:
Differentiating between whitelisting and blacklisting.
Identifying appropriate use cases (e.g., kiosks, servers, high-security environments).
Recognizing common bypass techniques (e.g., LOLBins, DLL side-loading).
Understanding the role of file hashes, digital signatures, and publisher rules.
Knowing built-in Windows tools: AppLocker and Windows Defender Application Control (WDAC).
Most Common Wrong Answers and Why
"Application whitelisting is the same as antivirus." Candidates choose this because both prevent malware, but whitelisting allows only approved apps, while antivirus uses signatures to block known malware. Whitelisting stops unknown malware; antivirus may not.
"Whitelisting is based on file extension." This is incorrect because file extensions can be renamed. Whitelisting uses hashes, signatures, or paths.
"Application whitelisting can be bypassed by renaming a malicious file." This is true only if the whitelist uses file names. Proper whitelisting uses hashes or signatures, which are not affected by renaming.
"Whitelisting eliminates the need for patch management." This is false; whitelisting does not address vulnerabilities in allowed applications. Patching is still required.
Specific Terms and Acronyms
AppLocker: Windows feature for application control.
WDAC: Windows Defender Application Control.
LOLBin: Living-off-the-land binary, a legitimate tool used maliciously.
Code Integrity Policy: A WDAC policy that defines which binaries are allowed.
Publisher Rule: A rule based on the digital signature of the software publisher.
Hash Rule: A rule based on the cryptographic hash of a file.
Path Rule: A rule based on the file location (least secure).
Trick Questions
Scenario: "An organization wants to ensure only approved software runs. What should they implement?" Answer: Application whitelisting, not blacklisting. Blacklisting misses unknown malware.
Scenario: "A user cannot run a legitimate application after whitelisting is implemented. What is the most likely cause?" Answer: The application is not on the whitelist, or its hash changed due to an update.
Comparison: "Which method provides the strongest security?" Answer: Publisher or hash rules, not path rules.
Decision Rule for Scenario Questions
When asked about application whitelisting, eliminate answers that:
Rely on signatures or definitions (those are blacklisting/antivirus).
Suggest blocking based on behavior alone (that is behavior-based detection).
Assume whitelisting prevents all attacks (it does not prevent exploits within allowed apps).
Focus on file extensions or names as the primary control (these are weak).
Application whitelisting allows only pre-approved software to run; everything else is blocked by default.
Whitelisting is more secure than blacklisting because it stops unknown malware.
Common whitelisting attributes: file hash, digital signature, publisher, and file path.
Windows tools: AppLocker (Enterprise) and Windows Defender Application Control (WDAC).
Bypass techniques include LOLBins, DLL side-loading, and script abuse.
Use publisher or hash rules instead of path rules for stronger security.
Always test whitelisting policies in audit mode before enforcing.
Whitelisting does not replace patch management or other security controls.
NIST SP 800-167 provides guidance on implementing application whitelisting.
Monitor whitelisting logs for blocked attempts to detect attacks or false positives.
These come up on the exam all the time. Here's how to tell them apart.
Application Whitelisting
Allows only pre-approved applications to run; everything else is blocked.
Uses hashes, digital signatures, or publisher rules to define allowed apps.
Effective against unknown malware and zero-day exploits.
Requires upfront effort to create and maintain the whitelist.
Provides strong security but may cause usability issues if not properly managed.
Application Blacklisting
Blocks known malicious applications; everything else is allowed.
Uses signatures, heuristics, or behavior patterns to identify malware.
Ineffective against new or unknown malware until signatures are updated.
Lower initial overhead but requires constant updates to signature databases.
More user-friendly but less secure against targeted attacks.
AppLocker
Available in Windows Enterprise and Education editions.
Rules can be based on publisher, path, or hash.
Can be managed via Group Policy or Local Security Policy.
Supports audit mode and enforcement mode.
Does not require virtualization-based security (VBS).
Windows Defender Application Control (WDAC)
Available in Windows 10/11 Enterprise, Pro, and Server.
Uses code integrity policies that can be signed and enforced at kernel level.
Managed via Group Policy, PowerShell, or MDM.
Supports multiple policy levels (e.g., Windows, Microsoft, Signed).
Can leverage VBS for stronger protection against kernel-level bypasses.
Mistake
Application whitelisting is the same as antivirus software.
Correct
Antivirus uses blacklisting (signatures) to block known malware. Whitelisting allows only approved applications, blocking all others including unknown malware. They are complementary but different.
Mistake
Whitelisting is only for executables (.exe files).
Correct
Whitelisting can also control scripts (PowerShell, VBScript), DLLs, Windows Installer files, and even macros. Tools like AppLocker and WDAC support these file types.
Mistake
Once whitelisting is implemented, no malware can run.
Correct
Malware can still run if it uses a whitelisted binary (LOLBin) or exploits vulnerabilities in allowed applications. Whitelisting reduces the attack surface but does not eliminate all risks.
Mistake
Path-based rules are just as secure as hash-based rules.
Correct
Path-based rules are the weakest because an attacker can write a malicious file to an allowed path. Hash-based rules ensure the file's integrity, and publisher rules verify the source.
Mistake
Application whitelisting is too difficult to manage in large environments.
Correct
Modern tools like WDAC and AppLocker support automated rule creation based on trusted publishers and can be managed via Group Policy or MDM. With proper planning, it is feasible for enterprises.
Application whitelisting is a security control that explicitly permits only pre-approved applications to run on a system. It denies everything by default, whereas blacklisting blocks known malicious applications but allows everything else. Whitelisting is more secure against unknown threats but requires more upfront management. For the exam, remember that whitelisting uses hashes, signatures, or publisher rules to identify allowed apps, while blacklisting relies on signature databases. In a scenario, if the goal is to prevent any unauthorized software from running, choose whitelisting.
The two primary Windows features are AppLocker and Windows Defender Application Control (WDAC). AppLocker is available in Windows Enterprise and Education editions and can control executables, scripts, and installers. WDAC is available in Windows 10/11 Enterprise, Pro, and Server and provides kernel-level code integrity policies. Both can be managed via Group Policy. For the exam, know that WDAC is more advanced and can use virtualization-based security. The question might ask which tool is built into Windows Enterprise for application control—the answer is AppLocker.
Common bypass techniques include using living-off-the-land binaries (LOLBins) like PowerShell or WMIC to execute malicious code, DLL side-loading where a malicious DLL is placed in the same directory as a whitelisted executable, and abusing path-based rules by writing malware to an allowed directory. Script control bypasses are also common if scripts are not explicitly restricted. For the exam, you may be asked to identify a bypass method in a scenario. Remember that whitelisting does not prevent abuse of whitelisted tools.
Yes, application whitelisting can prevent many ransomware attacks by blocking the execution of the ransomware binary if it is not on the whitelist. However, if the ransomware uses a whitelisted application (e.g., a PowerShell script) or exploits a vulnerability in a whitelisted app, it may still run. Whitelisting is a strong defense but should be combined with other controls like patching, least privilege, and EDR. For the exam, understand that whitelisting is effective against file-based ransomware but not against fileless attacks that use legitimate tools.
A publisher rule allows all files digitally signed by a specific publisher (e.g., Microsoft) regardless of version or file name. A hash rule allows a specific file based on its cryptographic hash. Publisher rules are easier to maintain because they automatically allow updates from the same publisher, while hash rules require updating when a file changes. Hash rules are more precise but require more maintenance. For the exam, know that publisher rules are preferred for commonly updated software, while hash rules are used for critical static files.
Audit mode allows the whitelisting policy to log what would be blocked without actually blocking execution. This helps administrators identify potential false positives and adjust rules before enforcing the policy. It is a critical step in deployment to avoid disrupting users. For the exam, remember that audit mode is used for testing and should be switched to enforce mode only after verification.
Application whitelisting enforces least privilege by ensuring that only necessary applications are allowed to run. It prevents users from running unauthorized software that could introduce risk. This aligns with the security principle of granting only the minimum access required. For the exam, you may be asked to identify controls that implement least privilege; whitelisting is one such control.
You've just covered Application Whitelisting and Control — now see how well it sticks with free SY0-701 practice questions. Full explanations included, no account needed.
Done with this chapter?