This chapter covers Shadow IT Discovery using Microsoft Defender for Cloud Apps (MCAS), a critical skill for the SC-200 exam. Shadow IT refers to cloud applications used by employees without explicit IT approval, posing security and compliance risks. MCAS provides tools to discover, analyze, and control these apps. Expect 10-15% of exam questions to touch on Shadow IT discovery, cloud app catalog, risk scores, and integration with Defender XDR. Mastering this topic is essential for passing the SC-200.
Jump to a section
Imagine a large corporation as a private library with a strict catalog system. Every book (cloud app) must be approved by the librarian (IT department) before it can be used. Employees are researchers who need specific books to do their work. However, some researchers start bringing in their own books from outside—perhaps a popular novel or a specialized journal not in the catalog. They keep these books in their personal lockers, and the librarian has no record of them. This is Shadow IT. Microsoft Defender for Cloud Apps (MCAS) acts as a hidden surveillance system. It monitors all the books being read in the library—not just the cataloged ones. It can detect when a researcher is using an unauthorized book, log it, and alert the librarian. The librarian can then decide to block the book, allow it with restrictions (e.g., only in certain reading rooms), or add it to the catalog. MCAS uses traffic logs from network proxies (like Microsoft Defender for Endpoint or third-party proxies), examines the app's reputation using the Cloud App Catalog (which has over 31,000 apps), and correlates user behavior to identify risky apps. It doesn't just see the book cover; it reads the content to check for malware or data exfiltration. The library's security is enforced through policies that can automatically block or sanction apps based on risk scores—like a librarian who not only bans a book but also issues a warning to the researcher.
What is Shadow IT and Why Does It Matter?
Shadow IT encompasses any cloud application, service, or device used within an organization without formal IT approval. Employees often adopt these apps for convenience, productivity, or to bypass restrictive policies. Common examples include unsanctioned file-sharing services (e.g., Dropbox, Google Drive personal accounts), communication tools (e.g., Slack free tier, WhatsApp), and productivity suites (e.g., Trello, Notion). The risk is significant: data leakage, compliance violations (GDPR, HIPAA), malware introduction, and inability to enforce data governance. MCAS addresses this by providing a unified visibility and control plane.
MCAS Shadow IT Discovery Mechanism
MCAS discovers Shadow IT through two primary methods: log-based discovery and API-based integration. Log-based discovery ingests traffic logs from network proxies, firewalls, or endpoint agents. The logs must contain specific fields: timestamp, user IP, user agent, source/destination IP, port, and bytes transferred. MCAS parses these logs, normalizes them, and correlates activity to cloud apps using its Cloud App Catalog. The catalog contains signatures for over 31,000 cloud apps, including known IP ranges, domain patterns, and HTTP characteristics. For each app, MCAS assigns a risk score (1-10) based on factors like encryption, data residency, security standards, and legal agreements. Apps with a score below 5 are considered high-risk.
Step-by-Step Data Processing
Log Collection: Logs are collected from supported sources: Microsoft Defender for Endpoint (MDE) network discovery logs, Defender for Identity, Azure AD sign-in logs, and third-party proxies (e.g., Zscaler, Palo Alto). Logs are uploaded to MCAS via the log collector (a virtual appliance) or directly via API.
Log Parsing: MCAS parses each log entry, extracting user identity (via IP-to-user mapping from Active Directory or Azure AD), app identifier (via destination IP/domain), and transaction details.
App Identification: The destination IP or domain is matched against the Cloud App Catalog. If a match is found, the app is identified; otherwise, it may be flagged as unknown.
Risk Assessment: Each identified app is scored based on ~90 attributes, including SOC 2 certification, ISO 27001, PCI DSS, data encryption at rest and in transit, and user authentication requirements.
Policy Enforcement: Administrators create discovery policies (e.g., block apps with risk score < 5) or app discovery policies (e.g., alert when a new app is discovered). Policies can auto-generate alerts, block traffic (via integration with Defender for Endpoint or proxy), or sanction the app for official use.
Key Components and Defaults
Cloud App Catalog: Over 31,000 apps; each has a risk score (1-10). Default high-risk threshold: score < 5. Apps are categorized by domain (e.g., CRM, HR, collaboration).
Log Collector: A Docker-based virtual appliance deployed on-premises or in Azure. It receives logs via Syslog, FTP, or SFTP and forwards them to MCAS. Default port: 443 for outbound communication.
Supported Log Formats: Common Event Format (CEF), Web Proxy (W3C), and custom JSON. MCAS auto-detects format based on source.
Data Retention: Shadow IT discovery reports retain data for 90 days (default). Can be extended with Azure Data Lake integration.
App Discovery Policies: Two types: (1) Discovery policy — triggers on app attributes (e.g., risk score, category), (2) App discovery policy — triggers on new app detection or usage anomalies.
Integration with Defender XDR
MCAS Shadow IT discovery is tightly integrated with Microsoft Defender XDR. Alerts generated by discovery policies appear in the Microsoft 365 Defender portal (security.microsoft.com). They can be correlated with other signals: user sign-in risk (Azure AD Identity Protection), device risk (Defender for Endpoint), and threat intelligence. This enables automated response using playbooks (e.g., block user access if they use a high-risk app). The integration also allows MCAS to apply session controls (via Defender for Cloud Apps Conditional Access App Control) to sanctioned apps, enabling real-time monitoring and control.
Configuration Steps for Discovery
To enable Shadow IT discovery: 1. In Microsoft 365 Defender portal, go to Cloud Apps > Discover > Create snapshot report or Continuous report. 2. For continuous reports, deploy the log collector on a server (Windows or Linux) using the provided script. The script pulls a Docker container that runs the collector. 3. Configure your proxy or firewall to forward logs to the collector’s IP and port (default 514 for Syslog). 4. In MCAS, define App discovery policies to set thresholds (e.g., block apps with risk score < 5). 5. Review discovered apps in the Discovered apps page. Filter by risk score, category, or usage volume.
Verification Commands
On the log collector server, verify the service status:
sudo docker psShould show a container named mcaslogcollector. Check logs:
sudo docker logs mcaslogcollector --tail 50In MCAS portal, navigate to Discover > Log collectors to see status (Connected, Disconnected, Error).
Common Issues and Troubleshooting
Logs not appearing: Check time synchronization (NTP). Logs must have timestamps within 24 hours of collection. Verify firewall rules (outbound HTTPS to *.cloudappsecurity.com).
User not identified: Ensure IP-to-user mapping is configured via Azure AD or Active Directory. If using a proxy that strips user info, enable user identification in MCAS settings.
App not recognized: The app may not be in the catalog. Submit a request to add it via the MCAS portal. Alternatively, create a custom app signature.
Deploy Log Collector
Deploy the MCAS log collector on a server (Windows Server 2016+ or Ubuntu 18.04+). The collector is a Docker container. Run the deployment script from MCAS portal: Settings > Cloud Discovery > Log collectors > Add log collector. The script installs Docker (if not present) and pulls the container. After deployment, configure your network proxy or firewall to forward logs to the collector's IP and port (default Syslog UDP 514). Verify the container is running with 'docker ps'.
Configure Log Source
In MCAS, define the log source: Settings > Cloud Discovery > Log collectors > select the collector > Configure. Choose the log format (e.g., Palo Alto, Zscaler, Defender for Endpoint). Provide the source IP of the proxy. Optionally, set user identification method (e.g., Active Directory lookup). The collector will parse incoming logs based on the selected format. For Defender for Endpoint, no log collector is needed; enable network discovery in MDE.
Define Discovery Policies
Create policies to govern discovered apps. Go to Cloud Apps > Policies > Policy management > Discovery policies. Two types: (1) App discovery policy: triggers on new app detection or usage anomalies. (2) Discovery policy: filters apps by attributes (risk score, category, etc.). For example, create a policy that blocks all apps with risk score < 5 and alerts security team. Policies can also auto-sanction apps (allow with monitoring) or unsanction them (block).
Review Discovered Apps
In MCAS, navigate to Cloud Apps > Discover > Discovered apps. This page shows all apps identified from logs. Columns: App name, risk score, category, users, transactions, traffic volume. Filter by risk score to see high-risk apps. Drill down into an app to see detailed usage: top users, IP addresses, data upload/download. From here, you can sanction or unsanction an app, which updates the app's status in the catalog and applies any associated policies.
Generate Snapshot Report
For ad-hoc discovery without continuous logs, upload a log file manually. Go to Cloud Apps > Discover > Create snapshot report. Upload a log file (e.g., from a proxy) in supported format (CEF, W3C, etc.). MCAS parses it and generates a report. This is useful for one-time assessments. The report is available for 30 days. You can compare multiple snapshot reports over time.
Enterprise Scenario 1: Global Retail Chain
A multinational retailer with 50,000 employees uses Microsoft 365 and a third-party proxy (Zscaler). They want to discover unsanctioned cloud apps used by employees. They deploy the MCAS log collector in three regional data centers (US, EU, Asia). Logs from Zscaler are forwarded to each collector via Syslog. Within a week, MCAS discovers 1,200 distinct cloud apps, including 150 high-risk apps (risk score < 5). The security team sanctions 50 apps (e.g., Slack, Trello) after evaluating their security posture. They block 100 high-risk apps (e.g., personal file-sharing sites) using MCAS integration with Zscaler to enforce blocking. A discovery policy alerts when a user accesses a blocked app more than 10 times in a day. This reduces data leakage incidents by 40%.
Enterprise Scenario 2: Financial Services Firm
A bank with strict compliance requirements (SOX, PCI DSS) uses Defender for Endpoint and Azure AD. They enable network discovery in MDE, which sends logs directly to MCAS. They create a discovery policy that flags any app not in the 'approved' list (maintained as a custom app tag). When an employee uses an unapproved app, an alert is generated in Microsoft 365 Defender. The SOC uses a playbook to automatically block the app on the user's device via Defender for Endpoint. They also set up a weekly report of top discovered apps for the compliance team. Performance: processing 10 million log entries per day with < 5 minute latency.
Common Pitfalls
Misconfigured Log Collector: Firewall blocking outbound HTTPS to *.cloudappsecurity.com causes logs to not reach MCAS. Ensure the collector server can reach the required endpoints (port 443).
Incorrect Log Format: If the proxy sends logs in a format not matching the collector configuration, parsing fails. Verify the log format and adjust collector settings.
User Identification Failure: Without proper IP-to-user mapping, MCAS cannot attribute usage to specific users. This requires Active Directory or Azure AD sync. In large environments, ensure the log collector has network access to domain controllers for LDAP queries.
Overblocking: Aggressive policies that block all low-risk apps can hinder productivity. A phased approach is recommended: first discover, then sanction critical apps, then block high-risk ones.
SC-200 Exam Focus on Shadow IT Discovery (Objective Code: 1.4)
The exam tests your ability to configure and interpret Shadow IT discovery in MCAS. Key areas: - Log Sources: Know which log sources are supported (Defender for Endpoint, Azure AD sign-in logs, third-party proxies). The exam may ask which source is best for a scenario (e.g., real-time vs. snapshot). - Cloud App Catalog: Understand that MCAS uses a catalog of over 31,000 apps with risk scores 1-10. The default high-risk threshold is < 5. Questions may ask about the number of apps or risk score ranges. - Discovery Policies: Be able to differentiate between discovery policies (filter by attributes) and app discovery policies (trigger on new app). Know that policies can sanction, unsanction, or block apps. - Log Collector: Know that it is a Docker container, deployed on-premises or in Azure. It forwards logs to MCAS via HTTPS (port 443). The exam may test troubleshooting: if logs are not appearing, check time sync or firewall rules. - Snapshot vs. Continuous Reports: Snapshot reports are for ad-hoc analysis (upload a log file). Continuous reports require a log collector. Know that snapshot reports expire after 30 days.
Common Wrong Answers (Trap Patterns)
Choosing 'MCAS automatically blocks all high-risk apps': MCAS does not block automatically; you must create policies to block. The default is only discovery.
Selecting 'Cloud App Catalog contains 10,000 apps': The actual number is over 31,000. Expect this exact number in a question.
Believing that MCAS can discover apps without any logs: MCAS requires traffic logs; it cannot discover apps purely from cloud API integration (like Office 365). API integration is for sanctioned apps.
Confusing 'sanction' with 'allow': Sanctioned apps are approved and monitored; unsanctioned apps are blocked. The exam may ask about the effect of sanctioning.
Edge Cases
Unidentified Users: If logs contain only IPs without user mapping, MCAS will attribute usage to 'unknown user'. The exam may ask how to resolve this: configure IP-to-user mapping via Azure AD or AD.
Custom Apps: If an app is not in the catalog, you can create a custom app in MCAS. The exam might test that you can manually add an app.
Log Collector High Availability: For large environments, deploy multiple collectors with load balancing. The exam may ask about scaling.
MCAS discovers Shadow IT using traffic logs from proxies, firewalls, or Defender for Endpoint.
The Cloud App Catalog contains over 31,000 apps, each with a risk score from 1 to 10.
Default high-risk threshold is risk score < 5.
Discovery policies can sanction (allow with monitoring) or unsanction (block) apps.
Log collector is a Docker container that forwards logs to MCAS via HTTPS (port 443).
Snapshot reports are for ad-hoc discovery; continuous reports require a log collector.
IP-to-user mapping is essential for attributing usage to specific users.
Integration with Defender XDR allows correlation with other security signals.
Alerts from discovery policies appear in Microsoft 365 Defender portal.
Custom apps can be added if not in the catalog.
These come up on the exam all the time. Here's how to tell them apart.
Snapshot Report
Manual upload of log file (CEF, W3C, etc.)
One-time analysis; data expires after 30 days
No log collector required
Best for quick assessments or audits
Cannot enforce real-time blocking
Continuous Report
Automatic log ingestion via log collector
Continuous monitoring with up-to-date data
Requires deploying a Docker-based log collector
Supports real-time policy enforcement (block, alert)
Data retained for 90 days (default)
Mistake
MCAS can discover Shadow IT without any log sources.
Correct
MCAS requires traffic logs from network proxies, firewalls, or endpoint agents (like Defender for Endpoint). Without logs, it cannot discover cloud app usage. API integration only monitors sanctioned apps.
Mistake
The Cloud App Catalog has exactly 15,000 apps.
Correct
The catalog contains over 31,000 cloud apps. The exact number is important for the exam; questions may reference this figure.
Mistake
Snapshots reports are continuous and update in real-time.
Correct
Snapshot reports are manual uploads of log files for ad-hoc analysis. They are not continuous and expire after 30 days. Continuous reports require a log collector.
Mistake
Sanctioned apps are automatically blocked.
Correct
Sanctioned apps are approved and monitored; they are not blocked. Unsanctioned apps are blocked. Sanctioning allows the app with possible session controls.
Mistake
The log collector can be deployed without Docker.
Correct
The MCAS log collector is a Docker container. Docker must be installed on the server. The deployment script handles this automatically.
Reveal each answer, then mark whether you got it right. Score 60%+ to unlock the next chapter.
To enable Shadow IT discovery, first set up a log source. For continuous discovery, deploy the MCAS log collector (a Docker container) on a server. Configure your proxy or firewall to forward logs to the collector. Then, in MCAS, create a continuous report and define discovery policies. For ad-hoc discovery, use a snapshot report by uploading a log file. Ensure logs contain user IP, timestamp, and destination IP.
MCAS supports Common Event Format (CEF), Web Proxy (W3C), and custom JSON formats. It also supports specific formats from vendors like Palo Alto, Zscaler, Blue Coat, and Check Point. When configuring the log collector, you select the format that matches your log source. MCAS auto-detects the format for some sources.
MCAS can block apps only if you create policies that enforce blocking. This requires integration with a proxy or Defender for Endpoint. For example, you can create a discovery policy that sets the app as 'unsanctioned', and if you have a proxy that can enforce URL filtering (e.g., Zscaler), MCAS can push the block list. Without such integration, MCAS can only alert.
Sanctioning an app means you approve it for use, and MCAS will monitor its usage. You can apply session controls (like Conditional Access App Control) to sanctioned apps. Unsanctioning means you block the app. Users attempting to access an unsanctioned app will be blocked (if enforcement is configured) or alerted.
By default, continuous discovery data is retained for 90 days. Snapshot reports are retained for 30 days. You can extend retention by integrating with Azure Data Lake or exporting data to Azure Storage. The exam may ask about the default retention period.
If the log collector cannot communicate with MCAS (outbound HTTPS to *.cloudappsecurity.com), logs are queued locally. The collector will attempt to send them periodically. If the outage persists, logs may be lost. Check firewall rules and ensure the collector server has internet access. Monitor collector health in MCAS portal under Discover > Log collectors.
Yes, if you use Microsoft Defender for Endpoint (MDE). Enable network discovery in MDE, which sends network logs directly to MCAS. No log collector is needed. This is a common scenario for organizations that don't have a third-party proxy. However, the coverage may be limited to devices managed by MDE.
You've just covered Shadow IT Discovery with MCAS — now see how well it sticks with free SC-200 practice questions. Full explanations included, no account needed.
Done with this chapter?