This chapter covers Azure Diagnostic Settings and Log Routing, a core monitoring concept tested heavily on the AZ-104 exam. You will learn how to configure diagnostic settings to route platform logs and metrics to multiple destinations, including Log Analytics workspaces, Azure Storage, and Event Hubs. Expect approximately 10-15% of exam questions to touch on this topic, often in scenario-based questions requiring you to select the correct destination for compliance, cost, or real-time analysis needs.
Jump to a section
Imagine Azure Diagnostic Settings as a large postal sorting office. Every Azure resource (like a VM, web app, or database) is a sender that produces letters (logs and metrics). Each letter has a destination requirement: some must go to a storage archive (Azure Storage), some to a real-time monitoring dashboard (Log Analytics), and some to a third-party partner (Event Hubs). The sorting office is the diagnostic setting you configure. You define a rule: "For every letter from this sender, create three copies and route one to each destination." The sorting office doesn't inspect the content; it just duplicates and routes based on the category (e.g., "AuditLog" or "PerformanceMetrics"). If you change the rule, the sorting office immediately updates its routing table. Importantly, the sender (resource) must exist before you can set up the sorting office—you can't pre-configure routing for a resource that doesn't exist. Also, if you want to change destinations later, you update the sorting rule, not the letters themselves. This analogy mirrors Azure Diagnostic Settings: you define a set of logs and metrics categories to stream to one or more destinations, and the platform handles the duplication and routing continuously. Misconfiguring a destination (e.g., providing a wrong storage account name) is like giving the sorting office a wrong address—letters pile up or get lost.
What are Diagnostic Settings?
Azure Diagnostic Settings are a configuration on each Azure resource that defines which platform logs and metrics to collect and where to send them. They are not a separate service but a feature of the Azure Monitor platform. Every Azure resource that emits logs or metrics has a 'Diagnostic settings' blade under the 'Monitoring' section. The core purpose is to enable streaming of telemetry data to external destinations for analysis, archiving, or integration.
Why Diagnostic Settings Exist
Without diagnostic settings, Azure resources generate logs and metrics internally but do not persist them beyond a short retention period (e.g., 30 days for platform metrics). For compliance, auditing, or advanced querying, you need to export this data. Diagnostic settings allow you to:
Send data to a Log Analytics workspace for querying with KQL (Kusto Query Language) and creating alerts.
Archive data to Azure Storage for long-term retention at low cost.
Stream data to Azure Event Hubs for real-time processing or ingestion into third-party SIEM tools.
How It Works Internally
When you enable a diagnostic setting on a resource, Azure Monitor creates a pipeline that continuously captures the specified log categories and metrics. The pipeline: 1. Collects data from the resource's built-in logging subsystem. 2. Applies any configured category filters (e.g., only 'Administrative' logs, not 'Audit'). 3. Duplicates the data stream for each enabled destination (up to 5 destinations per setting). 4. Transforms the data into the target format (e.g., JSON for Storage, tables for Log Analytics). 5. Sends the data to the destination endpoint.
The data flow is near real-time, with latency typically under 5 minutes for Log Analytics and under 10 minutes for Storage. Event Hubs delivery is within seconds.
Key Components
- Categories: Each resource type exposes specific log categories and metrics. For example, a virtual machine exposes 'Administrative', 'Security', 'ServiceHealth', 'Alert', 'Recommendation', 'Policy', and 'Autoscale' logs. Metrics include CPU, memory, disk I/O, and network throughput. - Destinations: - Log Analytics Workspace: Requires a workspace ID and shared key (or managed identity). Data is ingested into tables named after the resource type (e.g., 'AzureDiagnostics' for most resources, or resource-specific tables like 'VMConnection'). - Storage Account: Data is stored in blobs under a container named 'insights-logs-<category>' with a path structure like 'resourceId=/subscriptions/.../y=<YYYY>/m=<MM>/d=<DD>/h=<HH>/m=<mm>/PT1H.json'. The retention is controlled by the storage account's lifecycle management policy. - Event Hubs: Requires an Event Hubs namespace and event hub name. Data is sent as JSON events. The namespace must be in the same region as the resource (cross-region streaming is not supported). - Retention: Diagnostic settings themselves do not have a retention setting. Retention is managed at the destination (e.g., Log Analytics workspace data retention, Storage lifecycle policies).
Configuration Options
You can configure diagnostic settings via: - Azure Portal: Navigate to the resource, under 'Monitoring' > 'Diagnostic settings' > 'Add diagnostic setting'. Check the categories and select destinations. - PowerShell:
New-AzDiagnosticSetting -ResourceId <resourceId> -Name 'mySetting' -Category 'Administrative' -MetricCategory 'AllMetrics' -WorkspaceId <workspaceId> -StorageAccountId <storageAccountId> -EventHubAuthorizationRuleId <ruleId> -EventHubName <eventHubName>Azure CLI:
az monitor diagnostic-settings create --resource <resourceId> --name 'mySetting' --logs '["category": "Administrative", "enabled": true]' --metrics '["category": "AllMetrics", "enabled": true]' --workspace <workspaceId> --storage-account <storageAccountId> --event-hub <eventHubName> --event-hub-rule <ruleId>ARM templates: Define a resource of type 'Microsoft.Insights/diagnosticSettings' with properties for logs, metrics, and destinations.
Limitations and Defaults
A single diagnostic setting can have up to 5 destinations (any combination of Log Analytics, Storage, and Event Hubs).
You can create up to 5 diagnostic settings per resource.
The storage account must be in the same region as the resource. Log Analytics workspace and Event Hubs namespace must also be in the same region.
Destination resources must exist before configuring the diagnostic setting.
Data sent to Storage is in JSON format, each blob is a collection of records.
For Log Analytics, data is ingested into the 'AzureDiagnostics' table for most resources, but some resource types have dedicated tables (e.g., 'AzureActivity', 'AzureMetrics').
Metrics sent to Log Analytics are stored in the 'AzureMetrics' table.
Interaction with Related Technologies
Azure Policy: You can enforce diagnostic settings using Azure Policy. For example, a policy can automatically deploy a diagnostic setting to all virtual machines in a subscription, sending logs to a central Log Analytics workspace.
Azure Monitor Alerts: Alerts can be based on log queries from Log Analytics workspaces that are populated via diagnostic settings.
Azure Sentinel: Diagnostic settings are the primary method to ingest Azure resource logs into Azure Sentinel (a SIEM) by sending logs to a Log Analytics workspace that is connected to Sentinel.
Azure Storage Lifecycle Management: You can define rules to move archived logs to cooler tiers or delete them after a specified period.
Event Hubs Capture: You can enable Event Hubs Capture to automatically write events to Azure Storage for long-term retention, complementing diagnostic settings.
Performance and Scale Considerations
High-volume resources (e.g., Azure Firewall, Application Gateway) generate large amounts of log data. Sending all logs to multiple destinations can increase costs significantly.
Use category filtering to send only necessary logs. For example, send only 'Administrative' logs to Log Analytics and all logs to Storage for compliance.
For very high throughput, consider using Event Hubs with partitions to scale ingestion.
Latency: Storage delivery is typically under 10 minutes, Log Analytics under 5 minutes, Event Hubs under 2 minutes.
Common Configurations
Compliance Archiving: Send all logs and metrics to a storage account with a retention policy of 7 years.
Real-time Monitoring: Send specific categories to Log Analytics for alerting and dashboards.
SIEM Integration: Send all security-related logs to an Event Hubs namespace that feeds into a third-party SIEM.
Verification Commands
Azure CLI:
az monitor diagnostic-settings list --resource <resourceId>PowerShell:
Get-AzDiagnosticSetting -ResourceId <resourceId>REST API: GET https://management.azure.com/{resourceId}/providers/Microsoft.Insights/diagnosticSettings/{name}?api-version=2021-05-01-preview
Troubleshooting
Data not appearing in Log Analytics: Verify the workspace ID is correct, the agent is not needed (diagnostic settings are agentless), and the data may take up to 5 minutes to appear.
Storage blobs not created: Check that the storage account exists, is in the same region, and has the correct permissions (e.g., no firewall blocking Azure services).
Event Hubs not receiving: Ensure the Event Hubs namespace is in the same region and the authorization rule has 'Send' permission.
Multiple settings overlapping: If you have multiple diagnostic settings on the same resource, data may be duplicated. This is intentional for sending to different destinations with different filters.
Exam Tip
The AZ-104 exam often asks: "Which destination should you use for real-time analysis vs. long-term archiving?" Remember: Event Hubs for real-time, Storage for archiving, Log Analytics for querying and alerting. Another common question: "How many destinations can a single diagnostic setting have?" Answer: 5.
Identify the Resource
First, identify the Azure resource for which you want to configure diagnostic settings. This could be a virtual machine, Azure SQL database, key vault, or any resource that supports diagnostic settings. In the Azure portal, navigate to the resource and locate the 'Diagnostic settings' option under the 'Monitoring' section. Note that you cannot configure diagnostic settings for resource groups, subscriptions, or management groups directly; those require Azure Activity Log diagnostic settings, which are configured at the subscription level.
Select Log Categories and Metrics
Click 'Add diagnostic setting' to open the configuration blade. Here you see a list of available log categories and metrics for that resource type. For example, a virtual machine has categories like 'Administrative', 'Security', 'ServiceHealth', etc. Metrics are listed under 'AllMetrics'. You can select individual categories or 'AllMetrics'. You must enable at least one category or metric to save the setting. The exam tests that you know which categories are available for specific resources (e.g., Azure Key Vault has 'AuditEvent' logs).
Choose Destination Details
After selecting categories, choose one or more destinations. You can send data to a Log Analytics workspace (select workspace from dropdown), archive to a storage account (select storage account and optionally set retention days), or stream to an event hub (select Event Hubs namespace and event hub name, and provide an authorization rule with Send permission). You can also send to a partner solution via Event Hubs. Each destination requires the resource to exist in the same region as the source resource. You can select up to 5 destinations per setting.
Configure Retention (Storage Only)
If you choose to archive to a storage account, you can optionally set a retention policy in days for each log category. The retention policy applies to the storage account's lifecycle management. For example, setting retention to 365 days means blobs older than 365 days are automatically deleted. This is not available for Log Analytics or Event Hubs destinations; retention for those is managed at the destination resource level (e.g., Log Analytics workspace data retention setting).
Review and Create
Give the diagnostic setting a name (must be unique within the resource). Review your selections and click 'Save'. The setting takes effect immediately. Data will start flowing to the destinations within minutes. You can edit or delete the setting at any time. Note that you can create up to 5 diagnostic settings per resource, each with its own combination of categories and destinations. This allows you to, for example, send security logs to Log Analytics for alerting and all logs to storage for compliance.
Enterprise Scenario 1: Centralized Logging and Alerting
A large enterprise with hundreds of VMs and PaaS services needs to centralize logs for security monitoring and operational alerting. They create a Log Analytics workspace in a central region. Using Azure Policy, they automatically deploy a diagnostic setting on every new VM and Azure SQL database, sending 'Administrative' and 'Security' logs to the central workspace. The operations team builds dashboards and alerts on top of this data. They also configure a second diagnostic setting on critical resources to send all logs to a storage account for 7-year compliance. The challenge is cost: sending all metrics and logs from high-volume resources like Azure Firewall can be expensive. They mitigate by filtering categories and using retention policies. Common misconfiguration: forgetting to enable the policy assignment, resulting in missing logs for new resources. They use Azure Monitor Workbooks to visualize log trends and set up alert rules for anomalies.
Enterprise Scenario 2: Real-time SIEM Integration
A financial services company uses a third-party SIEM (e.g., Splunk) for security analytics. They need real-time ingestion of Azure security logs. They configure diagnostic settings on all Azure resources to stream 'AuditEvent' and 'Security' logs to an Event Hubs namespace. The SIEM consumes the events from the Event Hubs using a custom connector. The Event Hubs namespace is configured with multiple partitions to handle high throughput. They also enable Event Hubs Capture to archive raw events to Azure Storage as a backup. A common pitfall is that the Event Hubs namespace must be in the same region as the resource; cross-region streaming is not supported. They also need to ensure the authorization rule has 'Send' permission and that the SIEM can handle the volume. They monitor the Event Hubs backlog and adjust partitions as needed.
Enterprise Scenario 3: Cost-Effective Long-Term Archiving
A healthcare organization is required by regulation to retain audit logs for 10 years. They configure diagnostic settings on all Azure resources to send all logs and metrics to a general-purpose v2 storage account with cool access tier and lifecycle management rules that move blobs to archive tier after 30 days and delete after 10 years. They do not send logs to Log Analytics for cost reasons. The challenge is managing the large number of blobs; they use Azure Storage Analytics to monitor blob counts. They also set up a second diagnostic setting on critical resources to send 'Administrative' logs to Log Analytics for operational monitoring. Common issues: forgetting to enable the diagnostic setting after resource creation, or accidentally deleting the storage account. They use Azure Policy to enforce deployment and Azure Resource Locks to prevent accidental deletion.
AZ-104 Exam Focus on Diagnostic Settings
The AZ-104 exam tests diagnostic settings under objective 'Monitor and maintain Azure resources' (15-20% of exam). Specific sub-objectives include:
Configure diagnostic settings for resources (5.1)
Configure monitoring for Azure resources (5.2)
Interpret monitoring data (5.3)
Common Wrong Answers and Traps
Choosing Log Analytics for real-time streaming: Candidates often think Log Analytics is real-time, but it has a latency of 2-5 minutes. The correct real-time destination is Event Hubs.
Selecting 'Send to Log Analytics' for archiving: While Log Analytics retains data, it is not cost-effective for long-term archiving. The correct destination for long-term, low-cost archiving is Azure Storage.
Thinking diagnostic settings can be applied at subscription level: Diagnostic settings are per-resource. For subscription-level logs (Activity Log), you use a separate diagnostic setting on the subscription, not on individual resources.
Assuming you can send to any region: All destinations must be in the same region as the source resource. Cross-region streaming is not supported.
Forgetting that you can have multiple diagnostic settings: Many candidates think one setting per resource is the limit, but you can have up to 5.
Specific Numbers and Terms
Maximum destinations per diagnostic setting: 5
Maximum diagnostic settings per resource: 5
Latency to Log Analytics: <5 minutes
Latency to Storage: <10 minutes
Latency to Event Hubs: <2 minutes
Retention: Not configurable in diagnostic setting; managed at destination.
Required permissions: Microsoft.Insights/diagnosticSettings/write at resource scope.
Edge Cases
Activity Log diagnostic settings: Configured at subscription level, not resource level. They can send to Log Analytics, Storage, or Event Hubs.
Azure Policy: Can automatically deploy diagnostic settings. The policy definition uses 'DeployIfNotExists' effect.
Resource types without diagnostic settings: Some resources like Azure Front Door (classic) do not support diagnostic settings; they use Azure Monitor resource logs differently.
Data format in Storage: JSON blobs, path: insights-logs-<category>/resourceId=.../y=.../m=.../d=.../h=.../m=.../PT1H.json.
How to Eliminate Wrong Answers
If the question mentions 'real-time' or 'sub-second', eliminate Log Analytics and Storage; choose Event Hubs.
If the question mentions 'long-term retention' or 'compliance', eliminate Log Analytics (too expensive) and Event Hubs (not designed for archiving); choose Storage.
If the question mentions 'querying' or 'alerts', eliminate Storage and Event Hubs; choose Log Analytics.
If the question mentions 'SIEM' or 'third-party integration', choose Event Hubs.
Always check region constraints: if destinations are in different regions, the configuration will fail.
Diagnostic settings route platform logs and metrics to up to 5 destinations per setting, with a maximum of 5 settings per resource.
Destinations must be in the same Azure region as the source resource.
Event Hubs provides real-time streaming (<2 sec latency); Log Analytics provides querying and alerting (<5 min); Storage provides long-term archiving (<10 min).
Diagnostic settings do not collect guest OS logs; use Azure Monitor Agent for that.
Retention is managed at the destination, not in the diagnostic setting itself.
Azure Policy can automatically deploy diagnostic settings to new resources using DeployIfNotExists effect.
Activity Log diagnostic settings are configured at the subscription level, not per resource.
Common exam trap: Choosing Log Analytics for real-time needs — correct answer is Event Hubs.
These come up on the exam all the time. Here's how to tell them apart.
Log Analytics Workspace
Best for querying and alerting with KQL.
Higher cost per GB ingested.
Data retention configurable at workspace level (30 days to 2 years).
Latency <5 minutes.
Data stored in tables for efficient querying.
Azure Storage Account
Best for long-term, low-cost archiving.
Lower storage cost, especially using cool/archive tiers.
Retention managed via lifecycle policies (days to years).
Latency <10 minutes.
Data stored as JSON blobs, not directly queryable.
Event Hubs
Best for real-time streaming (<2 seconds latency).
Designed for high-throughput event ingestion.
Integrates with third-party SIEM tools.
No built-in querying or alerting.
Requires separate consumer to process events.
Log Analytics Workspace
Best for querying and alerting (latency <5 minutes).
Not designed for sub-second streaming.
Built-in querying with KQL and alert rules.
Higher cost for high volume.
Data directly available for analysis.
Mistake
Diagnostic settings can send logs to any Log Analytics workspace regardless of region.
Correct
The Log Analytics workspace must be in the same region as the source resource. Cross-region streaming is not supported.
Mistake
You can only have one diagnostic setting per resource.
Correct
You can have up to 5 diagnostic settings per resource, each with different category selections and destinations.
Mistake
Diagnostic settings are used to collect guest OS logs from virtual machines.
Correct
Diagnostic settings collect platform logs and metrics, not guest OS logs. For guest OS logs, you need the Azure Monitor Agent or the legacy Log Analytics agent.
Mistake
Sending logs to Log Analytics is free.
Correct
Ingesting data into Log Analytics incurs costs based on data volume. Only the first 5 GB per month per workspace is free.
Mistake
You can configure diagnostic settings on a resource that doesn't exist yet.
Correct
The resource must exist before you can create a diagnostic setting. You cannot pre-configure routing for a future resource; use Azure Policy to automate deployment.
Reveal each answer, then mark whether you got it right. Score 60%+ to unlock the next chapter.
Yes, but you need to create separate diagnostic settings for each workspace. A single diagnostic setting can only send to one Log Analytics workspace. However, you can create up to 5 diagnostic settings per resource, each targeting a different workspace.
Diagnostic settings collect platform-level logs and metrics from Azure resources (e.g., VM host metrics, Azure SQL logs). Azure Monitor Agent collects guest OS-level data from inside VMs (e.g., Windows Event Logs, performance counters). They are complementary; you may need both for full visibility.
Retention is not set in the diagnostic setting itself. Instead, configure a lifecycle management policy on the storage account to automatically delete blobs after a specified number of days. For example, you can set a rule to delete blobs in the 'insights-logs-*' containers after 365 days.
No. The Event Hubs namespace must be in the same Azure region as the source resource. Cross-region streaming is not supported for diagnostic settings.
First, check that the workspace ID is correct and that the workspace is in the same region. Data can take up to 5 minutes to appear. Also verify that you selected at least one log category or metric. If using a firewall on the workspace, ensure the 'Allow Azure services' setting is enabled.
You can have up to 5 diagnostic settings per resource, regardless of resource type. Each setting can have its own combination of categories and destinations.
Yes. You can create a policy with the 'DeployIfNotExists' effect to automatically deploy a diagnostic setting to resources that match the policy scope. This is a common method to enforce centralized logging.
You've just covered Diagnostic Settings and Log Routing — now see how well it sticks with free AZ-104 practice questions. Full explanations included, no account needed.
Done with this chapter?