Azure Monitor Diagnostic Settings for Security

Azure Monitor Diagnostic Settings are a configuration feature on each Azure resource that defines which logs and metrics are collected and where they are sent. They are the primary mechanism for exporting resource-level operational data—such as platform logs, activity logs, and performance counters—to destinations for analysis, archiving, or streaming. Without diagnostic settings, most Azure resources generate logs internally but do not make them accessible for external querying or alerting. Diagnostic settings bridge that gap by enabling a continuous flow of data from the resource to one or more of three destination types: a Log Analytics workspace (for querying and alerting), an Azure Storage account (for long-term archival and low-cost storage), or an Azure Event Hubs (for real-time streaming to third-party SIEM tools or custom applications).

From a security perspective, diagnostic settings are the foundation of detection and investigation. For example, Azure Key Vault logs all access attempts to secrets and keys, but those logs are only available if you configure a diagnostic setting to send them to a Log Analytics workspace. Without this, you cannot detect unauthorized access attempts or audit key usage. Similarly, Azure SQL Database audit logs are only accessible via diagnostic settings. The AZ-500 exam emphasizes that diagnostic settings are required to enable security monitoring for resources like Key Vault, SQL Database, Network Security Groups (NSGs), and Azure Firewall. Without them, these resources are effectively invisible to your security operations center (SOC).

When you enable a diagnostic setting on a resource, Azure Monitor begins collecting data from the resource's platform-level logs and metrics. The process is as follows:

4. Destination ingestion: - Log Analytics workspace: Data is ingested into a table specific to the resource type (e.g., KeyVaultAuditLogs, SQLSecurityAuditEvents). Ingestion is near real-time, typically within 2-5 minutes. - Storage account: Data is written to blobs in a container named insights-logs-<category> with a path structure including date and time. Data is flushed every few minutes or when a certain size threshold is reached. - Event Hubs: Data is streamed as events to an Event Hubs namespace, where external systems can consume it. Latency is typically under 1 minute.

You can configure diagnostic settings via the Azure portal, PowerShell, Azure CLI, or ARM templates. The portal interface is straightforward: navigate to the resource, click "Diagnostic settings" under "Monitoring," then add a setting. You must specify a name, select the destinations, and choose the log categories and metrics.

Azure CLI example:

az monitor diagnostic-settings create \
    --resource /subscriptions/{sub}/resourceGroups/{rg}/providers/Microsoft.KeyVault/vaults/{vault} \
    --name "KeyVaultDiagnostics" \
    --workspace /subscriptions/{sub}/resourceGroups/{rg}/providers/Microsoft.OperationalInsights/workspaces/{workspace} \
    --logs '[
        {
            "category": "AuditEvent",
            "enabled": true,
            "retentionPolicy": {
                "days": 90,
                "enabled": true
            }
        }
    ]' \
    --metrics '[
        {
            "category": "AllMetrics",
            "enabled": true,
            "retentionPolicy": {
                "days": 90,
                "enabled": true
            }
        }
    ]'

Verification: To verify that logs are flowing, query the Log Analytics workspace:

KeyVaultAuditLogs
| where TimeGenerated > ago(1h)
| count

If the count is zero, check the diagnostic setting configuration and ensure the resource is generating logs.

For IaC, you can define diagnostic settings in an ARM template. Example snippet:

{
    "type": "Microsoft.KeyVault/vaults/providers/diagnosticSettings",
    "apiVersion": "2017-05-01-preview",
    "name": "[concat(parameters('vaultName'), '/Microsoft.Insights/service')]",
    "properties": {
        "workspaceId": "[parameters('workspaceId')]",
        "logs": [
            {
                "category": "AuditEvent",
                "enabled": true,
                "retentionPolicy": {
                    "days": 90,
                    "enabled": true
                }
            }
        ],
        "metrics": [
            {
                "category": "AllMetrics",
                "enabled": true,
                "retentionPolicy": {
                    "days": 90,
                    "enabled": true
                }
            }
        ]
    }
}

By mastering diagnostic settings, you ensure that your Azure environment is observable and that security events are captured for analysis and alerting—a core responsibility of a security engineer.

A financial services company uses Azure Key Vault to store customer encryption keys and secrets. Their security operations center (SOC) needs to monitor all access attempts to detect anomalies such as repeated failed access or access from unexpected locations. The solution is to configure diagnostic settings on each Key Vault to send AuditEvent logs to a central Log Analytics workspace. The workspace is integrated with Azure Sentinel for advanced detection rules. In production, the company has 50+ Key Vaults across multiple subscriptions. They use Azure Policy to enforce that every Key Vault has a diagnostic setting with a specific Log Analytics workspace and 365-day retention. A common issue is that developers sometimes create Key Vaults outside the policy scope, leading to blind spots. To mitigate, they run a weekly Azure Resource Graph query to find Key Vaults without diagnostic settings. Performance considerations: Ingestion of AuditEvent logs is low volume (a few MB per day per vault), so cost is minimal. However, if they mistakenly enable all metrics (which are high-volume), costs can spike. They use a separate diagnostic setting for metrics to a storage account to save costs.

A large e-commerce company uses Network Security Groups (NSGs) to control traffic to their web servers. To detect network anomalies like port scanning or data exfiltration, they need NSG flow logs. They configure diagnostic settings on each NSG to send flow logs (category: 'NetworkSecurityGroupFlowEvent') to a Log Analytics workspace. The logs are then analyzed using KQL queries to identify unusual traffic patterns. In production, they have hundreds of NSGs. One challenge is that enabling flow logs on all NSGs can generate terabytes of data per day, leading to high ingestion costs. They mitigate by setting a retention of only 7 days in Log Analytics and archiving to storage for 90 days. A common misconfiguration is forgetting to enable the flow log category—only the default categories are selected. The exam tests that NSG flow logs require an explicit diagnostic setting; they are not enabled by default. Another pitfall is that flow logs are only generated for NSGs attached to subnets or network interfaces; unattached NSGs produce no logs.

A healthcare organization is required to stream audit logs from Azure SQL Database to their on-premises SIEM (Splunk) for real-time compliance monitoring. They configure diagnostic settings on each SQL database to stream to an Azure Event Hubs namespace. The Event Hubs is then consumed by Splunk via the Azure Event Hubs add-on. In production, they have 20 SQL databases. They must ensure the Event Hubs throughput units are sufficient to handle peak log rates (e.g., 1 MB/s). A common issue is that the diagnostic setting is created but the event hub is not accessible due to network restrictions (e.g., firewall or private endpoints). They also need to set the event hub retention to at least 1 day to handle temporary Splunk outages. The exam tests that streaming to Event Hubs is the only way to get near-real-time logs for external SIEMs, and that you must configure the event hub's consumer group appropriately to avoid conflicts.