Data Collection Rules (DCR) and Azure Monitor Agent

Azure Monitor Agent (AMA) is the next-generation data collection agent for Azure Monitor. It replaces the legacy Log Analytics Agent (also known as Microsoft Monitoring Agent or MMA) and the Azure Diagnostic Extension (WAD/LAD). AMA was designed to address several limitations of the legacy agents: - Centralized configuration: Legacy agents required per-machine configuration via VM extensions or custom scripts. AMA uses Data Collection Rules (DCRs) that are defined centrally in Azure and can be applied to multiple machines. - Single agent for multiple data sources: AMA can collect Windows Event Logs, Syslog, Performance counters, and custom logs using a single agent, whereas legacy required separate agents for different sources. - Data transformation: AMA supports data transformations using KQL before the data reaches its destination, reducing storage costs and enabling pre-processing. - Network isolation: AMA supports Azure Monitor Private Link Scopes, allowing data to be sent to Log Analytics workspaces over a private endpoint without traversing the public internet. - Security and compliance: AMA uses managed identities for authentication, eliminating the need for workspace keys stored on the VM.

AMA is the recommended agent for all new deployments, and Microsoft is deprecating the legacy agents. The AZ-104 exam focuses on AMA and DCRs, so you must be proficient in their configuration.

AMA runs as a Windows service (Windows) or a daemon (Linux). It communicates with Azure Monitor using the Azure Monitor Ingestion API over HTTPS (port 443). The agent uses a system-assigned or user-assigned managed identity to authenticate to Azure. Once authenticated, it retrieves its assigned DCRs from the DCR service. The agent then collects data according to the rules defined in the DCR and sends the data to the specified destinations (usually a Log Analytics workspace or Azure Storage).

The data flow is as follows: 1. Agent startup: AMA starts and registers with Azure, obtaining its managed identity token. 2. DCR retrieval: The agent queries the DCR service for any DCRs assigned to it (via data collection rule associations or DCRA). 3. Data collection: The agent reads the DCR's 'dataSources' section to determine what to collect (e.g., specific event logs, performance counters, syslog facilities). 4. Transformation: If the DCR includes a 'transformations' section, the agent applies KQL transformations to the data before sending. 5. Ingestion: The transformed data is sent to the destination specified in the 'destinations' section of the DCR.

A DCR is a JSON document with the following structure:

{
  "properties": {
    "dataSources": [
      {
        "kind": "extension",
        "name": "myDataSource",
        "configuration": {
          "dataSources": { ... }
        }
      }
    ],
    "destinations": {
      "logAnalytics": [
        {
          "workspaceResourceId": "/subscriptions/.../workspaces/MyWorkspace",
          "workspaceId": "...",
          "name": "myDestination"
        }
      ]
    },
    "dataFlows": [
      {
        "streams": [ "Microsoft-InsightsMetrics" ],
        "destinations": [ "myDestination" ],
        "transformKql": "source | where ..."
      }
    ]
  }
}

A DCR alone does not apply to any VM. You must create a Data Collection Rule Association (DCRA) to link a DCR to a VM or VMSS. The DCRA is a separate Azure resource that points to the DCR and the VM. You can associate multiple DCRs to a single VM, and a single DCR can be associated to many VMs. The association is done at the VM resource level. For example:

az monitor data-collection rule association create --name "myAssociation" --rule-id "/subscriptions/.../dataCollectionRules/myDCR" --resource "/subscriptions/.../virtualMachines/myVM"

When a VM has multiple DCRs, the agent merges the data sources from all DCRs. If there are conflicting data sources (e.g., two DCRs define the same performance counter with different sample rates), the behavior is as follows: For performance counters, the highest sample rate (lowest interval) wins. For event logs, the most inclusive level wins (e.g., one DCR collects 'Error' and another 'Warning', the agent collects both 'Error' and 'Warning'). This merging logic is important for exam scenarios.

One of the most powerful features of DCRs is the ability to transform data using Kusto Query Language (KQL) before it reaches the destination. The transformation is specified in the 'transformKql' property of a data flow. The KQL query acts on the incoming data stream and can filter rows, add or remove columns, or compute new values. For example:

source
| where SeverityLevel >= 4
| project TimeGenerated, Computer, EventID, Message
| extend EventCategory = "Critical"

This transformation filters events with severity level 4 or higher, projects only specific columns, and adds a new column 'EventCategory'. The transformation runs on the agent before sending data, reducing the amount of data ingested and stored. However, transformations are limited to the capabilities of KQL within the agent context; complex joins or external data lookups are not supported.

AMA can be installed via:

After installation, the agent automatically registers with Azure and retrieves its DCRs. You can verify the agent status using:

az vm extension list --vm-name myVM --resource-group myRG --query "[?name=='AzureMonitorWindowsAgent']"

Create a DCR:

az monitor data-collection rule create --resource-group myRG --name myDCR --location eastus --rule-file rule.json

Associate DCR to VM:

az monitor data-collection rule association create --name myAssoc --rule-id $(az monitor data-collection rule show -g myRG -n myDCR --query id -o tsv) --resource /subscriptions/.../virtualMachines/myVM

List DCRs:

az monitor data-collection rule list --resource-group myRG

Show DCR:

az monitor data-collection rule show --resource-group myRG --name myDCR

Delete DCR:

az monitor data-collection rule delete --resource-group myRG --name myDCR

Install AMA on Windows VM:

az vm extension set --vm-name myVM --resource-group myRG --name AzureMonitorWindowsAgent --publisher Microsoft.Azure.Monitor --version 1.0

Check agent status:

# On Windows VM
Get-Service -Name "AzureMonitorAgent"

A large enterprise has 500 Azure VMs spread across multiple subscriptions and regions. They need to collect Windows event logs (Security, Application) from all VMs, but only critical events from the Security log to reduce storage costs. They also need performance counters (CPU, Memory) from a subset of production VMs. Using AMA and DCRs, they create two DCRs: one for all VMs collecting Application events (all levels) and Security events (Critical only), and another for production VMs adding performance counters. They use Azure Policy to deploy AMA to all VMs and associate the first DCR. For production VMs, they additionally associate the second DCR via a separate policy. The agent merges the rules: all VMs get Application events, but only production VMs also collect performance counters. The enterprise saves costs by not collecting verbose Security events from non-production VMs. They also add a transformation to filter out events from specific event IDs known to be noise. Common pitfall: forgetting to assign the 'Monitoring Metrics Publisher' role on the Log Analytics workspace for performance counters, causing missing data.

A company has 200 on-premises servers running Linux that they want to monitor using Azure Monitor. They onboard these servers via Azure Arc, which registers them as Azure resources. They install AMA on each server using a script that also enables the managed identity. They create a DCR that collects syslog (auth, cron, daemon) with severity 'err' and 'warning', and performance counters for CPU and disk. They associate the DCR with the Arc-enabled server resources. Data flows to a central Log Analytics workspace. They use a transformation to mask IP addresses in syslog messages for privacy compliance. The challenge: network connectivity—the on-premises servers must be able to reach Azure Monitor endpoints. They use a firewall rule to allow outbound HTTPS to *.ods.opinsights.azure.com and *.oms.opinsights.azure.com. They also configure the agent to use a proxy if needed. Without proper network configuration, data collection fails silently.

A SaaS company ingests massive amounts of IIS logs from 1000 web servers into Log Analytics. The storage costs are high. They create a DCR with a transformation that filters out successful requests (HTTP 200) and keeps only errors and slow requests. The transformation uses KQL: 'source | where scStatus >= 400 or timeTaken > 5000'. This reduces data volume by 80%. They also route the filtered data to a cheaper Azure Storage account for long-term archival, using a second destination in the DCR. The performance impact: the transformation runs on each agent, consuming CPU. They monitor agent CPU usage and adjust the sample rate for performance counters accordingly. Misconfiguration: if the transformation syntax is wrong, the entire data stream fails, and no data is ingested. They test transformations in the Log Analytics demo environment before applying to production DCRs.