Designing Monitoring Solutions

Azure Monitor is the central platform for collecting, analyzing, and acting on telemetry from Azure resources and on-premises environments. It replaces the need for disparate monitoring tools by providing a unified solution. The primary goals are: - Detect and diagnose issues across applications and infrastructure. - Understand performance and resource utilization. - Proactively respond to critical conditions via alerts and automated actions. - Optimize resource usage and costs. - Meet compliance and auditing requirements.

Azure Monitor consists of several key components: - Data Sources: Metrics (numerical values at regular intervals) and Logs (events with timestamps). Metrics are stored in a time-series database and support near real-time alerting. Logs include activity logs, resource logs, and custom logs. - Data Collection: Agents (Azure Monitor agent, Log Analytics agent, Diagnostic extension) and APIs collect data from Azure resources, VMs, containers, and on-premises machines. The Azure Monitor agent is the preferred agent, replacing the older Log Analytics and Diagnostics agents. - Data Storage: Logs are stored in Log Analytics workspaces (each workspace has a unique workspace ID and key). Metrics are stored in the Azure Monitor metrics database. - Analysis and Visualization: Log Analytics queries (using Kusto Query Language, KQL) allow deep analysis. Dashboards, workbooks, and Power BI provide visualization. - Alerts and Actions: Alert rules evaluate metrics or logs at a defined frequency. When triggered, they fire an action group (email, SMS, webhook, ITSM, automation runbook, etc.). - Insights: Specialized monitoring experiences for specific services (Application Insights, Container Insights, VM Insights, Network Insights).

Using Azure CLI:

# Create Log Analytics workspace
az monitor log-analytics workspace create --resource-group rg-monitoring --workspace-name laworkspace1 --location eastus --sku PerGB2018

# Create metric alert for high CPU on VM
az monitor metrics alert create --name cpuAlert --resource-group rg-monitoring --scopes /subscriptions/.../virtualMachines/vm1 --condition "avg Percentage CPU > 80" --window-size 5m --evaluation-frequency 1m --action /subscriptions/.../actionGroups/ag1

# Query logs
az monitor log-analytics query --workspace laworkspace1 --query "Perf | where CounterName == '% Processor Time' | where Computer == 'vm1' | summarize avg(CounterValue) by bin(TimeGenerated, 5m)"

Using PowerShell:

New-AzOperationalInsightsWorkspace -ResourceGroupName rg-monitoring -Name laworkspace1 -Location eastus -Sku PerGB2018

Add-AzMetricAlertRuleV2 -Name cpuAlert -ResourceGroupName rg-monitoring -WindowSize (New-TimeSpan -Minutes 5) -EvaluationFrequency (New-TimeSpan -Minutes 1) -TargetResourceId /subscriptions/.../virtualMachines/vm1 -Condition (New-AzMetricAlertRuleV2Criteria -MetricName "Percentage CPU" -TimeAggregation Average -Operator GreaterThan -Threshold 80)

A large e-commerce company runs thousands of VMs and hundreds of databases across multiple Azure regions. They need real-time monitoring of application performance and infrastructure health. They deploy Azure Monitor with a Log Analytics workspace per region to reduce latency and comply with data residency requirements. They use VM Insights to collect performance counters and maps of dependencies. They create metric alerts for CPU, memory, and disk I/O with dynamic thresholds to reduce false positives. For the database tier, they use SQL Insights to monitor query performance and deadlocks. They set up log alerts for failed logins and suspicious SQL injections. All critical alerts trigger an action group that sends a webhook to their incident management system (ServiceNow) and pages the on-call engineer. They also use Azure Workbooks to create a single-pane-of-glass dashboard for executives showing uptime, latency, and error rates. A common issue they faced was alert fatigue due to poorly tuned thresholds; they resolved it by using dynamic thresholds and tuning evaluation windows.

A bank must comply with strict regulatory requirements for logging and monitoring. They use Azure Policy to enforce diagnostic settings on all resources, ensuring all platform logs are sent to a central Log Analytics workspace. They set retention to 365 days for security logs and 90 days for performance logs. They use data export to Azure Storage for long-term archival (7 years). They create custom log queries to detect unauthorized access attempts and send alerts to the security operations center (SOC). They also use Azure Monitor's private link to ensure logs never traverse the public internet. A common misconfiguration they encountered was not enabling diagnostic settings on new resources; they automated remediation using Azure Policy and Azure Automation runbooks. They also use Log Analytics workspace insights to monitor workspace health and data ingestion rates.

A SaaS provider hosts applications for multiple customers in a single Azure subscription. They need to monitor each tenant's resource usage separately. They use Log Analytics workspaces per tenant (or use a single workspace with resource-level permissions). They configure diagnostic settings on each tenant's resources to send logs to the appropriate workspace. They use role-based access control (RBAC) to restrict access to logs per tenant. They create custom workbooks that show tenant-specific metrics and logs. They use metric alerts with dimensions to alert on high resource usage per tenant. A challenge they face is cost allocation; they use Azure Monitor's usage and estimated costs feature to attribute costs to each tenant. They also use Azure Cost Management to track spending per tenant.