Azure Monitor and Log Analytics

Azure Monitor is the comprehensive monitoring service for Azure. It collects, analyzes, and acts on telemetry from Azure resources, on-premises environments, and other clouds. It provides a single pane of glass for metrics, logs, and alerts.

Without a centralized monitoring solution, administrators would have to manually check each resource's health, leading to delayed detection of issues. Azure Monitor automates this by aggregating data, setting thresholds, and triggering actions. It enables proactive management, cost optimization, and security compliance.

Azure Monitor collects two primary types of data: - Metrics: Numerical values representing resource health (e.g., CPU percentage, disk IOPS). They are lightweight and near real-time (1-minute granularity). Metrics are stored for 93 days by default. - Logs: Text-based events (e.g., error logs, audit trails). Logs are sent to a Log Analytics workspace where they can be queried using Kusto Query Language (KQL). Retention is configurable from 30 to 730 days (or 2 years with pay-as-you-go).

Data flows as follows: 1. Collection: Resources emit telemetry via diagnostic settings (for Azure resources) or agents (for VMs). The Azure Monitor agent collects data from VMs, including performance counters and events. 2. Storage: Metrics are stored in the Azure Monitor metrics database (time-series). Logs are stored in Log Analytics workspaces (structured tables). 3. Analysis: You can view metrics in Metrics Explorer, query logs in Log Analytics, or use pre-built dashboards. 4. Alerting: Alert rules evaluate data at regular intervals (e.g., every 5 minutes) and trigger actions (e.g., email, webhook, ITSM) when conditions are met. 5. Visualization: Data can be visualized in Azure Workbooks or exported to Power BI.

To create a Log Analytics workspace using Azure CLI:

az monitor log-analytics workspace create --resource-group myRG --workspace-name myWorkspace

To create a diagnostic setting:

az monitor diagnostic-settings create --resource <resource-id> --name myDiagSetting --workspace <workspace-id> --metrics '[{"category": "AllMetrics", "enabled": true}]' --logs '[{"category": "AuditLog", "enabled": true}]'

To query logs using KQL in Log Analytics:

AzureActivity
| where TimeGenerated > ago(1h)
| where OperationName == "Delete Virtual Machine"
| project TimeGenerated, Caller, Resource

To create a metric alert:

az monitor metrics alert create --name "High CPU" --resource-group myRG --scaleset <vmss-id> --condition "avg Percentage CPU > 80" --action-group <action-group-id>

Scenario 1: Monitoring a Production Web Application

A company runs a multi-tier web application on Azure VMs and App Services. They use Azure Monitor to collect metrics (CPU, memory, request rate) and logs (application errors, IIS logs). They create metric alerts for high CPU (>90%) and log alerts for error spikes (e.g., >100 HTTP 500 errors in 5 minutes). Action groups send emails to the DevOps team and trigger an Azure Automation runbook that scales out the VMSS. The Log Analytics workspace retains data for 90 days for troubleshooting. Common issues: misconfigured diagnostic settings (missing categories), incorrect KQL queries returning no data, and alert fatigue from overly sensitive thresholds.

Scenario 2: Security and Compliance Monitoring

A financial services firm must audit all administrative actions. They enable diagnostic settings on all Azure resources to send Activity Logs to a Log Analytics workspace. They create activity log alerts for resource deletions and role assignments. They also collect security events from VMs (Windows Event ID 4625 for failed logins) using the Azure Monitor agent. They use workbooks to create compliance dashboards showing unauthorized access attempts. Performance considerations: high-volume logs can increase costs; they set retention to 30 days for verbose logs and 1 year for security logs. Misconfiguration: forgetting to enable 'AuditLogs' category in diagnostic settings leads to missing data.

Scenario 3: Hybrid Monitoring with On-Premises Servers

An enterprise has on-premises servers running Windows and Linux. They use the Azure Monitor agent to send data to Azure Log Analytics (via Azure Arc or direct internet). They create custom metrics from log queries and set alerts for disk space < 10%. They use Azure workbooks to combine on-prem and cloud metrics. Scale considerations: each agent generates ~1-2 GB per month; they plan workspace capacity accordingly. Common pitfalls: firewall blocks outbound HTTPS to Azure, agent not configured with correct workspace key, and time zone mismatches in logs.