CloudWatch Metrics and Alarms

Amazon CloudWatch is a monitoring service for AWS resources and applications. It collects raw data in the form of metrics — time-ordered sets of data points. Each metric represents a variable to monitor, such as CPU utilization, network throughput, or disk I/O. Metrics are stored for up to 15 months, allowing historical analysis. An alarm watches a single metric or an expression based on multiple metrics over a specified time period, and performs one or more actions when the metric crosses a threshold. Alarms can trigger actions such as sending a notification via Amazon SNS, performing an EC2 Auto Scaling action, or invoking a Lambda function.

Metrics are organized into namespaces — containers for metrics from different sources. AWS services publish metrics to predefined namespaces like AWS/EC2, AWS/Lambda, AWS/RDS. You can also publish custom metrics to namespaces like Custom/MyApp. Each metric is uniquely identified by its namespace, metric name, and zero or more dimensions. Dimensions are key-value pairs that further refine the metric. For example, the CPUUtilization metric in the AWS/EC2 namespace can have a dimension InstanceId to isolate data for a specific EC2 instance.

When a metric is emitted, it includes: - Timestamp: The time when the data point was collected (default: current time). - Value: The numeric value of the metric. - Unit: The unit of measurement (e.g., Percent, Count, Bytes). - Dimensions: Optional key-value pairs.

CloudWatch stores metrics as a series of data points. You can retrieve statistics such as Average, Sum, Minimum, Maximum, Sample Count, and percentile values (e.g., p50, p90, p99). Statistics are computed over a specified period — the length of time to aggregate the data. The period can range from 1 second to 1 day (86400 seconds). For example, if you set a period of 300 seconds (5 minutes), CloudWatch aggregates all data points within each 5-minute interval into one data point representing the average, sum, etc.

Metrics have a resolution that determines the granularity at which data is stored: - Standard resolution: 1-minute granularity (data points aggregated every minute). - High resolution: 1-second granularity (data points stored at 1-second intervals). Custom metrics can be published with high resolution by setting the StorageResolution parameter to 1. High-resolution metrics are stored for 3 hours at 1-second resolution, then aggregated to 1-minute for 15 days, then to 5-minute for 63 days, and finally to 1-hour for 15 months.

An alarm watches a single metric or a metric math expression. The alarm evaluates the metric against a threshold over a specified number of evaluation periods. The alarm can be in one of three states: - OK: The metric is within the defined threshold. - ALARM: The metric has crossed and stayed beyond the threshold for the specified number of consecutive periods. - INSUFFICIENT_DATA: Not enough data is available to evaluate the alarm (e.g., newly created metric, missing data points).

#### Alarm Components - Metric: The metric to monitor (e.g., AWS/EC2 CPUUtilization). - Statistic: The statistic to evaluate (e.g., Average, Sum, Maximum). - Period: The length of time in seconds to evaluate the metric. - Evaluation Periods: The number of most recent periods to consider. - Datapoints to Alarm: The number of data points within the evaluation periods that must be breaching to trigger ALARM state. - Threshold: The value that triggers the alarm. - Comparison Operator: e.g., GreaterThanOrEqualToThreshold, LessThanThreshold. - Treat Missing Data: How to handle missing data points (e.g., breaching, notBreaching, ignore, missing). Default is missing which treats missing as not breaching. - Actions: Actions to take on state transitions (OK, ALARM, INSUFFICIENT_DATA). Common actions include SNS notifications, Auto Scaling policies, and EC2 actions (stop, terminate, reboot).

#### Alarm Evaluation Logic

The alarm evaluates the metric over the last N evaluation periods. The alarm transitions to ALARM if the metric breaches the threshold for at least DatapointsToAlarm out of the last EvaluationPeriods periods. For example, with EvaluationPeriods=3 and DatapointsToAlarm=2, the alarm goes to ALARM if 2 out of the last 3 periods are breaching. This provides tolerance for transient spikes.

You can create alarms based on expressions combining multiple metrics. Metric math supports arithmetic operations (+, -, *, /), statistical functions (AVG, SUM, MIN, MAX, COUNT), and time series functions (METRICS, FILL). For example, you could create an expression (m1 + m2) / 2 to average two metrics. The expression result is a time series that the alarm evaluates.

Dashboards allow you to create customizable views of metrics and alarms. You can add widgets (graphs, text, alarms) and arrange them. Dashboards can be shared across accounts or made public. They support automatic refresh and can display metrics from multiple regions.

CloudWatch Logs can be used to extract metrics from log data using metric filters. A metric filter defines a pattern to look for in log events and creates a metric when a match is found. You can then create alarms on those custom metrics. For example, you can count the number of ERROR log entries and alarm if errors exceed a threshold.

The CloudWatch agent collects metrics and logs from EC2 instances and on-premises servers. It can collect system metrics (CPU, memory, disk, network) and custom metrics (e.g., application performance). The agent can also collect logs and send them to CloudWatch Logs. It supports both Linux and Windows.

CloudWatch has a free tier: 10 custom metrics, 10 alarms, 1 million API requests, and 5 GB of log ingestion per month. Beyond that, you pay per metric, per alarm, per API request, and per GB of logs ingested and stored. High-resolution metrics cost more than standard resolution.

To put custom metric data:

aws cloudwatch put-metric-data --namespace "Custom/MyApp" --metric-data "[{\"MetricName\":\"Requests\",\"Value\":10,\"Unit\":\"Count\",\"Timestamp\":\"2025-04-01T12:00:00Z\"}]"

To create an alarm:

aws cloudwatch put-metric-alarm --alarm-name "HighCPU" --alarm-description "Alarm when CPU exceeds 80%" --metric-name CPUUtilization --namespace AWS/EC2 --statistic Average --period 300 --evaluation-periods 2 --threshold 80 --comparison-operator GreaterThanThreshold --dimensions Name=InstanceId,Value=i-1234567890abcdef0 --alarm-actions arn:aws:sns:us-east-1:123456789012:my-topic

To describe alarms:

aws cloudwatch describe-alarms --alarm-name-prefix "HighCPU"

A large e-commerce company runs a fleet of EC2 instances behind an Auto Scaling group. They need to automatically scale out when CPU utilization exceeds 70% for 5 minutes and scale in when it drops below 30% for 10 minutes. They create two CloudWatch alarms: HighCPU (Average CPUUtilization > 70 for 2 consecutive 5-minute periods) and LowCPU (Average CPUUtilization < 30 for 2 consecutive 5-minute periods). The HighCPU alarm triggers a scale-out policy that adds 2 instances; LowCPU triggers a scale-in policy that removes 1 instance. They also set up an SNS notification to the operations team for both alarms. In production, they notice that during flash sales, CPU spikes briefly above 70% but not for 10 minutes, so the alarm doesn't trigger unnecessarily. However, they also experience a slow ramp-up where CPU averages 68% for 10 minutes but never hits 70% — the alarm doesn't trigger, causing performance degradation. They adjust the threshold to 65% to catch this. Misconfiguration: setting EvaluationPeriods too low (e.g., 1) causes flapping (alarm toggles rapidly), which can trigger unnecessary scaling actions. Best practice: use at least 2 evaluation periods to dampen noise.

A financial services company runs a microservices application on ECS Fargate. They publish custom metrics for request latency, error count, and order volume. They create a composite alarm that triggers if ErrorCount > 10 AND Latency > 500ms for 5 minutes. This alarm invokes a Lambda function that runs a diagnostic script and posts a message to a Slack channel via SNS. They also create a dashboard showing real-time metrics for each microservice. In production, they encounter a problem where missing data points (e.g., due to a service restart) cause the alarm to go into INSUFFICIENT_DATA state, which they had set to ignore. They change TreatMissingData to notBreaching to avoid false alarms. They also learn that high-resolution metrics (1-second) are useful for debugging but cost more, so they only use them for critical services.

A startup uses AWS Budgets to set cost thresholds, but they also create CloudWatch alarms on the Billing metric (namespace: AWS/Billing, metric: EstimatedCharges). They set an alarm to notify when estimated charges exceed $1,000 for the month. This requires enabling billing alerts in the account settings. They also create a custom metric that tracks cost per service by using cost allocation tags and publishing metrics via a Lambda function. Misconfiguration: forgetting to enable billing alerts results in no data for the billing metric, causing the alarm to remain in INSUFFICIENT_DATA. They also set the period to 6 hours (21600 seconds) to avoid daily fluctuations. Common mistake: using Sum instead of Maximum — Sum accumulates charges over the period, which can exceed the threshold early in the month; Maximum gives the current estimated total.