CloudWatch Alarms: Composite and Metric Math

Composite alarms are a CloudWatch feature introduced to solve the problem of alarm noise and complex condition logic. Before composite alarms, you could only create simple threshold-based alarms on a single metric or a math expression. If you needed to trigger an action only when multiple conditions were true simultaneously (e.g., high CPU AND high memory), you had to create separate alarms and then use downstream logic in a Lambda function or SNS subscriber to evaluate the combined state. This was cumbersome and introduced latency. Composite alarms allow you to define a rule that references other alarms (called "child alarms") and evaluates their states using AND, OR, and NOT operators. The composite alarm itself has a state (OK, ALARM, INSUFFICIENT_DATA) that changes based on the rule. This simplifies architecture, reduces the number of actions, and provides a single point of truth.

A composite alarm monitors the state of its child alarms. Each child alarm is a standard CloudWatch alarm (metric alarm) that evaluates its own metric or expression. The composite alarm does not directly evaluate metrics; it only reads the state of child alarms. The state of a child alarm can be OK, ALARM, or INSUFFICIENT_DATA. The composite alarm rule is an expression that combines these states using: - ALARM("alarm-name") – true if the named alarm is in ALARM state. - OK("alarm-name") – true if the named alarm is in OK state. - INSUFFICIENT_DATA("alarm-name") – true if the named alarm has insufficient data. - AND, OR, NOT – logical operators. - Parentheses for grouping.

The composite alarm evaluates its rule every 10 seconds. It retrieves the current state of each referenced child alarm from CloudWatch and computes the rule. If the rule evaluates to true, the composite alarm transitions to ALARM state; if false, to OK state. If any child alarm is in INSUFFICIENT_DATA and the rule cannot be determined (e.g., an OR condition where one alarm is INSUFFICIENT_DATA and the other is OK), the composite alarm may also go to INSUFFICIENT_DATA. The exact behavior: if the rule evaluates to true, ALARM; if false, OK; if indeterminate due to missing data, INSUFFICIENT_DATA.

You can create composite alarms via the AWS Management Console, AWS CLI, or SDK. Using the AWS CLI:

aws cloudwatch put-composite-alarm \
    --alarm-name "MyCompositeAlarm" \
    --alarm-rule "ALARM(\"HighCPU\") OR ALARM(\"HighMemory\")" \
    --alarm-actions arn:aws:sns:us-east-1:123456789012:MyTopic \
    --ok-actions arn:aws:sns:us-east-1:123456789012:MyTopic \
    --insufficient-data-actions arn:aws:sns:us-east-1:123456789012:MyTopic

To verify, describe the alarm:

aws cloudwatch describe-alarms --alarm-names "MyCompositeAlarm"

Look for StateValue and StateReason. You can also view the composite alarm in the CloudWatch console under Alarms > Composite alarms.

Composite alarms integrate with: - SNS: Most common action – send notifications to email, SMS, Lambda, etc. - Auto Scaling: Trigger scale-in or scale-out policies based on composite alarm state. - EC2 Actions: Stop, terminate, reboot, or recover an instance. - Systems Manager OpsCenter: Create OpsItems automatically. - CloudWatch Dashboards: Display composite alarm state on a dashboard widget. - AWS Chatbot: Send notifications to Slack or Chime channels.

Composite alarms are often used in conjunction with metric math alarms. For example, you might create a metric math expression that computes a custom metric (like error rate = errors / total requests), create an alarm on that expression, and then use that alarm as a child in a composite alarm with other conditions.

Metric math allows you to query multiple CloudWatch metrics and apply mathematical expressions (arithmetic, functions, comparisons) to create new time series. This is useful when you need a metric that is not directly emitted, such as a ratio, sum, or custom aggregation. For example, you can compute CPU utilization percentage across all instances in an Auto Scaling group, or calculate the 95th percentile of latency. Metric math expressions can be used in dashboards, alarms, and CloudWatch Logs Insights queries.

When you define a metric math expression, you specify a list of metrics (each with its own ID, like "m1", "m2") and an expression that references those IDs. The expression can use: - Arithmetic: +, -, *, /, ^ - Functions: SUM, AVG, MIN, MAX, COUNT, STDDEV, PERCENTILE, METRICS, FILL, RATE, DIFF, TIME_SERIES, IF, CASE, CEIL, FLOOR, ABS, LOG, etc. - Comparison: <, >, <=, >=, ==, != - Logical: AND, OR, NOT - Constants: Numbers like 100, 0.5

The result is a new time series with the same timestamps and period as the input metrics, but with the expression applied point-by-point. If metrics have different periods, CloudWatch will resample them to the specified period (default 60 seconds). Missing data points can be treated using FILL (e.g., FILL(m1, 0) fills missing values with 0).

You can use metric math in dashboards and alarms. For an alarm, you define the metric math in the Metrics array of the alarm configuration. Example using AWS CLI:

aws cloudwatch put-metric-alarm \
    --alarm-name "ErrorRateAlarm" \
    --alarm-description "Alarm when error rate exceeds 5%" \
    --metrics '[
        {"Id": "e1", "Expression": "(m1 / m2) * 100"},
        {"Id": "m1", "MetricStat": {"Metric": {"Namespace": "MyApp", "MetricName": "Errors"}, "Period": 60, "Stat": "Sum"}},
        {"Id": "m2", "MetricStat": {"Metric": {"Namespace": "MyApp", "MetricName": "Requests"}, "Period": 60, "Stat": "Sum"}}
    ]' \
    --evaluation-periods 2 \
    --threshold 5 \
    --comparison-operator GreaterThanThreshold

To verify, describe the alarm and inspect the Metrics array. You can also test expressions in the CloudWatch console by building a graph with metric math.

Metric math is used in: - CloudWatch Dashboards: Create visualizations of derived metrics. - CloudWatch Alarms: Set thresholds on derived metrics. - CloudWatch Logs Insights: Not directly, but you can export logs to metrics and then use metric math. - AWS Compute Optimizer: Uses metric math for some recommendations.

A large e-commerce platform uses Auto Scaling groups to manage its web tier. They want to scale out only when both CPU utilization is high AND the request latency is high, to avoid scaling on CPU spikes from batch jobs. They create two metric alarms: 'HighCPU' (CPU > 70% for 3 minutes) and 'HighLatency' (latency > 500ms for 3 minutes). Then they create a composite alarm with rule ALARM("HighCPU") AND ALARM("HighLatency"). The composite alarm triggers an SNS notification and a scale-out policy. This prevents unnecessary scale-outs during CPU spikes caused by background tasks. In production, they monitor the composite alarm state and adjust thresholds. A common misconfiguration is setting the evaluation periods too short, causing flapping. They use a 3-minute evaluation period on child alarms to smooth out spikes.

A SaaS company monitors the health of its microservices. They have alarms for each service: 'AuthServiceDown', 'PaymentServiceDown', etc. They want a single page when any two services are down simultaneously (indicating a broader issue). They create a composite alarm with rule (ALARM("AuthServiceDown") AND ALARM("PaymentServiceDown")) OR (ALARM("PaymentServiceDown") AND ALARM("InventoryServiceDown")). This composite alarm pages the on-call engineer via SNS->Lambda->PagerDuty. The benefit: a single service down doesn't page (it's handled by the individual alarm's own action), but a correlated failure does. In production, they have 20+ child alarms and the composite alarm evaluates every 10 seconds, providing near-real-time detection. They also set an INSUFFICIENT_DATA action to alert if too many child alarms have missing data.

A data analytics company wants to alarm on error rate per request. They emit two metrics: Errors (count) and Requests (count) from their application. They create a metric math alarm using expression (m1 / m2) * 100 where m1 is Errors (Sum) and m2 is Requests (Sum), both with a period of 60 seconds. They set threshold to 5 (5% error rate). This avoids emitting a separate error rate metric, saving costs. They also use FILL(m2, 1) to avoid division by zero when there are no requests. In production, they monitor the alarm and adjust the period to 300 seconds for a smoother trend. A common pitfall is forgetting to set ReturnData: false on the intermediate metrics, which would incorrectly return those metrics as separate time series in the alarm.

Composite alarms can reference up to 100 child alarms. Each composite alarm is evaluated every 10 seconds, so for large numbers of composite alarms, there is a limit of 100 composite alarms per account (soft limit, can be increased). Metric math expressions can include up to 50 metrics per expression. For high-resolution metrics (1-second), metric math can still be used, but the period must be at least 1 second. CloudWatch charges per metric math expression used in alarms and dashboards, so using metric math can reduce costs by avoiding additional custom metrics.