CloudWatch Log Metric Alarms

CloudWatch Log Metric Alarms are a combination of two AWS services: CloudWatch Logs and CloudWatch Alarms. They allow you to define a metric filter on a log group, which extracts numeric data from log events and publishes it as a custom CloudWatch metric. You can then create an alarm on that metric that triggers when it crosses a threshold for a specified number of evaluation periods.

Logs are a rich source of operational data. By converting log data into metrics, you can:

The process involves three main steps: 1. Define a Metric Filter: You create a filter pattern that matches specific log events. The filter can count occurrences or extract a numeric value from the log event. For example, you can count all log lines containing "ERROR" or extract the value of a JSON field like latency. 2. Metric is Published: Each time a log event matches the filter, CloudWatch Logs publishes a data point to a custom CloudWatch metric. The metric is created automatically in the AWS/Logs namespace or a custom namespace you specify. 3. Alarm Evaluates: You create a CloudWatch alarm on that metric. The alarm periodically evaluates the metric against a threshold. If the metric breaches the threshold for the specified number of evaluation periods, the alarm changes state (e.g., from OK to ALARM).

Metric Filter: - Filter Pattern: A string that defines which log events to match. Patterns can be simple (e.g., "ERROR") or complex using JSON or space-delimited parsing. - Metric Transformations: You can specify a metric namespace, metric name, metric value, and default value. The metric value is typically 1 for count filters, or you can extract a numeric field from the log event. - Default Value: If no log events match during a period, CloudWatch can publish a default value (e.g., 0) to avoid missing data points.

Alarm: - Period: The length of time (in seconds) to evaluate the metric. Valid values: 10, 30, 60, 120, 180, 240, 300, 360, 420, 480, 540, 600, 900, 1200, 1800, 21600, 43200, 86400. Default is 300 (5 minutes). - Evaluation Periods: The number of consecutive periods the metric must breach the threshold to trigger the alarm. Default is 1. - Datapoints to Alarm: The number of data points within the evaluation periods that must be breaching. Default is the same as Evaluation Periods. - Statistic: The statistic to apply to the metric (e.g., Sum, Average, SampleCount, Minimum, Maximum). For log-derived metrics, Sum is common for count filters. - Threshold: The value that triggers the alarm. Can be static or anomaly detection. - Treat Missing Data: How to treat missing data points: notBreaching, breaching, ignore, missing. Default is missing which treats missing as missing.

Alarm States: - OK: Metric is within the threshold. - ALARM: Metric is outside the threshold for the specified evaluation periods. - INSUFFICIENT_DATA: Not enough data points to evaluate (e.g., no log events in the period).

Using AWS CLI:

Create a metric filter:

aws logs put-metric-filter \
  --log-group-name "/aws/lambda/my-function" \
  --filter-name "ErrorCount" \
  --filter-pattern "ERROR" \
  --metric-transformations \
      metricName=ErrorCount,metricNamespace=MyApp,metricValue=1,defaultValue=0

Create an alarm:

aws cloudwatch put-metric-alarm \
  --alarm-name "HighErrorRate" \
  --alarm-description "Alarm when error count > 10 per minute" \
  --metric-name ErrorCount \
  --namespace MyApp \
  --statistic Sum \
  --period 60 \
  --evaluation-periods 2 \
  --threshold 10 \
  --comparison-operator GreaterThanThreshold \
  --alarm-actions arn:aws:sns:us-east-1:123456789012:MyTopic

Verification:

A large e-commerce platform uses AWS Lambda for order processing. They want to be alerted if the error rate exceeds 5% of total invocations. They configure a metric filter on the Lambda log group to count "ERROR" log events and another filter to count all invocations (by matching every log line). They create a CloudWatch metric for error count and total count. Then, they use a CloudWatch math expression to compute error rate = (errorCount / totalCount) * 100, and create an alarm on that expression with a threshold of 5. The alarm triggers an SNS notification to the operations team. In production, they set evaluation periods to 3 (5-minute periods) to avoid flapping. They also set a default value of 0 for both metrics to handle periods with no logs. The system processes millions of requests daily, generating thousands of log lines per second. They ensure the metric filter is efficient to avoid high costs. Misconfiguration: Initially, they used a filter pattern that was too broad, matching many irrelevant lines, leading to high metric data point costs and false alarms. They refined the pattern to match only lines with specific error codes.

A financial services company monitors for unauthorized access attempts. They have a CloudTrail log group capturing API calls. They create a metric filter to count events where eventName is "ConsoleLogin" and errorMessage contains "Failed authentication". They set an alarm with a threshold of 10 failed logins in 5 minutes. The alarm triggers a Lambda function that revokes the user's credentials temporarily and sends an alert to the security team. They use a period of 300 seconds and evaluation periods of 1 for immediate response. Performance: The CloudTrail logs can be high volume, but the filter pattern is specific, so only relevant events are counted. They also use a default value of 0 to ensure the alarm goes to OK when no failed logins occur. Misconfiguration: Initially, they forgot to set a default value, so during quiet periods the alarm went to INSUFFICIENT_DATA, which they had to handle. They also learned that the alarm actions must have proper IAM permissions for Lambda invocation.

A video streaming service uses EC2 instances behind an ALB. They want to scale based on 99th percentile latency measured from application logs. They configure a metric filter to extract the latency field from JSON logs and publish it as a metric with statistic p99. They create a target tracking scaling policy using this custom metric. The auto scaling group adds instances when latency exceeds 200ms. They set the metric's period to 60 seconds and use a warm-up time for new instances. Common pitfall: The custom metric must be published continuously; if the application stops logging, the metric stops, and the scaling policy may not work correctly. They use a default value to publish a synthetic data point when no logs exist. Also, they ensure the metric filter is defined on the correct log group (e.g., /aws/ec2/instance-id).