CloudWatch Metric Filters from Log Groups

CloudWatch Metric Filters are rules that you define on a CloudWatch Log Group to scan incoming log events for specific patterns and convert matching events into CloudWatch metric data points. They allow you to generate custom metrics from log data without writing custom code or processing logs externally. For example, you can count the number of ERROR log entries, track the average response time from an access log, or monitor the frequency of specific HTTP status codes.

Before metric filters, you had to either send logs to a third-party monitoring tool or write custom scripts to parse logs and push metrics to CloudWatch. Metric filters provide a native, serverless, and cost-effective way to turn log data into actionable metrics. They are especially useful for operational health monitoring, SLA tracking, and security event detection.

When you create a metric filter on a Log Group, CloudWatch Logs starts scanning every log event ingested into that group in real time. The filter pattern is a space-delimited list of keywords or a JSON expression that defines what to look for. For each incoming log event, the filter engine evaluates the pattern against the log message. If the pattern matches, the filter emits a metric data point to CloudWatch.

- Filter Pattern Types: - Space-delimited patterns: Used for unstructured logs like syslog or Apache access logs. Each term is a literal string or a numeric placeholder. For example, [ERROR] matches events containing the word ERROR. - JSON patterns: Used for structured JSON logs. You specify a JSON path and a condition. For example, $.status = 500 matches events where the status field is 500.

- Metric Value Assignment: - Count: The simplest metric is a count of matching events. The filter emits a value of 1 for each match, and CloudWatch aggregates these counts over the metric's period (default 1 minute). - Extracted value: You can extract a numeric value from the log event using a pattern with a numeric placeholder ($N). For example, in an Apache log with format %h %l %u %t "%r" %>s %b, you can extract the response size by defining a pattern like [..., $size, ...]. The extracted value is then emitted as the metric value.

- Metric Namespace and Name: - Each metric filter defines a custom metric namespace (e.g., MyApp/LogMetrics) and a metric name (e.g., ErrorCount). These become the metric's identity in CloudWatch.

- Metric Value and Default Value: - For count metrics, the value is always 1 per match. For extracted value metrics, the value is the extracted number. You can also specify a default value (e.g., 0) to emit when no match occurs, which is useful for detecting absence of events.

- Metric namespace (string, max 255 chars) - Metric name (string, max 255 chars) - Metric value (either $1 for extracted value or 1 for count) - Default value (optional, number) - Log Group: A filter is attached to one Log Group. It cannot span multiple Log Groups. - Real-time scanning: Filters are evaluated as log events are ingested. There is no batch delay; however, there is a latency of up to 60 seconds for the metric to appear in CloudWatch. - Throttling: CloudWatch Logs has a throttling limit for metric filters. By default, you can have up to 100 metric filters per account per region. Each filter can emit up to 10 metric values per log event. If the log ingestion rate exceeds the filter's processing capacity, some events may not be evaluated (though this is rare and typically at very high throughput).

You can create a metric filter via AWS Management Console, AWS CLI, or SDK. Using AWS CLI:

aws logs put-metric-filter \
  --log-group-name "/aws/lambda/my-function" \
  --filter-name "ErrorCount" \
  --filter-pattern "ERROR" \
  --metric-transformations \
      metricName=ErrorCount,metricNamespace=MyApp,metricValue=1,defaultValue=0

To verify, list metric filters:

aws logs describe-metric-filters --log-group-name "/aws/lambda/my-function"

You can also test a filter pattern against sample log events using the test-metric-filter CLI command:

aws logs test-metric-filter \
  --filter-pattern "ERROR" \
  --log-event-messages "INFO: Success" "ERROR: Failed"

This returns which events matched and the emitted metric values.

The SOA-C02 exam often presents scenarios where you need to monitor specific log patterns. You might be asked to:

A large e-commerce platform runs hundreds of microservices on AWS Lambda and ECS. Each service writes structured JSON logs to CloudWatch Logs. The operations team needs to detect spikes in 5xx errors across all services. They create metric filters on each Log Group with a JSON pattern { $.status >= 500 } and emit a count metric named 5xxCount. They use a single dashboard to graph all 5xxCount metrics. An alarm is set on each metric to trigger if the sum over 5 minutes exceeds a threshold. This setup allows real-time alerting without custom log processing. One common issue: if a developer changes the log format (e.g., renames status to httpStatus), the filter stops matching silently. To mitigate, the team uses Logs Insights to periodically validate that the pattern still matches.

A SaaS company uses Application Load Balancer (ALB) access logs to monitor response times. They need to ensure that 99% of requests complete in under 500ms. They create a metric filter on the ALB Log Group that extracts the request_processing_time field using a space-delimited pattern. The pattern is [..., $request_time, ...] where $request_time is the numeric value at a specific position. They emit the average of $request_time as a custom metric. An alarm alerts if the average exceeds 400ms for 10 minutes. However, they discover that the filter only captures a sample of logs because the ALB logs contain multiple entries per request. They switch to a JSON pattern to accurately parse the ALB log format. This scenario highlights the importance of understanding the exact log format before creating filters.

Metric filters are evaluated in real time but have a per-account throttling limit of 100 filters. For high-volume log streams (e.g., 10,000 log events per second), the filter engine may fall behind, causing some events to be skipped. AWS recommends using Kinesis Data Firehose or Lambda subscriptions for advanced processing. Also, each filter adds cost: $0.50 per filter per month plus $0.10 per million log events scanned. At scale, these costs can be significant. A common misconfiguration is creating too many fine-grained filters (e.g., one per error type) instead of using a single filter with multiple metric transformations.