This chapter dives deep into the cost implications of CloudWatch Logs ingestion and custom metrics, a critical topic for the SAA-C03 exam under objective 4.4 (Cost Optimization). You will learn exactly how AWS charges for these services, the default values and thresholds that drive costs, and how to architect your monitoring to minimize expenses without sacrificing observability. Approximately 10-15% of exam questions touch on CloudWatch cost optimization, often in the context of choosing between logs and metrics or setting up cost-effective monitoring for large-scale applications.
Jump to a section
Imagine a high-end buffet restaurant where you pay for each plate you fill, but the cost per plate depends on what you put on it. CloudWatch Logs is like the restaurant's inventory system: every time a chef (your application) adds a dish (a log line) to the buffet, the system weighs the plate and charges you per gigabyte of food. But there's a twist: the restaurant also offers a premium service where chefs can customize dishes (custom metrics) and pay per 'spoonful' (per metric data point) instead of per plate. The restaurant's owner (you, the architect) must decide whether to let chefs dump everything on the buffet (logs) or to have them create special small plates (custom metrics) that are cheaper but require more effort. If chefs log too many verbose messages (like saying 'I picked up a carrot' every second), the buffet bill skyrockets. Conversely, if they only log critical events (like 'the kitchen is on fire'), costs stay low. The restaurant also has a loyalty program (CloudWatch Logs Insights) that lets you query past meals, but querying more data costs extra. The key to saving money is to serve only the most valuable dishes on the buffet and use smaller, cheaper plates for the rest.
What is CloudWatch Logs Ingestion and Why Does It Cost Money?
CloudWatch Logs is a service that collects, monitors, and stores log files from AWS resources (EC2, Lambda, VPC Flow Logs, etc.) and custom applications. Every log event ingested incurs a cost based on volume, measured in gigabytes (GB). The service charges for: - Data Ingestion: Per GB of log data uploaded to CloudWatch Logs. - Data Storage: Per GB per month for logs stored (first 5 GB free). - Data Transfer: Outbound data transfer to internet or other regions (not typically a major factor for most architectures, but relevant for cross-region log shipping). - Logs Insights Queries: Per GB of data scanned by queries.
Custom metrics, on the other hand, are data points you define and publish to CloudWatch. They cost per metric per month, and per API request for publishing. Custom metrics are typically cheaper for high-frequency, low-volume data compared to ingesting verbose logs.
How Ingestion Pricing Works Internally
CloudWatch Logs uses a 'pay-as-you-ingest' model. The first 5 GB per account per month is free (across all regions combined). After that, ingestion costs vary by region but typically range from $0.50 to $0.90 per GB. Storage costs are additional, typically $0.03 per GB per month after the first 5 GB.
Key Defaults and Timers: - Log events are retained indefinitely by default. - You can set retention policies from 1 day to 10 years, or never expire. - Log streams are automatically created per log source (e.g., each EC2 instance publishes to its own stream). - There is no limit on the number of log streams, but each stream has a 256 KB per second throughput limit.
Publishing Custom Metrics: - You can publish custom metrics via the PutMetricData API. - Each API call can contain up to 20 metric data points. - Each data point has a timestamp (up to 2 weeks in the past or 2 hours in the future) and a value (or statistic set). - Cost: $0.30 per custom metric per month (first 10 metrics free) + $0.01 per 10,000 API requests.
The Cost Trap: Logging Everything vs. Using Metrics
A common mistake is to log every application event at DEBUG level and send all logs to CloudWatch. For a high-traffic web server, this can easily generate 10-100 GB per day, costing hundreds of dollars monthly. In contrast, converting key performance indicators (e.g., request latency, error rate) into custom metrics reduces cost dramatically because:
Metrics are aggregated (e.g., you send one data point per minute representing the average latency, rather than 10,000 individual log lines).
Metrics have a fixed cost per metric, regardless of how many data points you send (though you pay per request).
Example Calculation: - Suppose an application generates 1 million log lines per day, each ~500 bytes (500 MB/day, ~15 GB/month). Ingestion cost: ~$7.50/month (at $0.50/GB). Storage: ~$0.30/month. Total ~$7.80/month. - If you instead publish 10 custom metrics with one data point per minute (43,200 data points/month), cost: 10 metrics * $0.30 = $3.00 + API requests (43,200/10,000 = 4.32 * $0.01 = $0.0432). Total ~$3.04/month. - However, you lose the raw log details necessary for debugging. The trade-off is granularity vs. cost.
Interacting with Related Technologies
AWS Lambda: Lambda automatically sends logs to CloudWatch Logs. Each invocation generates at least one log stream. For high-throughput Lambda functions, log costs can dominate. Solution: reduce log verbosity or use custom metrics for business KPIs.
VPC Flow Logs: These can generate massive amounts of log data. A typical VPC with hundreds of instances can produce 10+ GB/day of flow logs. Cost optimization: aggregate logs into a single log group, set a short retention (e.g., 7 days), and use metrics for traffic patterns instead.
CloudWatch Agent: The unified CloudWatch agent can collect both logs and metrics from EC2 and on-premises servers. You can configure it to send only specific log files and to filter out low-value log events.
Configuration and Verification Commands
To check your current log volume and cost, use:
aws cloudwatch get-metric-statistics --namespace AWS/Logs --metric-name IncomingBytes --start-time 2023-01-01T00:00:00Z --end-time 2023-01-31T23:59:59Z --period 86400 --statistics SumTo estimate cost, multiply the sum of IncomingBytes (in bytes) by the per-GB cost for your region.
To set a log retention policy:
aws logs put-retention-policy --log-group-name /aws/lambda/my-function --retention-in-days 7To publish a custom metric:
aws cloudwatch put-metric-data --namespace MyApp --metric-name RequestLatency --value 0.5 --unit Milliseconds --timestamp 2023-06-01T12:00:00ZAdvanced Cost Saving: Logs to S3 and Athena
For logs that must be retained for compliance but are rarely queried, you can export logs to Amazon S3 using ExportTask or via subscription filter to Kinesis Data Firehose. Once in S3, you can use Amazon Athena to query them, paying only for S3 storage (cheaper than CloudWatch Logs storage) and Athena query costs (per TB scanned). This is often cheaper for long-term retention.
Export cost comparison: - CloudWatch Logs storage: $0.03/GB/month. - S3 Standard storage: $0.023/GB/month (first 50 TB). - S3 Glacier Instant Retrieval: $0.004/GB/month (for infrequently accessed logs). - Plus, you avoid CloudWatch Logs Insights query costs.
Summary of Key Cost Drivers
Log verbosity: DEBUG vs. INFO vs. ERROR level.
Log frequency: How often logs are generated per source.
Number of log sources: Each EC2 instance, Lambda function, etc., creates its own log stream.
Retention period: Longer retention increases storage costs.
Log size per event: Larger log lines increase ingestion cost.
Custom metrics: Fixed per-metric cost but can reduce overall spend if used to replace high-volume logs.
Exam Tip: When to Choose Logs vs. Metrics
Use logs for: debugging, troubleshooting, compliance audits, detailed event tracing.
Use custom metrics for: alarm thresholds, dashboards, auto scaling triggers, cost-effective monitoring of high-frequency events.
The exam often presents a scenario where a company is spending too much on CloudWatch and asks you to recommend a cost optimization. The answer is usually: convert some log data to custom metrics, reduce log verbosity, set retention policies, or export logs to S3.
Identify Log Sources and Volume
First, inventory all sources sending logs to CloudWatch: EC2 instances (via CloudWatch agent or AWS Systems Manager), Lambda functions, VPC Flow Logs, API Gateway, etc. Use CloudWatch Logs Insights or the AWS Cost Explorer to estimate current log volume in GB per month. For new architectures, estimate based on expected traffic. This step is crucial because you cannot optimize what you do not measure. A typical web server can generate 1-2 GB per day per instance at INFO level, while a Lambda function might generate 0.1-0.5 GB per day per 100k invocations. Record the log groups and their current retention settings.
Set Retention Policies
For each log group, set an appropriate retention policy. By default, logs never expire, leading to infinite storage costs. Use the AWS CLI or console to set retention to 7, 14, 30, or 90 days depending on compliance and debugging needs. For logs that must be kept longer, export to S3 and then delete from CloudWatch. The command: `aws logs put-retention-policy --log-group-name <group> --retention-in-days <days>`. This is a quick win that can reduce storage costs by 90% or more.
Reduce Log Verbosity at Source
Adjust application logging level from DEBUG to INFO or ERROR. For example, in a Java application, change log4j properties. For Lambda, set the logging level in the function code or use a logging library that supports dynamic levels. This reduces both ingestion and storage costs because fewer log events are sent. A common exam scenario: a developer sets logging to DEBUG for troubleshooting but forgets to revert, causing high costs. The solution is to use a feature flag or environment variable to control log level in production.
Filter and Aggregate Logs with Metric Filters
Instead of sending every log event to CloudWatch, use metric filters to extract key data points. For example, create a metric filter on a log group that counts occurrences of 'ERROR' and publishes that count as a custom metric. Then, you can set an alarm on that metric. This reduces log volume because you can choose to not send the raw logs (or send only a sample). Metric filters are free, but the custom metrics they generate incur standard custom metric costs. This is a classic cost optimization pattern on the exam.
Export Infrequently Accessed Logs to S3
For logs that must be retained for compliance but are rarely queried (e.g., older than 30 days), export them to Amazon S3 using CloudWatch Logs export tasks. This moves the data out of CloudWatch Logs storage (expensive) to S3 (cheaper). You can then use S3 Lifecycle policies to transition to Glacier for even lower cost. To query, use Athena directly on S3. The export task can be automated via a scheduled Lambda function. Note: export tasks are charged per GB exported, but this is often less than the ongoing storage cost.
Scenario 1: E-commerce Platform with Spiky Traffic
A large e-commerce site runs on hundreds of EC2 instances behind an Application Load Balancer. They initially sent all application logs (DEBUG level) to CloudWatch, generating 500 GB per month at a cost of $250 for ingestion and $15 for storage. After a cost audit, they implemented the following:
- Set log retention to 14 days (down from infinite). - Changed log level to INFO in production. - Created metric filters for key business metrics (e.g., checkout latency, error rate). - Exported logs older than 7 days to S3 Glacier. Result: Ingestion dropped to 50 GB/month ($25), storage to $1.50, plus $0.50 for custom metrics. Total savings: ~$238/month.
Scenario 2: Serverless Application with High Lambda Invocations
A startup runs a serverless API with 10 million Lambda invocations per month. Each invocation logs a 2 KB entry, totaling 20 GB/month. Cost: $10 ingestion + $0.60 storage = $10.60. They wanted to monitor error rates and latency. Instead of logging every invocation, they:
- Reduced logging to ERROR only (from INFO), dropping volume to 500 MB/month. - Published custom metrics for latency (average, p99) and error count via the CloudWatch agent or embedded metric format. - Used CloudWatch Logs Insights only for debugging specific requests. New cost: $0.25 ingestion + $0.30 custom metrics = $0.55/month. Savings: $10.05/month.
Common Pitfalls
Not setting retention: Many engineers forget to set retention, leading to years of log data accumulating at high cost.
Over-metering: Creating too many custom metrics (e.g., one per user) can be more expensive than logs. AWS charges per metric per month, so 10,000 custom metrics cost $3,000/month.
Ignoring VPC Flow Logs: Flow logs can generate terabytes per month. Set them to aggregate and use S3 export with Athena for analysis.
Not using metric filters: Instead of sending raw logs, use filters to extract only what you need for alarms, reducing both ingestion and storage.
What SAA-C03 Tests on This Topic (Objective 4.4)
The exam expects you to recommend cost-optimized monitoring architectures. Specific focus areas: - Choosing between logs and custom metrics for different use cases. - Setting retention policies to minimize storage costs. - Using metric filters to reduce log volume. - Exporting logs to S3 for long-term retention. - Understanding the free tier: first 5 GB of log ingestion, first 10 custom metrics, and 10 metric alarms free per month.
Common Wrong Answers and Why Candidates Choose Them
'Enable detailed CloudWatch monitoring on all EC2 instances' — This increases cost because detailed monitoring sends metrics every 1 minute (vs. 5 minutes for basic). Candidates think 'more monitoring is better', but the question asks for cost optimization.
'Store logs in CloudWatch Logs indefinitely for compliance' — While compliance requires retention, indefinite storage in CloudWatch is expensive. The correct answer is to export to S3/Glacier.
'Use CloudWatch Logs Insights for all queries' — Logs Insights charges per GB scanned. For frequent queries, it's cheaper to use custom metrics or Athena on S3.
'Publish a custom metric for every HTTP status code' — This creates many metrics (200, 201, 301, 404, 500, etc.), each costing $0.30/month. A better approach is to publish a single metric with a dimension for status code, or use a metric filter.
Specific Numbers and Terms on the Exam
Free tier: 5 GB log ingestion, 10 custom metrics, 10 alarms.
Custom metric cost: $0.30 per metric per month.
Log ingestion cost: ~$0.50/GB (varies by region, but exam uses $0.50).
Retention: up to 10 years, but default is never expire.
Metric filter: can publish to a custom metric.
Export to S3: via ExportTask or subscription filter to Kinesis Firehose.
Edge Cases and Exceptions
Cross-account log aggregation: If you aggregate logs from multiple accounts into a central account, the central account pays ingestion costs. The exam might ask you to optimize by filtering before shipping.
Embedded Metric Format (EMF): Lambda can output logs in a structured JSON format that CloudWatch automatically converts to custom metrics, reducing the need for separate PutMetricData calls. This is a cost-saving technique.
High-resolution metrics: Custom metrics can have a resolution of 1 second (vs. standard 1 minute), but they cost more (per metric). The exam may test when to use high-res (e.g., for rapid scaling).
How to Eliminate Wrong Answers
If an answer suggests 'store all logs indefinitely in CloudWatch', eliminate it — it's never cost-optimized.
If an answer involves 'enable detailed monitoring on all instances', check if the scenario requires 1-minute granularity. If not, it's a cost increase.
If an answer says 'use CloudWatch Logs Insights for real-time monitoring', remember that real-time monitoring is better done with custom metrics and alarms.
Look for answers that combine: set retention, reduce verbosity, use metric filters, export to S3.
CloudWatch Logs ingestion costs $0.50/GB on average; first 5 GB per account free.
Custom metrics cost $0.30 per metric per month; first 10 metrics free.
Always set a log retention policy to avoid indefinite storage costs (default is never expire).
Reduce log verbosity in production from DEBUG to INFO or ERROR to cut ingestion volume.
Use metric filters to extract key data from logs and publish as custom metrics, then optionally stop sending raw logs.
Export logs older than N days to S3/Glacier for long-term retention at lower cost.
For high-frequency monitoring, prefer custom metrics over logs to reduce cost.
Detailed monitoring (1-minute) on EC2 costs extra; use basic (5-minute) unless needed.
These come up on the exam all the time. Here's how to tell them apart.
CloudWatch Logs
Charged per GB ingested and stored.
Best for detailed debugging and compliance.
Retention can be set from 1 day to 10 years.
Query costs with Logs Insights per GB scanned.
Default free tier: 5 GB/month.
Custom Metrics
Charged per metric per month + per API request.
Best for alarms, dashboards, and auto scaling.
Data points are aggregated and stored for 15 months.
No query cost; alarms evaluate at no extra charge.
Default free tier: 10 metrics/month.
Mistake
CloudWatch Logs ingestion is free for the first 5 GB per month per log group.
Correct
The free tier is 5 GB per account per month, not per log group. All log groups share the same 5 GB free allowance.
Mistake
Custom metrics are always cheaper than logs.
Correct
Custom metrics have a fixed cost per metric per month. If you create thousands of metrics (e.g., per user), the cost can exceed log ingestion. Use dimensions wisely to keep metric count low.
Mistake
Setting a retention policy deletes logs immediately.
Correct
The retention policy deletes logs older than the specified number of days. Logs within the retention period are unaffected. Deletion happens after the retention period expires, not immediately upon setting the policy.
Mistake
Metric filters are charged per filter.
Correct
Metric filters themselves are free. You only pay for the custom metrics they generate (if any) and for the log ingestion of the original logs. However, filters do not reduce ingestion cost unless you also stop sending the raw logs.
Mistake
You can export logs from CloudWatch Logs to S3 for free.
Correct
Export tasks incur charges for data transfer out of CloudWatch Logs (typically $0.03/GB) and for the S3 PUT requests. However, it is usually cheaper than long-term storage in CloudWatch.
Reveal each answer, then mark whether you got it right. Score 60%+ to unlock the next chapter.
First, reduce the log level in your Lambda code to INFO or ERROR. Second, use the Embedded Metric Format (EMF) to output structured logs that automatically create custom metrics, allowing you to stop sending verbose logs. Third, set a short retention period (e.g., 7 days) for the Lambda log group. Finally, if you need long-term retention, export logs to S3 via a subscription filter to Kinesis Firehose.
Basic monitoring sends metrics every 5 minutes at no additional cost. Detailed monitoring sends metrics every 1 minute and costs $2.10 per instance per month (for 7 metrics). For cost optimization, use basic monitoring unless you need 1-minute granularity for auto scaling or rapid response.
No, CloudWatch Logs Insights charges $0.005 per GB of data scanned. However, the first 5 GB of log ingestion is free, but queries on that data still incur costs. To save money, limit the time range of your queries and use filters to scan only necessary log groups.
You can create an export task via the AWS CLI or console, but it is not automatic. For automated exports, use a subscription filter to deliver logs to Amazon Kinesis Data Firehose, which can then write to S3. Alternatively, use a Lambda function triggered by a CloudWatch Events rule to create export tasks on a schedule.
CloudWatch Logs storage costs $0.03/GB/month. S3 Standard costs $0.023/GB/month, and S3 Glacier Deep Archive costs $0.001/GB/month. Additionally, exporting logs from CloudWatch to S3 incurs a data transfer cost of about $0.03/GB. For long-term retention, S3 is significantly cheaper.
You get 10 custom metrics free per account per month. After that, each custom metric costs $0.30/month. Note that each unique combination of metric name and dimensions counts as a separate metric.
A metric filter scans incoming log events for a pattern (e.g., 'ERROR') and publishes a count to a CloudWatch metric. This allows you to alert on errors without storing the raw log lines. If you then stop sending the raw logs (by changing log level or filtering at source), you reduce ingestion cost. However, the metric itself costs $0.30/month.
You've just covered CloudWatch Cost: Logs Ingestion and Custom Metrics — now see how well it sticks with free SAA-C03 practice questions. Full explanations included, no account needed.
Done with this chapter?