Logging and Monitoring with Amazon CloudWatch. This is the system that gives you a live feed of everything happening inside your AWS account, from a single server's CPU usage to the exact error message a user received when a function crashed. For the DOP-C02 exam, you need to master this because it is the central nervous system for troubleshooting problems, setting up automatic responses, and proving your architecture is healthy – without it, you are flying blind.
Jump to a section
A simple way to picture Logging and Monitoring with Amazon CloudWatch
Have you ever wondered how you would know if your house's security system was actually working while you were on holiday?
Imagine you have a smart home. You install a security camera (a sensor) pointed at your front door. This camera is your 'monitoring agent.' It watches the front door and records a timestamp every time someone approaches. These timestamps, along with short video clips, are your 'logs' – a chronological record of events. The camera sends these clips to a central hub in your house, which is like Amazon CloudWatch's 'Logs' feature – a single place to store all the event records from different sensors.
Now, you also have a temperature sensor in the basement that measures the ambient temperature every five minutes. The reading '22.5 degrees Celsius' is a single data point, or a 'metric.' The central hub stores all these temperature readings over time, allowing you to see a graph of temperature changes. This is exactly what 'CloudWatch Metrics' does – it collects and stores numerical data points from your IT resources, like CPU usage or memory consumption.
But you didn't install all of this just to watch – you want to be alerted. So, you set up a rule: if the temperature in the basement drops below 10 degrees, send a text message to your phone. In CloudWatch, this rule is an 'Alarm.' The alarm watches the temperature metric and, when the condition is met, it triggers an action, like sending a notification via Amazon Simple Notification Service (SNS). Finally, you create a display on your phone that shows the front door camera feed, the temperature graph, and a list of all recent alerts. This is your personalised dashboard – in CloudWatch, a 'Dashboard' is a customisable page where you can visualise your metrics and logs all in one place. Your entire home security system, with its sensors, records, rules, and central display, is a perfect analogy for how CloudWatch lets you monitor and log your entire AWS environment.
CloudWatch is the native monitoring and observability service for Amazon Web Services. Think of it as a central command centre that collects, analyses, and acts on data from every corner of your AWS environment. It covers four main pillars: Metrics, Logs, Alarms, and Dashboards. Let us break down each one in plain language.
CloudWatch Metrics
A 'metric' is a fundamental unit of data. It is simply a variable that you want to measure over time. For example, the percentage of your EC2 instance's CPU being used is a metric. The amount of disk space consumed on a database is a metric. AWS services send these metrics to CloudWatch automatically for free. These are called 'standard metrics' and they are sent at a baseline frequency (e.g., every 5 minutes).
But you can also create your own custom metrics. Perhaps your application tracks the number of items in a shopping cart. You can send that data to CloudWatch yourself using the CloudWatch API or the 'PutMetricData' command. These custom metrics can be sent more frequently, such as every minute, for a small fee.
Metrics are stored and visualised. You can look at a graph of your EC2 CPU utilisation over the last hour, day, or week. This helps you spot trends, like a gradual increase in memory usage that might indicate a memory leak. Each metric is uniquely defined by its namespace (a container for metrics, like 'AWS/EC2'), its metric name (like 'CPUUtilization'), and its dimensions (like the specific InstanceId).
CloudWatch Logs
While metrics are numbers, logs are text. A log is a record of an event, typically including a timestamp and a description. When your application writes a message to the console saying 'ERROR: Connection timed out,' that message can be sent to CloudWatch Logs.
CloudWatch Logs is a centralised service for storing, monitoring, and accessing your log files. Instead of logging into every single server to read log files, you send all logs to one place. You can set up an agent (a small piece of software) on your EC2 instances to automatically forward log files from your application or the operating system to CloudWatch.
A key feature here is 'Log Groups' and 'Log Streams.' Think of a Log Group as a folder for a specific type of log, like logs for the application 'my-app.' Inside that folder, each individual server or process creates a 'Log Stream' – a sequence of log events from that specific source. This organisation lets you search across all your logs for a specific error.
The most powerful feature of CloudWatch Logs is 'Logs Insights.' This is an interactive query engine. You can write a query in a language similar to SQL to search, filter, and analyse your log data. For instance, you can query to 'find the 20 most common error messages in the last 24 hours' without having to read through thousands of lines of text manually.
CloudWatch Alarms
An alarm watches a single metric over a specified time period and performs an action based on the metric's value relative to a threshold. This is the 'action' part of monitoring. You cannot watch graphs all day; you need the system to wake you up when something wrong happens.
An alarm has three states: OK (the metric is within the threshold), ALARM (the metric has breached the threshold), and INSUFFICIENT_DATA (not enough data points to determine the state). When an alarm transitions to the ALARM state, it can trigger an action, such as: sending a notification to an email list via Amazon Simple Notification Service (SNS), automatically stopping or terminating an EC2 instance, or triggering an Auto Scaling policy to add more servers.
You can also set 'Composite Alarms' which watch multiple metrics at once. For example, 'Alert me only if CPU is high AND memory is high,' which reduces false alarms from transient spikes.
CloudWatch Dashboards
A dashboard is a customisable home page in the CloudWatch console that you can use to monitor your resources in a single view. You can add graphs of metrics, text widgets, and even live lists of alarms. You can create dashboards for different purposes: an 'Operations dashboard' showing overall system health, or a 'Production dashboard' showing the specific metrics of your main application. Dashboards can be shared with other team members or even made public (if you intend to display them on a big TV screen in the office).
CloudWatch Agent
This is a crucial piece of software. The CloudWatch Agent is installed on your EC2 instances or on-premises servers. It collects system-level metrics (like memory usage, disk I/O, and swap utilisation) that are not collected by the default EC2 monitoring. It also collects custom log files and sends them to CloudWatch Logs. Understanding how to install and configure the CloudWatch Agent is a common exam topic.
CloudWatch Contributor Insights
This advanced feature analyses time-series data to identify the top contributors to your systems. For example, you can see which IP address is making the most requests to your API, which URL is the most accessed, or which error code is appearing most frequently. This is extremely useful for debugging performance bottlenecks and security issues.
CloudWatch Synthetics
This is a managed service for running 'canaries' – configurable scripts that run on a schedule to monitor your endpoints and APIs. They simulate a user performing a workflow (like logging in and adding an item to a cart) and can report back the success, failure, or performance of that flow. This is proactive monitoring, catching issues before real users do.
Why it Exists
Before CloudWatch, you had to set up your own monitoring infrastructure. You had to install a database to store metrics, a graphing tool like Grafana, a logging system like the ELK stack (Elasticsearch, Logstash, Kibana), and a notification system. CloudWatch bundles all of this into a single, managed service that integrates deeply with every other AWS service. It replaces the need to build and maintain your own monitoring stack.
1. Install and Configure the CloudWatch Agent
This is the starting point for gaining visibility. You must install the Unified CloudWatch Agent on your EC2 instance (or on-prem server). You then create a configuration file (usually JSON) that defines which metrics (memory, disk, etc.) and which log files to collect. The agent reads this config and sends data to CloudWatch.
2. Create Log Groups and Set Retention
Once the agent is running, it sends log events to a specified Log Group. Before the logs arrive, you should create the Log Group in the CloudWatch console and set a retention policy (e.g., 14 days). Without a retention policy, logs are stored indefinitely and cost you money. After creation, you can view the streams inside the group.
3. Define and Create Alarms for Key Metrics
Identify the critical metrics that indicate trouble, like CPU, memory, or a specific custom metric. For each metric, create an alarm. Define the threshold (e.g., CPU > 80%), the evaluation period (e.g., 2 consecutive periods of 5 minutes), and the action (e.g., send an email via SNS). This step turns raw data into actionable notifications.
4. Build a Dashboard for Visual Overview
Instead of checking individual graphs, create a single CloudWatch Dashboard. Add a line widget for your main metrics (CPU, memory), a number widget for the current alarm count, and a log widget to show live errors. This gives you a single-pane-of-glass view of your system's health, essential for operations.
5. Use Logs Insights for Deep Analysis
When an alert triggers, do not guess the cause. Open CloudWatch Logs Insights. Write a query to filter for the time of the alert and search for error codes (e.g., 'filter @message like /ERROR/'). Use 'stats count()' to see which errors are most frequent. This turns raw text into structured, queryable data for rapid troubleshooting.
6. Set Up a Synthetics Canary for Proactive Checks
Do not wait for a problem to break. Create a CloudWatch Synthetics canary. Write a simple Node.js or Python script that simulates a user action (like logging into your app). Set it to run every 5 minutes. Create an alarm on the canary's 'successPercent' metric. If the canary fails, you get alerted before real users suffer.
Imagine you are the sole IT person for a small e-commerce start-up. You have a weekend sale launching on Friday at 8 AM. You cannot afford for the site to go down. Here is how you would use CloudWatch in a real-world scenario.
Step 1: Set up the Foundation with the CloudWatch Agent
On Monday, you install the CloudWatch Agent on your single t3.medium EC2 instance running the website. You configure it to send the standard memory and disk metrics to CloudWatch. You also configure it to forward the application logs from /var/log/myapp/error.log to a new Log Group called '/myapp/errors'. You verify the metrics appear in the CloudWatch console by checking the 'CWAgent' namespace.
Step 2: Create Alarms for Critical Resources
On Tuesday, you create your first Alarm. You look at the 'CPUUtilization' metric for your instance. You set a threshold: if CPU > 80% for two consecutive periods of 5 minutes (10 minutes total), transition to ALARM. You configure this alarm to send a notification to an SNS topic that is subscribed to your email and your personal phone via SMS. You create another alarm for the 'Memory % Used' metric to trigger if it goes above 90%.
Step 3: Automate a Response with Auto Scaling
Mid-week, you realise that an email alert is not enough on a busy day. You cannot log in and manually start a new server at 3 AM. So, you create an 'Auto Scaling group' (a group of EC2 instances that automatically adjust in number based on demand). You then modify your CPU alarm action. Instead of just sending an email, the alarm now triggers a 'Scale Out' activity in your Auto Scaling group, which deploys a second EC2 instance to handle the load. You also create a 'Scale In' alarm that triggers when CPU is low, terminating the extra instance to save money.
Step 4: Build a Dashboard for the Sale
On Thursday, the day before the sale, you build a CloudWatch dashboard. You add a line graph of CPUUtilization for your instance. You add a number widget showing the current number of healthy instances in your Auto Scaling group. You add a log widget showing the most recent error entries from your '/myapp/errors' Log Group. You put this dashboard on a monitor in your home office so you can glance at it while working.
Step 5: Use Logs Insights to Debug a Problem
The sale goes live. At 10 AM, you get an alert on your phone. The CPU alarm triggered. You check the dashboard and see the Auto Scaling group launched a second instance. However, users start reporting a '500 Internal Server Error' on the checkout page. You open CloudWatch Logs Insights. You query the '/myapp/errors' Log Group with this query:
fields @timestamp, @message
filter @message like /500/
sort @timestamp desc
limit 20
You find a repeated error: 'DatabaseConnectionError: Cannot connect to RDS endpoint.' You immediately investigate your RDS database and find it has reached its maximum connections limit. You increase the limit and the errors stop. Without CloudWatch Logs Insights, you would have had to SSH into the server and grep through log files, wasting precious time.
Step 6: Proactive Monitoring with Synthetics
After the sale, you set up a CloudWatch Synthetics canary that runs a script every minute to log in to your site, search for a product, and add it to the cart. If this canary fails, it triggers an alarm. This catches problems, like a broken login page, instantly, often before any real user reports it.
The DOP-C02 exam focuses heavily on the operational aspects of CloudWatch. They want to test if you can design and implement monitoring solutions for complex, highly-available architectures. Do not just study what a metric is; study how to aggregate them, how to set up detailed monitoring, and how to use them to trigger automated actions.
Metrics – What They Love to Test
Standard vs. Detailed Monitoring: This is a classic trap. For EC2, standard monitoring sends metrics every 5 minutes for free. Detailed monitoring sends them every 1 minute for an extra cost. The exam will ask: 'You need to detect a spike in CPU within 1 minute. Which should you enable?' Answer: Detailed monitoring. They will also test that you need detailed monitoring for Auto Scaling to react quickly or for a fast-scaling alarm.
Resolution and Period: Metrics have a resolution (how fine-grained the data is) and a period (the length of time you are evaluating). Standard is 5-minute resolution, Detailed is 1-minute. There are also 'High-Resolution' custom metrics that can go down to 1 second. An alarm that uses a period of 60 seconds can only work with metrics that have a resolution of 60 seconds or better.
Custom Metrics: Know how to publish custom metrics using the PutMetricData API. You can publish them to a specific namespace and with specific dimensions. Dimensions are key-value pairs that help you filter the metric. For instance, a dimension could be 'InstanceType' with a value of 't3.micro'. The exam might ask you to design a solution for sending application-level metrics.
Logs – The Exam's Favourite
Log Groups and Log Streams: Know the difference. A Log Group is a container for Log Streams. A Log Stream is a sequence of events from a single source. You set retention policies on the Log Group level.
CloudWatch Logs Agent vs. Unified CloudWatch Agent: This is a huge trap. The old 'CloudWatch Logs Agent' only sends logs. The new 'Unified CloudWatch Agent' sends both logs and metrics (including memory and disk metrics). The exam will ask you to recommend the 'Unified CloudWatch Agent' for collecting both memory metrics and logs. Never choose the old agent if the requirement also mentions metrics.
Logs Insights: You will be asked to understand the query syntax. They might ask which query finds errors, or which command counts log entries. Remember: 'fields' to pick columns, 'filter' to narrow, 'stats count()' to aggregate, 'sort' to order, 'limit' to restrict results.
Subscription Filters and Vended Logs: Know that you can stream logs from a Log Group to other services using a 'Subscription Filter.' For example, you can stream all your logs to Amazon OpenSearch Service for real-time analytics, to Amazon S3 for archival, or to Amazon Kinesis Data Firehose for analysis. 'Vended Logs' are logs that AWS services (like Application Load Balancer or VPC Flow Logs) publish directly to a Log Group in your account.
Alarms – The Action Trigger
Alarm States and Actions: Know OK, ALARM, INSUFFICIENT_DATA. Know that an alarm can trigger an SNS notification or an Auto Scaling policy. They love to test that you can create 'EC2 Action' alarms to stop, terminate, or reboot an instance.
Composite Alarms: The exam will test this. Composite alarms combine multiple metric alarms into a single alarm with a single state. They reduce noise. 'Create a composite alarm that is in ALARM only if AlarmA and AlarmB are both in ALARM.'
Missing Data Treatmnet: How does an alarm behave when it does not receive data for a period? Options: notBreaching, breaching, ignore, missing. This is a tricky detail they love to test. For instance, if a server is terminated, the alarm should go to ALARM (missing data treated as breaching) to notify the ops team.
Dashboards – Simple but Specific
Cross-Account Dashboards: The exam will test that you can build a single dashboard that displays metrics from different AWS accounts using cross-account observability. This is key for organisations with multiple accounts.
Dashboard Sharing: Know that you can share dashboards with other AWS accounts or even make them public (with a warning about security).
ServiceLens and other additions
CloudWatch ServiceLens: This integrates CloudWatch with AWS X-Ray (a service for tracing requests through your application). It provides a unified view of your application's health, performance, and dependencies. The exam may ask you to use ServiceLens to visualise the flow of a request through microservices.
Container Insights: A feature specifically for monitoring containerised applications on Amazon ECS, Amazon EKS, and Kubernetes. It collects metrics like CPU and memory utilisation per pod or container.
Lambda Insights: Specific to Lambda functions. It collects CPU time, memory used, and cold start times. The exam might ask you to enable this to identify a function with a high cold start latency.
Standard EC2 monitoring is every 5 minutes for free; Detailed monitoring is every 1 minute at an additional cost.
The Unified CloudWatch Agent is required to collect memory, disk, and custom log files from inside an EC2 instance.
Log Groups are containers for log streams; log streams are sequences of events from a single source.
Composite alarms combine multiple metric alarms to reduce noise and require all sub-alarms to be in ALARM.
Logs Insights allows you to query your logs with a SQL-like language to find specific errors and patterns.
CloudWatch Synthetics canaries simulate user workflows to proactively detect website issues before real users do.
Custom metrics can be published using the PutMetricData API and can have a resolution as low as 1 second (high-resolution).
Subscription filters allow you to stream log data to services like Amazon S3, OpenSearch, or Kinesis for further analysis.
These come up on the exam all the time. Here's how to tell them apart.
Standard Monitoring (EC2)
Free of charge, enabled by default
Collects metrics every 5 minutes
Sufficient for basic alerting and long-term trends
Detailed Monitoring (EC2)
Incurs a per-instance per-hour charge
Collects metrics every 1 minute
Required for fast-scaling Auto Scaling and high-frequency alarms
CloudWatch Logs Agent
Can only collect and send log files
Older agent, now deprecated by AWS
Cannot collect system metrics like memory or disk
Unified CloudWatch Agent
Can collect both log files and system metrics (memory, disk, etc.)
Current recommended agent from AWS
Single configuration file for both logs and metrics
Metric Alarm
Watches a single metric (e.g., a specific CPU value)
Evaluates data from that one metric over time
Can have false alarms from transient spikes on one metric
Composite Alarm
Combines the states of multiple metric alarms using AND/OR logic
Reduces noise and false alarms by requiring multiple conditions
Ideal for complex conditions like 'CPU high AND memory high'
CloudWatch Logs
A storage service for raw log data from various sources
Provides a console to view log streams in real time
Basic filtering is limited to searching for a string
CloudWatch Logs Insights
An interactive query engine within CloudWatch Logs
Uses a SQL-like query language for complex analysis (e.g., aggregation, grouping)
Purpose-built for deep troubleshooting and pattern detection across terabytes of logs
Mistake
Standard monitoring sends metrics every 1 minute.
Correct
Standard monitoring for EC2 sends metrics every 5 minutes. Detailed monitoring (paid) sends them every 1 minute.
The word 'standard' sounds like it should be adequate, and people confuse it with the frequency of other services. The exam specifically tests this distinction.
Mistake
CloudWatch automatically monitors CPU, memory, and disk space on every EC2 instance.
Correct
CloudWatch automatically sends CPU utilisation and a few other hypervisor-level metrics. Memory, disk, and swap metrics require installing the CloudWatch Agent on the instance.
People assume the 'cloud' monitors everything. AWS monitors the hypervisor (the host), not the guest OS (your virtual server). The agent is required to see inside the guest.
Mistake
The CloudWatch Logs Agent and the Unified CloudWatch Agent are interchangeable.
Correct
The Unified CloudWatch Agent is the newer, recommended agent that can collect both logs and system metrics (memory, disk). The old Logs Agent only collects logs.
AWS changed the naming, causing confusion. Beginners often think they just pick one agent at random, but the exam will test the specific capabilities of each.
Mistake
An alarm triggered by a high CPU metric will automatically fix the problem.
Correct
An alarm triggers an action (like an SNS notification or an Auto Scaling policy). It does not directly fix the cause; it only executes the configured action. You must configure the action to do the fixing.
People anthropomorphise the alarm, thinking it has built-in healing powers. They forget they must define the action (e.g., 'launch a new instance').
Mistake
CloudWatch Logs stores logs permanently and for free.
Correct
CloudWatch Logs has a default retention of 'Never expire,' but data older than 14 days costs more per GB to store. You must set an explicit retention policy (e.g., 30 days) to control costs.
Beginners see 'unlimited' storage and think it is free. AWS charges for storage of logs, especially as they age into long-term archival.
Reveal each answer, then mark whether you got it right. Score 60%+ to unlock the next chapter.
CloudWatch Metrics store numerical data points (like CPU percentage) that you can graph and create alarms on. CloudWatch Logs store text records (like error messages) that you can search and analyse with Logs Insights.
The CloudWatch Agent software itself is free. However, you pay for the storage of the logs and metrics it sends, as well as for custom metrics and detailed monitoring frequency.
Yes. When creating an alarm, you can choose an EC2 Action. Options include 'Stop', 'Terminate', or 'Reboot' the instance. The alarm must be in the ALARM state to trigger the action.
Use CloudWatch Logs Insights. First, ensure all servers send logs to the same Log Group (or a set of related Log Groups). Then, write a query like: 'fields @timestamp, @logStream, @message | filter @message like /database error/ | sort @timestamp desc'.
Install and configure the Unified CloudWatch Agent on your instance. Edit its configuration file to enable the 'mem' section. The agent will then send the 'mem_used_percent' metric to CloudWatch automatically.
Metric data sent by the CloudWatch Agent is buffered on the instance. If the instance crashes, any buffered data that has not been sent to CloudWatch is lost. For critical data, consider sending metrics directly from the application using the PutMetricData API, which sends synchronously.
You've finished Logging and Monitoring with Amazon CloudWatch. Continue through the DOP-C02 study guide to build a complete picture of the exam.
Done with this chapter?