The objective "Respond to incidents and troubleshoot common AWS issues using logs and monitoring" is the safety net of cloud operations. When things break in AWS—and they will, eventually—you need a reliable method to find the fault and fix it fast. This chapter gives you that method, explaining the core tools and mental models used by the best SysOps administrators.
Jump to a section
A loud bang and a violent wobble leads to a sudden loss of control. You have a flat tyre on a busy motorway. You must pull over safely, identify the problem, and fix it without causing a crash. This is incident response and troubleshooting: something goes wrong (an incident), and you need a calm, methodical process to diagnose the cause (troubleshooting) and restore normal operation.
Your first instinct is to stop and panic, but that is dangerous. Instead, you check your mirrors, signal, and coast to the hard shoulder. You then assess the damage: is it a puncture, a blowout, or a slow leak? You check the spare tyre, the jack, and the tools. You do not just start hammering the wheel—you follow steps. Each step gives you more information. If the spare is flat too, you call roadside assistance. That is like escalating to a support team. Finally, you change the tyre and drive away. The analogue in AWS: a monitoring alarm fires (the bang), you use logs and metrics (your tools), you investigate root causes (the puncture), and you apply a fix or rollback (the spare tyre) until a permanent solution is ready. Every driver knows a flat tyre process; every sysadmin knows an incident response playbook.
Incident response and troubleshooting is the structured process of detecting that something is wrong in your AWS environment, figuring out what caused it, and taking action to make it right again. It is the IT equivalent of being a detective. AWS provides a set of services that act as your evidence-gathering toolkit. The most important of these are CloudWatch, CloudTrail, and AWS Config.
CloudWatch is the monitoring service. Think of it as sensors and dashboards. You can tell CloudWatch to track metrics—a metric is a numeric measurement, like CPU utilisation of a server or the number of incoming requests to a website. You can set thresholds, and when a metric crosses that threshold, CloudWatch fires an alarm. That alarm is your notification that something may be wrong. For example, if CPU utilisation on an EC2 instance goes above 90% for five minutes, an alarm can email you. CloudWatch Logs is a feature that collects log files from your applications and AWS services. A log file is a time-stamped record of events—like a diary of everything that has happened. Troubleshooting often starts with searching CloudWatch Logs for error messages.
CloudTrail is the audit and compliance service. It records every API call made in your AWS account. An API call is a request made by a user, a service, or a script to an AWS service. For example, when someone launches a new EC2 instance, deletes an S3 bucket, or changes a security group permission, CloudTrail logs who made the request, from which IP address, and what the result was. This is invaluable for incident response because it tells you the sequence of changes that caused the incident. Was a rule accidentally removed from a security group? CloudTrail will show you exactly when and by whom.
AWS Config is a configuration assessment service. It keeps a history of your resource configurations and evaluates them against your desired rules. For example, you can create a rule that says "any S3 bucket must have encryption enabled." If someone creates a bucket without encryption, AWS Config flags it as non-compliant and can trigger an automatic remediation. This helps you identify misconfigurations that lead to security incidents before they become problems.
When an incident occurs, you follow a pattern: detect, diagnose, respond, and recover. Detect means your alarm goes off or a user reports an issue. Diagnose means you look at CloudWatch metrics, CloudWatch Logs, and CloudTrail events to find the root cause. Respond is the immediate action to stop the damage, like shutting down a compromised server or rolling back a bad deployment. Recover means implementing a permanent fix, like patching a server or fixing a security group rule.
A common troubleshooting scenario involves an EC2 instance becoming unreachable. Your first step is to check the EC2 console for the instance state. Is it running, stopped, or terminated? If running, you check the system status checks and instance status checks in the EC2 console. If those fail, the issue may be with AWS hardware (rare) or the instance's operating system. You then use EC2 Serial Console or look at the system log in the instance screenshot to see error messages. You might also check the associated security group and network ACL (access control list) to see if they are blocking traffic.
Another scenario is an application that suddenly slows down. You look at CloudWatch metrics for CPU, memory, and network I/O (input/output). If CPU is high, you may need to scale up to a larger instance or scale out by adding more instances behind a load balancer. If memory is low, you investigate the application code for a memory leak. If network I/O is high, you check for unexpected traffic or a DDoS attack. Each metric narrows the possibilities.
Automation is a key part of modern incident response. Instead of waiting for a human to fix a repeated problem, you can use AWS Systems Manager Automation to create runbooks. A runbook is a set of pre-defined steps to respond to a specific type of incident. For example, an automation runbook could respond to a high CPU alarm by: (1) capturing a snapshot of the instance's disk, (2) attaching a new, larger instance type, and (3) sending a notification to the administrator. This reduces downtime and human error.
Why does all this matter? Without structured incident response, teams waste hours arguing about what went wrong instead of fixing it. Logs and monitoring turn blind panic into a data-driven investigation. For the SOA-C02 exam, you must know the names and purposes of these services, how to configure alarms, and how to read a CloudTrail event to identify a root cause. The exam loves testing your ability to choose the right logging tool for a specific problem and your understanding of the difference between a metric, a log, and a configuration item.
Incident Detection via Alarm or User Report
The process begins when a CloudWatch Alarm changes to the 'ALARM' state (e.g., CPU > 90% for 5 minutes) or when a user reports an issue. This is the trigger that tells you something needs investigation. Without a detection mechanism, you would not know an incident has occurred.
Initial Assessment and Log Gathering
Open the CloudWatch console and look at related metrics to understand the scope. Is one instance affected or many? Has this happened before? Simultaneously, open CloudWatch Logs for the affected service and search for error messages. This step gives you the immediate status and context.
Pinpoint the Change with CloudTrail
Navigate to CloudTrail and filter events from the past hour. Look for API calls that modify resources associated with the incident, such as 'StopInstances', 'ModifySecurityGroupRules', or 'UpdateFunctionCode'. The goal is to find the single change that initiated the problem.
Confirm the Root Cause and Apply Immediate Fix
Cross-reference the CloudTrail event with the log error messages. For example, CloudTrail shows a security group rule was removed, and logs show connections being refused. The fix might be to re-add the rule or to roll back the deployment. Apply the fix and verify the alarm clears.
Post-Incident Automation and Documentation
Create a Systems Manager Automation runbook that captures the steps you just followed. Update your runbooks. Also, consider adding a preventive measure, such as a Config rule that rejects security group changes that open SSH to the world. This step ensures the same incident does not recur.
Retrospective and Improvement
Schedule a brief meeting to review what happened, how long the response took, and what could be improved. This might lead to better alarms (shorter evaluation periods), more detailed logging, or additional training. This step closes the loop and improves your team's mean time to resolution (MTTR).
Consider a mid-sized e-commerce company that relies on AWS to run its online store. One Tuesday afternoon, the customer support team starts getting reports: users see an error page when they try to add items to their cart. The company has a SysOps engineer on call, Sam, who receives an automated alert from CloudWatch. The alarm is on a metric called 'HTTP 5xx Error Count', and it has spiked above the threshold.
Sam opens the AWS Management Console. First, he looks at the CloudWatch dashboard for the production environment. He sees that the error rate is high across all web servers, not just one. This suggests the problem is not a single faulty server but something more systematic—perhaps the database is slow or the application code has a bug that was just deployed. Sam then opens CloudTrail and filters events for the past hour. He looks for any 'UpdateFunctionCode' (a change to an AWS Lambda function) or 'CreateDeployment' from CodeDeploy. He finds that the development team deployed a new version of the shopping cart microservice thirty minutes before the errors started. That is a strong suspect.
Sam switches to CloudWatch Logs. He opens the log group for the shopping cart service and searches for the string 'ERROR' in the most recent log stream. He sees a stack trace indicating that the application is trying to access a DynamoDB table that no longer exists. The deployment script had accidentally removed the table reference. Sam alerts the development team, who re-deploy the previous version of the microservice. Within five minutes, the error rate drops back to zero. Sam then creates a runbook in Systems Manager: the runbook will automatically roll back a deployment if the 5xx error rate spikes after a new deployment. He also adds a CloudWatch Alarm on the DynamoDB table's Read/Write capacity to catch similar issues earlier.
Step by step, Sam's actions were:
He acknowledged the alarm and assessed the severity.
He used CloudTrail to pinpoint a recent change.
He used CloudWatch Logs to find the exact error message.
He identified the root cause (missing DynamoDB table).
He implemented a rollback as the immediate fix.
He automated the detection and response to prevent recurrence.
In another scenario, the company's security team notices unusual outbound traffic from an EC2 instance. Sam again uses CloudTrail to see which IAM user launched the instance and from which region. He uses AWS Config to check if the instance has an overly permissive security group. He finds the security group allows all outbound traffic. He revokes that rule and isolates the instance by detaching it from the load balancer. Then he takes a forensic snapshot of the disk for analysis. This real-world troubleshooting saves thousands of pounds in potential data exfiltration and keeps the company's customer data safe.
The SOA-C02 exam tests incident response and troubleshooting in several distinct ways. Expect scenario-based questions where you are given a description of a problem and four possible solutions. Your job is to pick the most efficient, correct answer. The examiners love to set traps by offering a solution that works but is more expensive or complex than necessary.
Here are the specific concepts you must master:
CloudWatch Alarms: Know the difference between a 'breach' threshold (any point exceeding the threshold) and an 'anomaly detection' band (a machine learning model of normal behaviour). Questions will ask: "Which alarm type should you use for a metric that normally has seasonal patterns?" The answer is Anomaly Detection.
CloudWatch Logs Insights: This is a query language to search logs. The exam tests what queries you would write to find a specific error. For example, you might need to filter for 'ERROR' in a log group and count occurrences. Know the syntax: fields @timestamp, @message | filter @message like /ERROR/ | stats count() by bin(5m).
CloudTrail: You must be able to read a CloudTrail event JSON. The exam will show you a snippet and ask who made the call ('userIdentity.arn'), what the call was ('eventName'), and whether it succeeded ('errorCode' and 'errorMessage'). Trap: CloudTrail is enabled by default for management events (like creating resources), but you must enable it for data events (like reading an object from S3). Exam questions will test whether you know this.
AWS Config: Questions love to ask how to detect a resource that is out of compliance with a policy (e.g., an S3 bucket that is not encrypted). The answer is usually not CloudWatch, but AWS Config with a managed rule. Trap: Config does not monitor real-time performance; it evaluates configuration periodically or on change. CloudWatch monitors performance.
Systems Manager Automation: The exam tests your ability to choose an automation runbook for a common incident, like a stuck EC2 instance. Knowing which SSM document to use (e.g., AWS-RebootEC2Instance or AWS-RunShellScript) is key. Trap: you might be tempted to use a manual diagnostic step when an automation document exists.
Root cause analysis: The exam gives you a multi-step scenario (e.g., database is slow, then a server goes down). You must identify the single root cause that explains both symptoms. This tests your ability to think in terms of dependencies—for example, a database overload causes application timeouts, which cause health check failures, which cause an auto-scaling group to terminate instances.
Incident response lifecycle: The exam expects you to know the phases: Detection, Analysis, Containment, Eradication, Recovery, and Post-Incident Activity. A question might ask: "What is the first step after detecting a security incident?" The correct answer is analysis (gathering evidence), not containment (stopping the attacker) because you need to understand what you are containing.
Logging sources: Know that application logs go to CloudWatch Logs, OS logs go to CloudWatch Logs (via the CloudWatch Agent), database logs are sent to CloudWatch Logs, and VPC flow logs (network traffic meta-data) are stored in CloudWatch Logs or S3. The exam tests which service stores which type of log.
Trap pattern: The exam often offers "Enable detailed billing monitoring" as a troubleshooting step. That is never the right answer for a performance or security incident. It is only relevant for cost analysis.
Key definitions to memorise:
Metric: a numerical measurement that changes over time.
Alarm: a threshold-based notification.
Log: a time-ordered record of events.
Configuration item: a resource's current state and attributes.
Automation: a scripted, repeatable response to an event.
CloudWatch Alarms notify you when a metric crosses a threshold; they are the starting point for incident detection.
CloudTrail records who did what and when in AWS; it is the essential forensic tool for finding the root cause of changes.
AWS Config evaluates whether your resources comply with internal policies; it catches misconfigurations before they cause incidents.
The four-step incident response pattern is: Detect, Diagnose, Respond, and Recover.
Application logs in CloudWatch Logs often contain the exact error message needed to fix a problem in minutes.
Systems Manager Automation runbooks let you script your most common troubleshooting steps, reducing downtime and human error.
CloudWatch Logs Insights is a query language that lets you search and analyse log data at scale.
Always check EC2 system status checks first when an instance is unreachable, before checking security groups or network ACLs.
These come up on the exam all the time. Here's how to tell them apart.
CloudWatch Metrics
Numerical data points (e.g., CPU %, latency in ms).
Ideal for automated alarms and dashboards.
Shows 'what' is happening but not the detail.
CloudWatch Logs
Textual event records (e.g., error messages, timestamps).
Ideal for deep debugging and root cause analysis.
Shows 'why' something is happening.
CloudTrail Management Events
Records actions that create, modify, or delete AWS resources.
Enabled by default and free.
Examples: 'RunInstances', 'CreateBucket'.
CloudTrail Data Events
Records actions that read, write, or access resources.
Must be explicitly enabled and costs extra.
Examples: 'GetObject' from S3, 'Invoke' on Lambda.
AWS Config
Checks configuration compliance against rules.
Focuses on resource settings (e.g., encryption enabled).
Operates continuously based on change or schedule.
Amazon Inspector
Checks for software vulnerabilities and unintended network exposure.
Focuses on security posture (e.g., CVEs, open ports).
Performs periodic or ongoing vulnerability scans.
EC2 Status Checks
Tests if the instance is reachable and its system is healthy.
Two types: System (AWS hardware) and Instance (OS).
Reported as pass/fail in the EC2 console.
EC2 CloudWatch Metrics
Track performance (e.g., CPU, network, disk I/O).
Numeric values that change over time.
Used to set alarms for performance issues.
Manual Troubleshooting
Human performs each step, clicking through the console.
Prone to human error and slow to execute.
Does not require scripting or pre-built runbooks.
Systems Manager Automation
Scripted runbook runs steps automatically.
Reduces recovery time and standardises the process.
Requires up-front creation and testing of runbooks.
SSH Access for EC2
Requires a public IP address and open SSH port in security group.
Creates a security risk if not tightly controlled.
Logs all keystrokes to CloudTrail (if configured).
Systems Manager Session Manager
Does not require a public IP or open inbound ports.
Provides secure, auditable shell access without bastion hosts.
Integrates with IAM for fine-grained permission control.
Mistake
CloudTrail captures all activity in my AWS account automatically, including every API call and every data operation.
Correct
CloudTrail captures management events by default, but it does not capture data events (like S3 GET requests or Lambda invocations) unless you explicitly enable a data event trail. You must configure it separately.
Beginners assume 'all activity' means everything, but AWS distinguishes between actions that change resources (management events) and actions that use resources (data events) for cost and noise reasons.
Mistake
If an EC2 instance is unreachable, the first step is always to check the security group.
Correct
The first step is to check the system status checks in the EC2 console. If those show a failure (like 'lost network connectivity'), the problem is at the AWS hypervisor level, not your security group. Checking security groups comes after verifying the instance is healthy at the hardware level.
Newcomers tend to think access control is the only thing that can block traffic, forgetting that the physical host or AWS networking could be the issue.
Mistake
You can use AWS Config to monitor real-time CPU utilisation of an EC2 instance.
Correct
AWS Config is for configuration compliance (what resources exist and how they are configured), not for performance metrics. CloudWatch monitors utilisation. Config records infrastructure state, not dynamic metrics.
Both services evaluate 'rules' or 'alarms', but beginners confuse the purpose of each. Config is about 'what settings are applied', CloudWatch is about 'how well the resource is performing'.
Mistake
The best way to troubleshoot a failing application is to look at the CloudWatch metrics immediately.
Correct
While metrics give you the 'what' (high error rate, CPU spike), the fastest way to find the 'why' is often to examine the application logs in CloudWatch Logs for error messages. Metrics tell you something is wrong; logs tell you exactly what is wrong.
Metrics are numbers, logs are words. Beginners gravitate to numbers because they seem objective, but a single error message in a log can save hours of metric analysis.
Mistake
If you stop and start an EC2 instance, you lose all the logs stored on that instance's disk.
Correct
When you stop an instance, the root volume (EBS volume) remains attached and retains all data, including logs. Only a termination deletes the volume. Stopping does not destroy data.
People confuse 'stop' with 'terminate'. In physical servers, switching off does not wipe the disk, but cloud terminology exaggerates the fear of data loss.
Mistake
Every CloudWatch alarm should use a static threshold like 'CPU > 80%' for all scenarios.
Correct
Some metrics have variable baselines. CloudWatch Anomaly Detection uses machine learning to create a band of expected values. Static thresholds cause false alarms for metrics that change with traffic patterns, like request count on a website that gets more traffic during the day.
Beginners want simplicity (one number fits all), but real-world patterns are seasonal. The exam tests your ability to recognise when to use static vs. anomaly detection.
Reveal each answer, then mark whether you got it right. Score 60%+ to unlock the next chapter.
CloudWatch Logs stores application and system logs (text records of events like error messages). CloudTrail records API calls to AWS services (who did what when). Logs help you debug applications; CloudTrail helps you audit changes to your AWS environment.
Enable CloudTrail with a trail that applies to all regions. This ensures that no matter where an action occurs, you have a record. A single-region trail can miss incidents that start in a region you do not monitor.
Use the EC2 console. Select the instance, then Actions > Monitor and Troubleshoot > Get System Log. This shows the console output from the operating system, which often includes boot errors and kernel panics.
Yes. Use CloudWatch Alarm Actions to trigger an SNS topic that calls a Lambda function or use Systems Manager Automation runbooks. You can also use Auto Scaling lifecycle hooks to respond to instance failures without custom code.
A stuck instance is one that fails status checks (e.g., 'Instance reachability check failed'). You can use Systems Manager Automation runbook 'AWS-RecoverEC2Instance' to automatically stop and start the instance on new hardware. This can be triggered by a CloudWatch Alarm.
Yes, you are charged per GB of data scanned by your query. Always write specific filters (e.g., filter by timestamp and known error strings) to minimise the amount of data scanned and control costs.
Check AWS Config for recent compliance changes to your resources. If there are no changes, check CloudWatch metrics for performance anomalies. A sudden spike in error rate with no config change usually indicates a code deployment or traffic surge, not a misconfiguration.
You've just covered Incident Response and Troubleshooting — now see how well it sticks with free SOA-C02 practice questions. Full explanations included, no account needed.
Done with this chapter?