For exam objective 4.2 of the DOP-C02, you must master building automated workflows that fix common infrastructure problems without human intervention. This concept is called automated remediation, and it forms the backbone of self-healing systems in the cloud — architectures that detect failures and repair themselves automatically. Understanding this is critical because the DevOps Engineer Professional exam expects you to design systems that minimise downtime and manual toil, not just to know that automation exists.
Jump to a section
A simple way to picture Automated Remediation and Self-Healing Systems
A modern self-cleaning oven is a closed system with pre-programmed responses to its own operational hazards.
When grease and food residue accumulate inside the oven, continued use at normal temperatures creates a fire risk. In a traditional oven, the user must notice the buildup, manually select a cleaning cycle, lock the door, and wait. If the user forgets or ignores it, the hazard grows.
A smart self-cleaning oven with automated remediation works differently. Internal sensors constantly monitor the level of carbon compounds on the heating elements. When the residue reaches a predefined threshold, the oven's control system automatically initiates a high-temperature burn-off cycle. It locks the door, ramps up to 500 degrees Celsius, and incinerates the debris to ash. Crucially, if the oven detects that the kitchen temperature is too high or that the smoke detector in the room is active, it delays the self-cleaning cycle until conditions are safe. This is not a scheduled cleaning — it is an autonomous response to a detected abnormal state. The oven self-heals its dirty condition without a human pushing a button.
This maps precisely to automated remediation in AWS. An EC2 instance is the oven. Its disk space filling up with log files is the grease buildup. CloudWatch alarms are the internal sensors. An AWS Lambda function that deletes old log files or triggers an Auto Scaling action is the self-cleaning cycle. AWS Systems Manager Automation documents act as the safety interlocks that check conditions before acting. The entire system — detect, decide, act, verify — runs without a human opening the server door.
Automated remediation is the practise of setting up automatic responses to common operational problems. Think of it as programming your infrastructure to fix itself. Self-healing systems take this further: they are designed from the ground up to detect faults, diagnose the root cause, and execute a recovery plan — all without a human pushing a button or SSH-ing into a server.
To understand why this matters, you first need to know what it replaces. In traditional IT operations (often called 'Ops'), when a server runs out of disk space or a critical process crashes, a human receives a page or an alert. That person logs into the machine, investigates, and manually runs commands to fix the problem. This is slow, expensive, and error-prone. If the person is asleep or on holiday, the system stays broken. Automated remediation eliminates the human bottleneck.
The core components of an automated remediation system in AWS are:
CloudWatch Alarms: These are the sensors. You configure them to monitor a metric — for example, 'DiskUsedPercent' on an EC2 instance. When the metric crosses a threshold (say, 85%) for a certain number of consecutive data points, the alarm changes state from OK to ALARM.
Amazon EventBridge: This is the decision engine. You create a rule that says 'when a CloudWatch alarm enters ALARM state, trigger this specific action.' EventBridge can send the alert to multiple targets simultaneously.
AWS Lambda: This is the action runner. A Lambda function is a small piece of code that runs when triggered. For disk space remediation, the Lambda function could connect to the EC2 instance via AWS Systems Manager (SSM), run a command to delete old log files, and then confirm the disk usage has dropped below the threshold.
AWS Systems Manager Automation: This is the orchestration layer. Instead of a simple Lambda function, you can use an SSM Automation document that defines a multi-step workflow: stop the instance, detach the corrupted volume, attach a new volume, restart the service, verify health, and then clean up. Automation documents can include conditional steps and error handling.
AWS Auto Scaling: This is the ultimate self-healing mechanism for compute resources. An Auto Scaling group can automatically replace a failed EC2 instance by launching a new one from a golden AMI (Amazon Machine Image). Combined with a health check, if an instance fails the health check for a set period, Auto Scaling terminates it and launches a replacement — no script required.
The sequence of events in an automated remediation flow looks like this:
A resource (EC2, RDS database, Elastic Load Balancer) begins to degrade — disk fills up, memory spikes, latency increases.
CloudWatch monitors the relevant metric every minute (or faster with detailed monitoring).
The metric crosses the alarm threshold, and the CloudWatch alarm transitions to ALARM.
The alarm publishes a message to an SNS topic or directly to EventBridge.
EventBridge triggers the remediation action — typically a Lambda function or an SSM Automation document.
The remediation action diagnoses the problem, executes the fix (e.g., clearing /tmp, rotating logs, scaling out), and verifies the fix.
Optionally, the remediation action sends a success or failure notification to an operations chat channel or ticketing system.
If the remediation fails, a second-level alarm can page a human.
Why does AWS care about this for the DevOps Engineer Professional exam? Because the exam tests your ability to design systems that are resilient and require minimal manual intervention. You will see questions about how to chain these services together, which service is appropriate for a given scenario, and how to handle errors within the remediation workflow itself.
Real AWS services used in automated remediation:
AWS Config rules: These can trigger remediation when a resource becomes non-compliant with a policy. For example, if an S3 bucket becomes publicly accessible, AWS Config can automatically apply a bucket policy to make it private again.
AWS Trusted Advisor: This service checks your account against AWS best practices. You can configure it to trigger automated actions for specific checks, such as closing an idle RDS instance.
Amazon Inspector: For security vulnerabilities, Inspector findings can trigger Lambda functions that patch or quarantine affected instances.
The difference between simple automation and a self-healing system is that a self-healing system includes feedback loops. After the fix is applied, the system checks the metric again. If the problem persists, it can escalate to a different remediation step — for example, instead of deleting old logs, it could launch a new instance from a clean image. This layered approach prevents the same problem from happening repeatedly.
The DOP-C02 exam expects you to know the exact configuration options for each of these services. You should understand the concept of lifecycle hooks in Auto Scaling, which allow you to run custom actions before an instance is launched or terminated. You should also know the difference between 'replacing' an instance and 'remediating' an instance — the exam will test which approach is appropriate for different failure modes.
Define the metric and threshold
Choose what you monitor (e.g., CPU utilisation, disk space, error count) and set the alarm threshold. This step determines when remediation triggers. Set it high enough to avoid false alarms but low enough to act before failure.
Create the CloudWatch alarm
Configure the alarm to evaluate the metric over a sliding window. For example, 'average CPU > 80% for 3 consecutive periods'. The alarm will change state to ALARM when the condition is met.
Configure the EventBridge rule
Create a rule that listens for the alarm entering ALARM state. This rule acts as the trigger for the remediation action. You can target multiple actions (Lambda, SSM, SNS) from one rule.
Implement the remediation action
Write the code or SSM document that executes the fix. For a Lambda function, include logic to diagnose the problem, apply the fix, and verify the result. For SSM Automation, define steps with success and failure branches.
Test the remediation in isolation
Simulate the failure condition on a non-production instance. Verify that the alarm triggers, the action runs, and the metric returns to OK. This step catches misconfigurations (wrong IAM role, missing dependencies) before deployment.
Add a fallback and escalation mechanism
If the primary remediation fails, the system should try a different approach (e.g., replace the instance instead of repairing it) or page an on-call engineer. This ensures no failure mode goes unhandled.
Monitor the remediation system itself
Create alarms on the Lambda function's error count or the SSM Automation's failure status. If the remediation tool itself is broken, the whole system fails silent. This step provides visibility into the health of the automation.
Imagine you are the DevOps engineer at a company called FinFlow, which processes financial transactions 24/7. They run a fleet of 50 EC2 instances behind an Application Load Balancer. Each instance runs a Java application that writes logs to the local disk at a rate of 1 GB per hour. The instances have 30 GB of disk space. If the disk fills up, the Java application crashes, and the transactions fail. The old way was: an operations team member receives a PagerDuty alert at 3 AM, SSHes into the instance, runs 'sudo rm -rf /var/log/old/*', restarts the application, and goes back to sleep. This manual process takes 15 minutes and happens twice a week.
You design an automated remediation system. Here is exactly what you build:
You enable detailed CloudWatch monitoring on all EC2 instances to push metrics every 60 seconds instead of every 5 minutes. This gives you faster detection.
You create a CloudWatch alarm on the 'DiskSpaceUtilization' metric (using the CWAgent that you install via SSM) with a threshold of 85% for 2 consecutive data points. This means the alarm triggers when disk usage stays above 85% for two minutes.
You create an SNS topic called 'disk-remediation-actions' and subscribe your EventBridge rule to it.
You write an AWS Lambda function in Python that uses boto3 to connect to the EC2 instance via AWS Systems Manager Run Command. The Lambda function sends a command to the instance that runs a script located at /opt/finflow/cleanup.sh. The script deletes log files older than 24 hours from /var/log/finflow/app.log.
You configure the Lambda function to wait for the command to complete and then publish a success metric to CloudWatch. If the command fails, the Lambda function publishes a failure metric, which triggers a second alarm that pages the on-call engineer.
You attach the cleanup script to the instance's launch template so that it is present on every new instance from the start.
For a safety net, you add a second CloudWatch alarm at 95% disk usage. This alarm triggers an SSM Automation document that stops the instance, detaches the root volume, attaches a new 50 GB volume from a snapshot, attaches the old volume as a secondary device so the data is not lost, restarts the instance, and verifies the application is healthy. This is the self-healing fallback.
In practise, you would also create an AWS Config rule that checks every instance has the CWAgent installed. If an instance is missing the agent, Config auto-remediates by installing it via SSM. This is a preventative remediation — it stops the problem before it starts.
The real world complexity: you need to handle the case where the remediation itself fails. For example, if the Lambda function times out because the cleanup script hangs, your system should not go into an infinite loop. You add a CloudWatch alarm on the 'Errors' metric of the Lambda function. If errors exceed 0 for 3 consecutive minutes, the alarm triggers a different action — perhaps tagging the instance as 'quarantined' and routing its traffic away using the load balancer's target group deregistration.
Additionally, you need to consider permissions. The Lambda function must have an IAM role that allows it to call 'ssm:SendCommand', 'ec2:DescribeInstances', and 'cloudwatch:PutMetricData'. The EC2 instance must have an IAM role that allows SSM to execute commands on it. These are common exam traps: failing to attach the correct IAM role will break the automation.
Finally, you test the remediation by simulating the failure. You manually fill the disk on a test instance and verify that the Lambda cleanup runs, the alarm clears, and no human intervention was needed. Only then do you deploy the configuration to production.
The DOP-C02 exam tests automated remediation in three distinct ways: scenario-based multiple-choice questions, multi-step design questions, and troubleshooting questions where you must identify why a remediation system failed.
Core exam topics you must memorise:
The exact difference between CloudWatch Alarms, EventBridge rules, and Systems Manager Automation. The exam loves to test which service is best for a given scenario. For example: 'You need to trigger a multi-step workflow that includes a conditional branch and a rollback. Which service do you use?' The answer is SSM Automation, not Lambda, because SSM Automation documents support steps, branching, and rollbacks natively.
The concept of lifecycle hooks in Auto Scaling. You will be asked what happens when an instance is in a 'terminating:wait' state. The answer is that the lifecycle hook pauses termination so you can run a final backup or de-registration script.
AWS Config automatic remediation and the difference between 'auto-remediate' and 'manual remediation'. Config can trigger SSM Automation documents when a resource becomes non-compliant.
The difference between reactive remediation (fix after alarm) and proactive remediation (fix before alarm). For example, using AWS Health events to trigger remediation before a scheduled maintenance event affects your instances.
Common exam traps:
Trap: The question describes a simple one-step action like 'stop an EC2 instance when CPU exceeds 90%' but lists multiple seemingly correct answers. The trap is that AWS does not allow CloudWatch Alarms to directly stop an instance. You must use a Lambda function or SSM Automation as the intermediary. The correct answer will include an EventBridge rule or an SNS topic that triggers the action.
Trap: The question asks for 'the most cost-effective automated remediation for a low-priority development instance'. Beginners choose Auto Scaling replacement, but that creates and destroys instances, which costs money. The correct answer is often a Lambda function that stops the instance during off-hours and starts it in the morning.
Trap: The question says 'remediating a security group that allows SSH from 0.0.0.0/0'. The correct answer is AWS Config auto-remediation using a custom SSM document that applies a secure security group. The trap is thinking you should modify the security group directly via a Lambda function — Config is the managed service designed for this.
Trap: For self-healing, the exam tests your understanding of 'disposable instances'. An instance that has local state (ephemeral data) cannot be simply replaced. The correct remediation is to fail over to another instance in a different Availability Zone. The exam will ask what to do when the instance has a local file that must be preserved.
Definitions you must memorise:
'Warm standby': A replicated environment that is running but not serving traffic until a failover occurs.
'Health check': A request made by the load balancer to a target to determine if it is healthy.
'Grace period': The time Auto Scaling waits after launching an instance before checking its health.
'Stateful' versus 'stateless': Stateful instances store data locally and cannot be replaced without data loss. Stateless instances can be replaced instantly. The exam expects you to use stateless designs to enable self-healing.
Question type patterns:
Pattern 1: 'A developer reports that their EC2 instance's disk fills up every night. They want automated remediation. What is the most efficient solution?' The answer is a CloudWatch alarm on disk usage -> EventBridge -> Lambda function that deletes old log files. The trap answer is 'run a cron job inside the instance' — but the question asks for automated remediation, not manual scripting.
Pattern 2: 'Which AWS service allows you to define a multi-step remediation workflow with conditions?' Answer: AWS Systems Manager Automation.
Pattern 3: 'An RDS instance is failing due to memory pressure. What automated remediation is appropriate?' Answer: Scale up the instance class using AWS Systems Manager Automation, not Lambda, because Lambda cannot modify RDS instance classes directly without the AWS SDK being invoked correctly.
To pass this section, memorise the AWS Well-Architected Framework's 'Reliability Pillar' which explicitly discusses self-healing designs. The exam references this framework directly. Know the five key services: CloudWatch, EventBridge, Lambda, SSM Automation, and AWS Config. Understand how to chain them together. The exam will not ask you to write code, but it will ask you to identify the correct sequence of services and the correct IAM permissions needed for each step.
Automated remediation is a feedback loop: CloudWatch monitors a metric, triggers an EventBridge rule, which invokes a Lambda function or SSM Automation document to fix the issue and then verifies the fix.
AWS Systems Manager Automation documents can define multi-step workflows with conditional branches and automatic rollbacks, making them superior to Lambda for complex remediation sequences.
Self-healing systems require instances to be stateless — if state must be preserved, use a separate data store (like EFS or RDS) so the instance can be replaced without data loss.
AWS Config rules with automatic remediation can fix compliance violations (like open security groups) immediately, without waiting for a human to respond.
Lifecycle hooks in Auto Scaling allow you to run custom actions (backups, de-registration) before an instance is terminated or after it is launched, enabling safe self-healing.
The exam expects you to know when to use reactive remediation (fix after alarm) versus proactive remediation (fix using AWS Health events before the problem affects you).
Every automated remediation action must have error handling: if the remediation fails, the system should either retry, escalate to a different action, or page a human — not silently fail.
IAM permissions are the most common reason a remediation system fails in both the real world and the exam: the Lambda function needs an execution role, and the target resource (EC2, RDS) needs a service role.
These come up on the exam all the time. Here's how to tell them apart.
CloudWatch Alarm + Lambda
Best for simple one-step actions like deleting old files or restarting a service.
Requires you to write and manage custom code in a Lambda function.
No built-in rollback; you must implement error handling manually.
Systems Manager Automation
Designed for multi-step workflows with conditional branches, approvals, and automatic rollbacks.
Uses pre-built or custom Automation documents (YAML/JSON) — no custom code needed.
Includes built-in error handling and the ability to specify different behaviour for each step.
Reactive Remediation
Triggers after a problem is detected (e.g., disk > 85%).
Fixes issues that have already begun to degrade performance.
Can be too slow if the failure happens very quickly (e.g., memory exhaustion in seconds).
Proactive Remediation
Uses AWS Health events or scheduled actions to prevent problems before they occur.
Example: proactively rotate logs every hour via a cron-style Lambda, even before the disk threshold is reached.
Reduces risk of downtime by addressing issues before they become critical.
Auto Scaling Group (Replacing Instances)
Replaces a failed instance entirely using a launch template and golden AMI.
Works only for EC2 compute resources in an Auto Scaling group.
Effective for stateless applications but destroys local data on the failed instance.
AWS Config Auto-Remediation
Fixes configuration drift on any supported AWS resource (S3, security groups, IAM roles).
No instance replacement — it modifies the resource configuration directly.
Used for compliance violations (e.g., S3 bucket becomes public, security group opens SSH).
Mistake
A CloudWatch alarm can directly stop or terminate an EC2 instance without any other service.
Correct
A CloudWatch alarm can only change state. It cannot directly modify AWS resources. You must use an intermediary like Auto Scaling, Lambda, or Systems Manager Automation to execute the action.
Beginners see the CloudWatch console showing 'Stop instance' as an action and assume it is a direct call. In reality, AWS uses CloudWatch Events (now EventBridge) to connect the alarm to the action; the alarm itself does not perform the stop.
Mistake
Self-healing means the system never fails, so you do not need backups.
Correct
Self-healing systems repair themselves after a failure, but they cannot recover from data corruption or deletion of the entire infrastructure. Backups (snapshots, RDS automated backups) are still essential for disaster recovery.
The term 'self-healing' sounds magical, leading beginners to believe it replaces disaster recovery. In reality, a self-healing system replaces a failed compute instance with a new one from a golden image, but if the golden image itself is compromised or the data is lost, healing cannot restore it.
Mistake
Automated remediation and auto scaling are the same thing.
Correct
Auto scaling is a specific type of automated remediation for compute instances — it replaces failed instances. But automated remediation is broader: it includes fixing a misconfigured security group, clearing disk space, restarting a service, or rerouting traffic. Auto scaling is one tool in that toolkit.
Beginners hear 'AWS scales automatically' and equate all automation with Auto Scaling. The exam tests multiple forms of remediation (AWS Config, Lambda, SSM) and expects you to recognise when Auto Scaling is not the best fit (e.g., for non-compute resources like RDS or S3).
Mistake
You should deploy the same remediation to all environments (dev, test, prod) without changes.
Correct
Remediation actions must be tailored to the environment. In development, stopping an instance is acceptable. In production, stopping might cause an outage; you would instead fail over to a healthy instance or trigger a health check replacement. The remediation itself should also be tested in a lower environment first.
Beginners often think 'automation is automation' and apply the same Lambda function to all accounts. The DOP-C02 exam tests your ability to design environment-aware systems: production remediation must be less aggressive and include human approval steps for certain actions (like terminating an instance).
Reveal each answer, then mark whether you got it right. Score 60%+ to unlock the next chapter.
No, the DOP-C02 exam does not ask you to write code. It tests your understanding of which AWS services to chain together (e.g., CloudWatch -> EventBridge -> Lambda) and what IAM permissions are needed. You should know the capabilities of each service, not the syntax.
Automated remediation is the act of fixing a specific problem automatically. Self-healing is a design philosophy where the entire system is built to detect and recover from failures without manual intervention, often using multiple layers of remediation and elastic scaling.
Yes, a Lambda function with the correct IAM permissions can call the EC2 StopInstances API. However, the exam often prefers AWS Systems Manager Automation for multi-step workflows because it includes built-in error handling and rollback capabilities.
Because CloudWatch alarms only change state from OK to ALARM or INSUFFICIENT_DATA. They are monitoring constructs, not action executors. You must route the alarm to a service like Auto Scaling, Lambda, or SSM Automation that has permission to perform the action.
A stateless instance does not store critical data locally. All application data is kept in external services like databases (RDS), object storage (S3), or file systems (EFS). This allows the instance to be destroyed and replaced at any moment without data loss, which is essential for self-healing.
Use a mechanism like a DynamoDB table or a CloudWatch metric to track how many times a remediation has been attempted for a specific resource. After a few attempts, stop the automatic remediation and escalate to a human. This is called a 'circuit breaker' pattern.
You've finished Automated Remediation and Self-Healing Systems. Continue through the DOP-C02 study guide to build a complete picture of the exam.
Done with this chapter?