EventBridge for Automated Remediation

Amazon EventBridge is a serverless event bus that ingests, filters, transforms, and routes events from various sources to targets. It evolved from Amazon CloudWatch Events, offering enhanced capabilities like schema registry, event archiving, and replay. EventBridge enables decoupled, event-driven architectures where producers and consumers are independent. For automated remediation, EventBridge can monitor events from AWS services (e.g., EC2 instance state changes, CloudTrail API calls, AWS Health events) and trigger automated responses such as running a Lambda function to stop an unhealthy instance or sending a notification to an SNS topic.

EventBridge operates on a publish-subscribe model. Events are JSON objects that describe a change in state. The flow is:

- Event Buses: Default bus for AWS services, custom bus for your applications, and partner buses for SaaS events. You can create up to 100 custom event buses per account (soft limit). - Rules: Up to 300 rules per event bus. Each rule can have up to 5 targets. - Event Patterns: Must match the event's source, detail-type, resources, and detail fields. Patterns use exact matching, prefix matching, suffix matching, numeric matching, and existence matching. Example pattern:

{
    "source": ["aws.ec2"],
    "detail-type": ["EC2 Instance State-change Notification"],
    "detail": {
      "state": ["stopped"]
    }
  }

To create a rule for automated remediation via AWS CLI:

aws events put-rule \
  --name "UnhealthyInstanceRemediation" \
  --event-pattern '{"source":["aws.health"],"detail-type":["AWS Health Event"],"detail":{"service":["EC2"],"eventTypeCategory":["issue"]}}' \
  --state ENABLED

Then add a target (Lambda function):

aws events put-targets \
  --rule "UnhealthyInstanceRemediation" \
  --targets "Id"="1","Arn"="arn:aws:lambda:us-east-1:123456789012:function:myRemediationFunction"

To verify the rule:

aws events list-rules --name-prefix "Unhealthy"
aws events list-targets-by-rule --rule "UnhealthyInstanceRemediation"

A large e-commerce company runs thousands of EC2 instances across multiple Availability Zones. They receive AWS Health events when an instance is scheduled for retirement or has an underlying hardware issue. They use EventBridge to automatically launch a replacement instance and attach the existing EBS volumes. The rule is configured on the default event bus with a pattern matching source: aws.health and detail-type: AWS Health Event. The target is a Lambda function that checks the event details, identifies the affected instance, creates an AMI if needed, launches a new instance in the same subnet and security group, and updates the Route 53 DNS record. The Lambda function is idempotent—if retried, it checks if the replacement already exists. The company also configures a dead-letter queue to capture any events that fail after retries, and a CloudWatch alarm on the DLQ for monitoring. Performance considerations: at peak, they might receive dozens of health events per hour, well within EventBridge limits. A common misconfiguration is forgetting to grant EventBridge permission to invoke the Lambda function, causing silent failures.

A financial services company must ensure that no unauthorized security group rules are created. They use AWS CloudTrail to log API calls, and EventBridge to react to AuthorizeSecurityGroupIngress events. The rule pattern matches source: aws.ec2 and detail-type: AWS API Call via CloudTrail. The target is a Lambda function that inspects the new rule; if it violates company policy (e.g., allows SSH from 0.0.0.0/0), the function immediately revokes the rule using revoke_security_group_ingress. The Lambda function also sends an alert to the security team via SNS. The company archives all events for 90 days for audit purposes. A challenge is that CloudTrail events can be delayed by up to 15 minutes, so remediation is not instantaneous. They also need to handle cases where the rule was created by a privileged user—the Lambda function checks the IAM user and only reverts if not authorized. Misconfiguration: if the event pattern does not match the exact CloudTrail event format, the rule never triggers.

A gaming company uses Auto Scaling Groups (ASGs) to handle variable traffic. Occasionally, an ASG fails to launch an instance due to insufficient capacity or a bad AMI. They use EventBridge to detect EC2 Auto Scaling Instance Launch Failure events. The rule triggers a Step Functions state machine that analyzes the failure reason, adjusts the launch configuration (e.g., different instance type), and retries the launch. The state machine includes a wait step to avoid immediate retry, and a human approval step for certain failure types. They also archive events for replay during post-mortems. A common issue is that the Step Functions execution can exceed the 5-minute timeout for EventBridge target invocation, so they set the target to asynchronous invocation. Performance: they handle hundreds of events per day during peak launches.