Incident and Event Response practice questions

The security group for the RDS instance has changed during failover.

Why it fails: Security groups remain the same.

The DNS record for the RDS endpoint has not propagated to the application's DNS resolver.

Why it fails: RDS DNS updates quickly and is not usually the issue.

The database instance is still in the process of failover and is not yet accepting connections.

Why it fails: The status is Available, so failover is complete.

The role does not have permission to describe instances, so the condition cannot be evaluated.

Why it fails: DescribeInstances works, so the condition is evaluated.

The policy's Resource element is set to '*' and must be restricted to specific instance ARNs.

Why it fails: Resource '*' is acceptable for these actions.

The policy does not include the ec2:RebootInstances action.

Why it fails: Missing actions are not the issue; the allowed actions fail.

Use Amazon GuardDuty to automatically modify the security group membership of the instance.

Why it fails: GuardDuty detects threats but does not perform remediation; it can trigger EventBridge to invoke a Lambda, but the option does not mention that.

Use AWS Shield Advanced to automatically apply the quarantine security group to the instance.

Why it fails: Shield Advanced provides DDoS protection, not instance isolation.

Use AWS Lambda functions triggered by Amazon EventBridge to remove the instance from the security group and attach the quarantine group.

Why it fails: Lambda can modify security groups, but the question asks for a combination; this option omits the triggering mechanism beyond EventBridge, which is not specific to compliance.

Configure the CloudWatch Agent to collect memory and system logs and publish them to CloudWatch Logs.

Why it fails: CloudWatch Agent collects metrics and logs, but not memory dumps.

Use AWS Systems Manager Run Command to execute a script on the instance that collects diagnostics before it is terminated.

Why it fails: Run Command requires the instance to be running and communicating; the instance may fail before the command executes.

Enable EC2 instance metadata service (IMDS) to capture diagnostic data that persists after termination.

Why it fails: Instance metadata is lost when the instance is terminated.

Configure the Auto Scaling group to use ELB health checks and set the health check grace period to 600 seconds.

Why it fails: This delays health checks but does not prevent high CPU instances from being used.

Create a scheduled scaling action to add more instances during peak hours.

Why it fails: Scheduled scaling does not respond to real-time CPU spikes.

Increase the health check timeout to 10 seconds and the interval to 20 seconds to give instances more time to respond.

Why it fails: This would allow slow instances to remain in service longer, potentially worsening the issue.

Rotate the access key by creating a new key and updating the application.

Why it fails: Rotating the access key creates a new key but leaves the original, compromised key active and usable. Simply updating the application does not revoke or deactivate the old key, so the unauthorized party retains the ability to make API calls with it, and the incident remains uncontained.

Change the IAM policy attached to the user to deny all actions.

Why it fails: Changing the IAM policy to deny all actions on the user is ineffective as an immediate mitigation because IAM policy updates are subject to eventual consistency and can take tens of seconds or longer to propagate. Moreover, it does not invalidate the access key itself, so the attacker could still call APIs before the denial propagates, and the change could also break legitimate workloads that rely on the same IAM user.

Contact AWS Support to have the key disabled.

Why it fails: Contacting AWS Support to disable the key is not an immediate action; the support process involves verification and may not be available in a timely manner. The team should take direct actions like deleting the key.

Enable CloudWatch Logs Insights on the Lambda function's log group.

Why it fails: Logs do not capture input payload unless the function logs it.

Use CloudWatch Events to capture Lambda invocation errors.

Why it fails: CloudWatch Events does not capture the payload.

Configure a dead-letter queue (DLQ) for the Lambda function.

Why it fails: DLQs are for asynchronous invocations and capture messages, not payload from S3 events.

Modify the canary script to add more logging.

Why it fails: You cannot modify the pre-built canary scripts that CloudWatch Synthetics provides (e.g., Heartbeat, API canary) to inject additional logging; they are managed runtimes intended for immediate use. For custom diagnostic logging, you would need to author a new custom canary script using the Synthetics SDK, then redeploy it—so modifying the script is not a valid, immediate diagnostics step and does not help you interpret an existing failure.

Examine CloudWatch metrics for the canary.

Why it fails: CloudWatch metrics for a canary, such as SuccessPercent, Duration, and Failed, provide aggregated counts and timing data that tell you a failure happened but not why. They lack the granular diagnostic content needed to debug an error—no endpoint, response payload, stack trace, or specific assertion failure appears in the metric data. Metrics only help you detect the problem, not resolve it.

Check CloudTrail for CanaryRun API calls.

Why it fails: CloudTrail records control-plane API activities such as CreateCanary, StartCanary, and StopCanary, but it does not log the internal execution output or script-level errors of a canary run. A CanaryRun API call in CloudTrail would only indicate that a run was triggered or invoked, not what happened during that run or why it failed, so CloudTrail is not useful for diagnosing a failed canary.

The upload succeeds without encryption.

Why it fails: The upload fails because the IAM policy explicitly denies s3:PutObject when the encryption header is missing. The condition key s3:x-amz-server-side-encryption is checked against the request; if it is null, the Deny statement matches and the request is rejected. Thus, attempting an unencrypted upload will always receive an Access Denied error from S3.

The upload succeeds with SSE-S3 encryption.

Why it fails: Simply enabling SSE-S3 on the bucket does not satisfy the IAM condition; the policy requires the x-amz-server-side-encryption: AES256 header to be present in the PutObject call itself. Even if the object would ultimately be encrypted with SSE-S3 due to bucket default encryption, the request lacks the explicit header at authorization time, so the deny condition triggers. The user must explicitly set the encryption header in the upload request to pass the policy check.

The upload succeeds and is automatically encrypted with SSE-S3.

Why it fails: Automatic encryption via a bucket's default encryption policy cannot override an IAM deny condition. When the user uploads without specifying an encryption header, S3 may apply bucket-level default encryption after authorization, but the IAM policy is evaluated first and rejects the request based on the missing header. Since the condition is a Deny, it takes precedence over any bucket setting, so the upload fails with Access Denied before encryption can be applied.

No action will be taken because the period is 60 seconds.

Why it fails: The period only defines the aggregation interval for the metric; it does not disable alarm actions. With a period of 60 seconds and two evaluation periods, the alarm requires two consecutive minutes of the metric breaching its threshold before entering ALARM. Once in ALARM state, the configured recovery action fires, so the period does not prevent the instance from being recovered.

The instance will be rebooted.

Why it fails: A reboot action would restart the existing operating system from the same underlying host, and is typically used for instance status check failures (e.g., OS crashes). However, this alarm's configured action is explicitly the recovery action, not a reboot. CloudWatch executes only the action that is defined in the alarm, and the note indicates recovery is the selected action, so the instance will not be rebooted.

The instance will be stopped.

Why it fails: The stop action would hibernate or stop the instance, which is a separate EC2 lifecycle action that can be attached to a CloudWatch alarm. The alarm in the exhibit has a recover action associated with it, not a stop action. Furthermore, an instance recovery automatically migrates the running instance to new hardware without transitioning through the stopped state, so stopping does not occur.

The AWS WAF web ACL is blocking requests from that region.

Why it fails: AWS WAF blocking requests typically returns HTTP 403 Forbidden, which is a 4xx client error, not a 5xx server error. Therefore, it would not cause a high 5xx error rate.

The S3 bucket policy does not grant access to the CloudFront origin access identity (OAI).

Why it fails: Missing OAI permissions cause S3 to return 403 Access Denied, which CloudFront forwards as a 403 error to the viewer (4xx), not a 5xx error. Thus, this does not explain the 5xx errors.

The CloudFront distribution's default TTL is too short.

Why it fails: Default TTL being too short would increase the frequency of requests to the origin, potentially causing performance issues, but it does not directly cause 5xx error rates.

The Auto Scaling group's lifecycle hooks are not configured.

Why it fails: Lifecycle hooks control what happens during instance termination at the EC2 Auto Scaling level, letting you run cleanup scripts before shutdown, but they do not influence how the Application Load Balancer routes traffic or how existing connections are handled. Even with hooks configured, the instance is eventually terminated and the ALB's deregistration process still closes connections after the configured delay, so missing hooks cannot be the cause of sessions being dropped specifically when a deployment replaces instances.

The stickiness duration is set too low.

Why it fails: A low stickiness duration causes the load balancer to stop pinning a client to one backend after a short period, so sessions could naturally migrate earlier, but this is a per-request cookie timeout independent of instance replacement. During a rolling deployment the ALB explicitly deregisters instances and begins draining them; an expired stickiness cookie would not make the ALB send traffic to a terminating instance in the first place, so this does not explain active sessions being cut off at deployment time.

The target group's health check interval is too long.

Why it fails: The health check interval controls how frequently the ALB sends health checks and how quickly it marks an instance unhealthy. A longer interval delays the removal of a failing or terminating instance from rotation, so it would cause slower failover and perhaps a longer period of errors, not the abrupt termination of active sessions, because the ALB continues to send traffic to the instance until then and existing connections are not affected by health check timing.

Security group inbound rules for the ALB

Why it fails: Inbound rules affect client access, but 503 is from ALB.

SSL certificate expiration on the ALB

Why it fails: SSL issues cause 502 or 400, not 503.

Enable AWS Organizations to block public access

Why it fails: Enabling AWS Organizations to block public access is not a valid modification to the CloudFormation template itself; it is an organization-level setting configured via service control policies (SCPs) in AWS Organizations, not in the template. Moreover, blocking public access organization-wide does not selectively allow a specific AWS account—it would deny all public access, including the cross-account access the bucket policy is intended to permit. To fix the template, the bucket policy must explicitly identify the allowed account as the Principal.

Change the Principal to "*" and add a condition for aws:SourceIp

Why it fails: Keeping the Principal as "*" even with an aws:SourceIp condition leaves the bucket effectively public to any user or service from the allowed IP ranges, and it fails to restrict by identity. This is particularly dangerous in a CloudFormation context because requests originating from AWS services may not carry the expected source IP, and cross-account access from the intended account could be blocked if the IP is not included. The condition also does not prevent unintended access from other principals that happen to originate from a trusted IP, so it is not a viable substitute for an account-scoped Principal.

Set the bucket's 'BlockPublicAccess' property to true

Why it fails: Setting the bucket's BlockPublicAccess property to true is a valid security best practice, but it does not by itself grant access to the specific AWS account. If the bucket policy still uses Principal "*" and you enable BlockPublicAccess with BlockPublicPolicy, AWS will reject the public policy, leaving the bucket inaccessible to the intended account; if you change the Principal to the allowed account, BlockPublicAccess is unnecessary for cross-account access because that policy is not public. Thus, this option does not modify the policy in the required way and is an incomplete answer.

VPC Flow Logs

Why it fails: VPC Flow Logs capture only network traffic metadata at the ENI level, such as source/destination IPs, ports, and packet decisions, and they contain no visibility into RDS control-plane operations. A Multi-AZ failover is a service-level lifecycle event orchestrated by the RDS infrastructure, not a network protocol anomaly, so flow logs would show zero relevant entries for the failover itself. They are useful for troubleshooting connectivity, not for auditing database availability events.

RDS MySQL error logs

Why it fails: RDS MySQL error logs record engine diagnostics such as crashes, startup/shutdown, replication errors, and authentication failures. While a failover might eventually cause errors on the old primary, the failover trigger is an infrastructure decision made by the RDS service (e.g., health detection or manual reboot with failover), not an event written to the MySQL error log. The error log may indicate secondary symptoms but cannot provide the authoritative reason or the control-plane action that initiated the failover.

RDS slow query logs

Why it fails: RDS slow query logs capture SQL statements whose execution time exceeds `long_query_time`, and are designed solely for performance tuning and index optimization. They are database-engine output and have no relationship to RDS instance lifecycle events such as Multi-AZ failover. Even if queries slow down after a failover, the slow query log would only show the queries, not the failover event or its cause, making it irrelevant for diagnosing why the failover occurred.

Contact AWS Support to fix the stack

Why it fails: Contacting AWS Support is not the appropriate first step because CloudFormation provides the ContinueUpdateRollback API as a self-service recovery operation for stacks in UPDATE_ROLLBACK_FAILED. Support cannot directly modify your stack resources; they would likely guide you to use the same API or troubleshoot underlying resource issues. Only if the API fails or reveals a service-side issue should you escalate to Support.

Delete the stack and recreate it

Why it fails: Deleting the stack is usually impossible while it is in UPDATE_ROLLBACK_FAILED because the stack is not in a terminal state, and DeleteStack typically requires a stable state or will fail when resources are inconsistent. Even if deletion were attempted, it could orphan or partially delete resources, and any dependencies referencing those resources would break. The correct approach is to first resume and complete the rollback, then delete or update cleanly.

Submit another stack update with the desired configuration

Why it fails: CloudFormation explicitly prevents submitting a new stack update while the stack is in the UPDATE_ROLLBACK_FAILED state; UpdateStack will return a ValidationError and reject the request. The stack must be moved to a stable state (such as UPDATE_ROLLBACK_COMPLETE) via ContinueUpdateRollback before any subsequent update operation can be initiated. Attempting a new update without recovering the stack only wastes time and does not address the failed rollback.

Amazon GuardDuty

Why it fails: Amazon GuardDuty is a security/threat detection service that uses machine learning, anomaly detection, and threat intelligence feeds to analyze AWS CloudTrail events, VPC Flow Logs, and DNS logs. It identifies malicious activities such as credential compromise, crypto mining, or unusual network behavior, but it has no visibility into application-level performance metrics like Lambda duration, API Gateway latency, or error rate. For an incident response plan focused on operational performance degradation, GuardDuty would only be relevant if a security event is the root cause, not as a primary monitoring tool for service health.

AWS Config

Why it fails: AWS Config is a service that records and evaluates the configuration state of AWS resources against desired policies, enabling compliance auditing, change management, and security governance. It tracks things like whether a Lambda function's memory size changed, whether an API Gateway stage has logging enabled, or whether a security group rule was modified. It does not collect or analyze runtime performance data such as request latency, error rates, or throttling, so it cannot detect ongoing performance degradation that would trigger an incident response.

AWS CloudTrail

Why it fails: AWS CloudTrail records API call history for an AWS account, capturing details like who made the call, from which IP address, when, and what parameters were used. This audit log is invaluable for compliance, security investigations, and understanding which actions altered infrastructure or IAM permissions. However, CloudTrail does not capture application-level performance telemetry such as Lambda invocation duration, API Gateway response latency, or HTTP error counts, so it cannot proactively alert on performance incidents; it is useful only for forensic analysis after an incident has been detected.

Terminate the instance immediately

Why it fails: Terminating the instance immediately is harmful because the default EBS root volume is deleted upon termination (unless DeleteOnTermination is false), and any ephemeral instance store data, memory, and process artifacts are lost forever. The incident response team would lose critical forensic evidence such as running malware binaries, open network connections, and cryptographic material, making attribution and root-cause analysis impossible. Additionally, termination may trigger Auto Scaling replacement, potentially letting the same compromise re-spawn before containment is complete.

Create an AMI of the instance for forensic analysis

Why it fails: Creating an AMI is an indirect and risky forensic method: it captures the EBS volume state but fails to preserve volatile memory, active process listings, or network connections, which are often the most valuable evidence. The AMI itself becomes a potential vector for accidentally deploying the compromised environment later, and the snapshot process may be inconsistent if the instance is actively writing to disk. A direct snapshot of the root EBS volume, taken after stopping the instance, is the preferred method for preserving disk evidence without the overhead of a reusable image.

Immediately terminate the instance to trigger a replacement

Why it fails: Terminating without verification may cause unnecessary disruption.

Do nothing, because the Auto Scaling group will automatically replace the instance

Why it fails: Auto Scaling replaces only if the instance is terminated or marked unhealthy.

Check the security group rules to ensure they allow traffic

Why it fails: Status checks are not affected by security groups.

Manually stop the rollback and continue with the update

Why it fails: Stopping a rollback and trying to continue the update is invalid because CloudFormation does not provide a mechanism to resume an update after a failure once rollback is in progress. Even if you manually stop the rollback, the stack remains in the UPDATE_ROLLBACK_COMPLETE state with the failed resources potentially stuck or partially updated, and any subsequent update will still fail unless the underlying issue is resolved. This approach leaves the infrastructure in an undefined state and ignores the root cause.

Re-launch the stack with the same template

Why it fails: Re-launching the stack with the same template is ineffective because if the template, parameters, or region are unchanged, the update will encounter the exact same failure condition. This brute-force retry does not provide any additional diagnostic information and may also fail on a transitory resource dependency that could have been identified by inspecting the stack events. The correct first step is to troubleshoot the actual error rather than repeat the operation blindly.

Delete the stack and re-launch it

Why it fails: Deleting the stack and re-launching it is an overly destructive action that removes all managed resources, potentially including data volumes or production databases, and does nothing to diagnose the failure. A re-launch using the same template will almost certainly reproduce the same error, making this approach both risky and ineffective. It should never be used as a troubleshooting method when non-destructive diagnostics like stack events are available.