Hard Difficulty Questions

ECS service auto recovery

Why wrong: ECS service auto recovery, also known as automatic recovery for Amazon ECS, detects when a task or the underlying EC2 instance fails due to hardware or infrastructure issues and restarts the affected tasks on a healthy host. It does not evaluate deployment progress, validate a new task definition, or trigger a rollback when an application version fails to deploy. Consequently, it is effective for instance-level resilience but entirely unrelated to deployment failure handling.

ECS managed scaling

Why wrong: ECS managed scaling leverages Application Auto Scaling or Cluster Auto Scaling to adjust the number of running tasks based on demand metrics such as CPU utilization, memory pressure, or request count per target. It neither monitors the health of an actively updating deployment nor decides whether a new task definition is fit for production, and it cannot revert to a prior version when a rollout fails. Therefore, while it maintains capacity, it is not a deployment health mechanism.

CloudWatch alarm actions

Why wrong: CloudWatch alarm actions allow you to invoke an action, like an auto scaling policy or an SNS topic, when a metric threshold is breached, but ECS does not natively treat a CloudWatch alarm as a deployment rollback signal. To perform a rollback with CloudWatch, you would need to custom-build a Lambda function or a separate orchestration process that modifies the service, which is neither native nor automatic to the deployment lifecycle. The deployment circuit breaker, in contrast, integrates alarm monitoring directly into the ECS update process.

Use direct database connections from each microservice pod.

Why wrong: Direct connections increase connection count and can overwhelm the database.

Use Amazon ElastiCache for Redis to cache database query results.

Why wrong: Caching reduces read load but does not manage connections.

Use AWS CloudFormation with a ChangeSet and a custom rollback Lambda function triggered by CloudWatch alarms.

Why wrong: CloudFormation does not have native blue/green deployment with automatic rollback for ECS.

Use AWS CodeBuild to run a build that creates a new task definition, then update the ECS service manually, and use CloudWatch alarms to trigger a rollback via a Lambda function.

Why wrong: CodeBuild is not designed for deployment orchestration.

Use Amazon ECS service auto scaling with step scaling policies based on CloudWatch alarms.

Why wrong: Auto scaling adjusts capacity, not deployment strategies.

The condition StringEquals on InstanceType is incorrectly formatted.

Why wrong: The condition is syntactically correct.

The pipeline does not have permission to call ssm:GetParameter because the resource is not specified.

Why wrong: The action is allowed on all resources.

The policy must include a 'Resource' for the 'ec2:DescribeInstances' action to be valid.

Why wrong: DescribeInstances can be allowed on all resources.

Create a CloudWatch Logs log group for the error metric.

Why wrong: Log group already exists.

Create a CloudWatch Logs subscription filter to stream errors to a Lambda function that calculates the error rate.

Why wrong: A CloudWatch Logs subscription filter streams log data in real time to a Lambda function, but it does not provide the metric filter needed to continuously evaluate the error rate against a threshold over a 5-minute window. The requirement is for a metric filter that emits a metric to CloudWatch Metrics, enabling a CloudWatch Alarm to compare the error proportion against total requests. This option is tempting because subscription filters are commonly used for real-time log processing and alerting, and would be correct if the team needed to perform custom, stateless transformations on each log event rather than aggregate a rolling error-rate metric.

Create a CloudWatch metric for the error count.

Why wrong: Metric filter creates the metric automatically.

AWS CloudTrail and Amazon EventBridge

Why wrong: AWS CloudTrail is an audit service that records API activity but doesn't evaluate security group configurations against desired compliance standards. While EventBridge can route CloudTrail events to targets, it has no native remediation action; any corrective workflow would require an additional compute service like Lambda or Step Functions to interpret and modify SG rules, making this pair incomplete for automated remediation.

Amazon GuardDuty and AWS Lambda

Why wrong: Amazon GuardDuty is a threat detection service that analyzes findings from sources like VPC Flow Logs and DNS logs; it doesn't assess whether a security group's inbound rules violate corporate security policies. Lambda can execute remediation code, but without a Config rule or similar trigger, GuardDuty won't generate the necessary SG compliance findings, so the combination fails to detect the original non-compliant state. For SG configuration issues, GuardDuty is the wrong detective control.

AWS Config and Amazon Simple Notification Service (SNS)

Why wrong: AWS Config correctly detects SG rule violations via managed or custom rules, and it can send evaluation results to an SNS topic. However, SNS is a pub/sub notification service that only delivers messages to subscribers (e.g., email or chat) and cannot invoke a remediation workflow on its own; a human must manually act on the notification. For automated remediation, Config must be integrated with AWS Systems Manager Automation or another execution service, not just SNS.

The upload fails because the bucket policy requires the object ACL to be set, but the default ACL allows the upload anyway.

Why wrong: This statement misunderstands how bucket policy conditions are evaluated. The policy's Condition element checks the x-amz-acl request header at the time of the request, not the object's ACL after S3 applies its default. Since the upload omitted the required header, the condition fails immediately, and the default ACL behavior (which would otherwise grant a private ACL) is irrelevant. An unsatisfied condition in an allow statement results in an implicit deny, so the upload fails.

The upload succeeds because the bucket policy does not explicitly deny the request.

Why wrong: In AWS S3 authorization, an action is allowed only if an applicable policy statement explicitly permits it; if no statement matches, the default result is deny. The bucket policy's only allow statement for s3:PutObject includes a condition that the request must carry the x-amz-acl header set to bucket-owner-full-control. Because the condition is not satisfied, the statement does not apply, so the request falls through to an implicit deny. The absence of an explicit Deny statement does not create an allowance when no allow statement is effective.

The upload succeeds because the bucket policy allows s3:PutObject for any principal.

Why wrong: This option incorrectly treats the Principal and Action elements as unconditional. While the policy statement may specify Principal * and s3:PutObject, it also contains a Condition element requiring the x-amz-acl header to equal bucket-owner-full-control. When a Condition is present, all conditions must be satisfied for the allow to take effect; otherwise the statement is a no-op for that request. Therefore, the policy does not permit 'any principal' without the required header, and the upload is denied rather than allowed.

The RDS instance has drifted from the template definition.

Why wrong: Drift alone does not cause replacement during update; it is reported separately.

The deletion policy is set to 'Retain'.

Why wrong: Deletion policy affects what happens when a resource is deleted, not replacement.

The stack policy prevents updates to the RDS instance.

Why wrong: Stack policy can allow or deny updates, but not cause replacement.

Use a Network Load Balancer with TLS listeners and target groups.

Why wrong: Incorrect. A Network Load Balancer (NLB) with TLS listeners does not provide a security policy to enforce specific TLS versions and ciphers; it only supports basic TLS termination. ALB offers predefined security policies for granular control.

Place a CloudFront distribution in front of the ALB and configure the origin protocol policy.

Why wrong: Incorrect. CloudFront is used for content delivery and caching; it can be placed in front of an ALB, but its origin protocol policy only controls encryption between CloudFront and the ALB, not the overall TLS version and cipher enforcement. This approach is less efficient and adds latency.

Install a self-signed certificate on each EC2 instance and configure the web server.

Why wrong: Incorrect. Installing self-signed certificates on each EC2 instance is inefficient and hard to manage at scale. It does not provide centralized enforcement of TLS 1.2 and specific ciphers across all traffic.

Launch a proxy server in a public subnet and route traffic through it.

Why wrong: A proxy server such as an EC2 instance in a public subnet can forward outbound HTTP/HTTPS traffic, but it requires manual configuration (e.g., setting HTTP_PROXY environment variables) and is a single point of failure unless you deploy and manage multiple instances. You must handle OS patching, monitoring, auto-scaling, and throughput capacity. It also creates a predictable bottleneck, and a NAT Gateway—a managed, redundant service—offers the same functionality with far less operational burden, making the proxy approach less scalable and more costly to operate.

Assign an Elastic IP to the instance in the private subnet.

Why wrong: Assigning an Elastic IP to a private subnet instance is ineffective because private subnets have no route to an Internet Gateway. The Elastic IP is a static public address, but without an IGW route in the subnet, AWS will not advertise a public route for that IP; the instance's traffic cannot leave the VPC, and the public IP is ignored. To make an Elastic IP usable, the instance must reside in a public subnet, which contradicts the requirement of keeping the instance private. Thus this option fails to provide any outbound internet connectivity.

Use a VPC endpoint to the Amazon Linux repository.

Why wrong: A VPC endpoint provides private connectivity to AWS services such as S3, DynamoDB, or API Gateway, not to arbitrary internet destinations like the Amazon Linux yum repository. The Linux package repositories are hosted on the public internet, and VPC endpoints will not route to them, even if you create a gateway endpoint or interface endpoint. Without a NAT gateway or a proxy with internet access, the private instance cannot download updates or install packages. Therefore this option does not meet the requirement for accessing the Amazon Linux repository.

Increase the instance class to improve failover performance.

Why wrong: Increasing the instance class only scales compute and memory capacity; it does not affect how quickly Amazon RDS detects a primary failure or promotes the standby in Multi-AZ deployments. Failover time depends on database health checks, crash recovery, and the time to update the DNS endpoint, not on whether you use a smaller or larger instance type. In fact, a larger instance could even take longer to restart if logs and buffers need recovery.

Use a Route 53 weighted record with health checks to point to both instances.

Why wrong: Route 53 weighted records with health checks cannot provide seamless RDS failover because the RDS Multi-AZ architecture uses a single primary DNS endpoint; the standby is not a separate endpoint you can route traffic to. Even if a health check marked the primary as unhealthy, the application would still need to continue writing to the new primary, but a weighted record set would require the app to manually switch or load balance between endpoints, which is not appropriate for a synchronous write workload. RDS automatically updates its own DNS endpoint on failover, so adding another DNS routing layer adds complexity, potential TTL propagation delays, and does not preserve existing connection states.

Configure the application to use the read replica endpoint for writes.

Why wrong: Read replicas in Amazon RDS are read-only by design; they use asynchronous replication from the primary and cannot accept any write statements. Sending application writes to the read replica endpoint would immediately fail with an error because the database engine enforces the replica's read-only status. Furthermore, even if you were to attempt to promote a read replica to a primary, that is a separate process and does not happen automatically during a Multi-AZ failover, so the application would have no consistent write target.

Increase the deployment timeout in CodeDeploy.

Why wrong: Increasing the CodeDeploy deployment timeout does not fix the problem because the application_start.sh hook fails immediately with a non-zero exit status, not because it exceeds a time limit. The timeout only sets the maximum duration that CodeDeploy waits for a deployment to complete; it does not retry or ignore a failed lifecycle script. This change would simply delay the eventual failure while the script still errors out on the missing path.

Remove the application_start.sh script from the AppSpec file.

Why wrong: Removing application_start.sh from the AppSpec file would eliminate the immediate failure, but it would also prevent the application from being started during the deployment, leaving the service down and defeating the purpose of the ApplicationStart lifecycle event. This action masks the underlying bug rather than fixing it, and it could lead to a successful deployment with an unusable application, which is worse than a failed deployment that alerts the team. The correct approach is to repair the script logic, not to remove the hook entirely.

Reinstall the CodeDeploy agent on all instances.

Why wrong: Reinstalling the CodeDeploy agent on all instances is unnecessary because the agent itself is functioning correctly; the evidence shows it is executing application_start.sh and relaying its non-zero exit status. The failure originates in the user-authored script, not in the agent's execution framework, and reinstalling will not change the absence of the path or the script's logic. This action would consume time and resources without affecting the actual root cause, and it would not prevent the same script failure from recurring after redeployment.

StartInstances on an instance in us-east-1

Why wrong: Even though the request targets an instance in us-east-1, the policy contains no statement that explicitly allows the ec2:StartInstances API action; its Allow statement is scoped only to DescribeInstances. IAM defaults to an implicit deny for any action not matched by an explicit Allow, and the Deny statement in the policy does not mention StartInstances at all. Therefore the request is denied regardless of the instance's region or other conditions.

DescribeInstances on an instance in us-west-2

Why wrong: The policy's Deny statement is designed to block DescribeInstances specifically when the value of the ec2:Region condition key is 'us-west-2', and that condition evaluates to true for this request. Although the Allow statement would otherwise apply to DescribeInstances, an explicit Deny always overrides any Allow in IAM's evaluation logic. As a result, the request is denied even though the action itself is generally permitted by the policy.

StopInstances on an instance in us-east-1

Why wrong: The request fails because the policy only grants DescribeInstances; no statement includes ec2:StopInstances, so the action is not explicitly allowed. IAM does not infer permissions from related actions—having permission to describe an instance does not grant permission to modify its state. Since no explicit Allow matches StopInstances, the policy's default implicit deny applies and the request is rejected.

Disable public access to the S3 bucket using block public access settings.

Why wrong: Applying Block Public Access settings removes all public internet access to the log bucket, which is a standard security hygiene measure to prevent anonymous reads or writes. This control does not restrict IAM principals inside the AWS account who have s3:PutObject or s3:DeleteObject permissions, so they can still manipulate log objects. It also does not generate any detectable hash or digest to verify the logs have not been changed, so it is not a log-integrity solution.

Enable MFA delete on the S3 bucket.

Why wrong: MFA delete requires a second authentication factor to permanently delete object versions or change the bucket's versioning state. While it makes destructive operations like deleting log files harder, it does not detect or prevent an authenticated user with PutObject permissions from overwriting existing CloudTrail log objects. It also offers no cryptographic proof that log content has not been altered, so it fails to provide the forensic assurance required.

Use AWS Backup to create backups of the CloudTrail S3 bucket.

Why wrong: AWS Backup can back up the S3 bucket that stores CloudTrail logs for disaster recovery, allowing you to restore logs after accidental deletion or regional data loss. However, the backup copies are point-in-time snapshots, and CloudTrail's digest files are not included in a way that anchors them to the original log chain, so a restored bucket cannot prove the logs were tampered with before the backup. An attacker could alter logs and then a later backup would store the tampered data, making this an availability control, not a forensic integrity control.

The condition key 'aws:SecureTransport' is misspelled; it should be 'aws:SecureTransport' with a capital 'T'

Why wrong: The condition key is correct.

The bucket is not versioned, so the policy does not apply to object versions

Why wrong: Versioning does not affect the HTTPS enforcement.

The policy uses Deny, but an Allow policy from another statement overrides it

Why wrong: Deny always overrides Allow.

Create a new IAM group with a deny-all policy and add all users to it.

Why wrong: This fails because the root user is not an IAM persona; it cannot be added to an IAM group, and IAM group policies have no effect on root credentials. While an explicit deny group would block listed IAM users, any user omitted from the group, or any IAM role created later, remains unaffected, and the root user retains unrestricted administrative access. An SCP is the only mechanism that can constrain the root user in an organizational unit or account.

Attach an IAM policy denying all actions to all IAM users in that account.

Why wrong: This only locks down IAM identities; the root user is not subject to IAM user or group policies and cannot have an IAM policy attached. Because the root user has administrative-level access by default, it would remain fully capable of performing actions in the affected account, including creating new IAM users or modifying the deny policy. To restrict root, an organization-level SCP must be used, as IAM policies alone cannot constrain root credentials.

Apply an SCP that denies all actions to the root user only.

Why wrong: This leaves a glaring gap: although the root user is restricted, every IAM user, group permissions, and role still retain their existing allowed actions. In a multi-account organization, an attacker who compromises an IAM user (or the affected account's own operations staff) could continue to perform actions, create new credentials, and exfiltrate data, so the incident response goal of full containment is not met. A comprehensive SCP must target all principals, not only the root user.

Use CloudWatch Logs Insights to query container logs

Why wrong: CloudWatch Logs Insights is a query engine for searching and analyzing log event payloads, so it can reveal application errors or response codes, but it does not natively ingest or correlate numeric time-series metrics like ECS CPU utilization, nor can it join those logs to X-Ray trace segments. To triangulate a problem you would need to manually export timestamps, metric data, and trace IDs, which misses the point of a unified observability correlation. This makes it a poor choice for directly linking a container's CPU spike to a specific request trace.

Configure CloudWatch Synthetics canaries to measure latency

Why wrong: CloudWatch Synthetics canaries are scheduled Node.js or Python scripts that simulate end-user requests against public or internal HTTP endpoints, measuring availability, latency, and screenshot capture; they do not expose container-level CPU or memory metrics at all. Even if a canary runs inside a VPC and hits an internal load balancer, the result is a black-box external measurement of response time, not a host or task-level CPU telemetry feed. Therefore, while it can detect a slowdown, it cannot identify that the cause is the ECS container's CPU saturation, and it cannot associate a trace with that metric.

Set up a Prometheus server on an EC2 instance to scrape container metrics

Why wrong: Standing up a Prometheus server on an EC2 instance means self-managing service discovery, scrape targets, persistent storage, retention policies, and alerting infrastructure, which is far more operational overhead than the AWS-native Container Insights and ServiceLens combination. Additionally, Prometheus metrics are just one part of observability; you would still need a separate tracing solution like AWS X-Ray and build your own correlation mechanism between the Prometheus CPU gauge and trace IDs. AWS offers a managed Prometheus-compatible service, but for this scenario the native CloudWatch and X-Ray integration yields correlated metrics and traces without additional servers.

Switch from rolling to immutable updates.

Why wrong: Immutable updates launch a fully new Auto Scaling group with the new version and only shift traffic after all instances pass health checks. While this strategy can improve overall deployment safety, it does not address the root cause of the problem: the application needs more time to stabilize before its first health check. The same aggressive health check interval applies to the new immutable instances, so a slow-starting application will still be marked unhealthy and replaced. Therefore, changing the deployment strategy is an unrelated and heavier-handed change than simply tuning the health check timing.

Increase the batch size to 75%.

Why wrong: Raising the batch size to 75% means that a larger fraction of your environment's instances are updated and taken out of service at the same time. This simultaneously reduces the capacity available to serve production traffic and magnifies the impact of any single instance failing a health check during the deployment. With more instances churning at once, the ELB is more likely to see enough failures to trigger scaling actions or reject traffic, making the deployment less stable, not more. The existing configuration was likely already causing too many instances to fail at once; increasing the batch size worsens that problem.

Decrease the deployment cooldown time.

Why wrong: The deployment cooldown is a deliberate pause between batches that gives instances time to complete startup, pass health checks, and begin serving traffic before the next batch is touched. Decreasing this cooldown means the next batch starts before the current instances have fully stabilized, compounding startup latency and health-check failures across overlapping batches. This premature progression can cause multiple instances to fail health checks concurrently, leading to a cascading failure. Shortening cooldown is therefore the opposite of what is needed; you would want more time, not less, for instances to recover.

Make the S3 bucket publicly accessible to allow CodeDeploy to write objects.

Why wrong: Making the S3 bucket publicly accessible opens the website bucket to anonymous read and, more importantly, anonymous write if you allow s3:PutObject for everyone. The CodeDeploy/CodePipeline service role cannot automatically write to a public bucket without explicit IAM authorization; public access policies only apply to anonymous principals, not to the authenticated deployment role. This removes all access control and violates least privilege, making it both insecure and an ineffective fix.

Add a step in CodeBuild to copy the artifacts to the S3 bucket using the AWS CLI.

Why wrong: Adding a CodeBuild step to copy artifacts with the AWS CLI sidesteps the S3 deployment action but does not resolve the underlying authorization failure that caused the pipeline to fail. It also introduces a new requirement to attach s3:PutObject permissions to the CodeBuild service role, so you still need a policy fix, and you lose CodePipeline's built-in deployment tracking. This is a workaround that obscures the root cause, not a correction to the deployment configuration.

Change the Object Ownership setting to 'ObjectWriter' to enable ACLs.

Why wrong: Changing Object Ownership to ObjectWriter enables ACLs and makes the account that uploads an object the owner, but it does not grant CodeDeploy's role any permission to bypass the missing bucket policy or ACL. It also re-enables ACLs, which are considered a legacy permission mechanism and can create confusion when cross-account uploads occur. AWS recommends keeping Object Ownership at BucketOwnerEnforced and using bucket policies instead, so this change would be a security regression without fixing the deployment action.

Manually update the resources to match the previous template, then resume the rollback.

Why wrong: Manually updating the resources to match the previous template does not resume CloudFormation's rollback operation; the stack remains in UPDATE_ROLLBACK_FAILED state. Any manual changes outside CloudFormation create drift and risk inconsistencies. The designed recovery is to fix the root cause and use ContinueUpdateRollback, which lets CloudFormation finish the rollback in a controlled manner.

Use CloudFormation's 'SignalResource' to manually complete the rollback.

Why wrong: SignalResource is not used for rollback; it is a cfn-signal mechanism to coordinate provisioning of a specific resource that has a CreationPolicy or WaitCondition. It cannot manually complete a rollback. For a stack in UPDATE_ROLLBACK_FAILED, the correct operation is ContinueUpdateRollback, not a signal.

Delete the stack and recreate it from the last known good template.

Why wrong: Deleting the stack and recreating it from the last known good template is destructive and often fails because the stack is in UPDATE_ROLLBACK_FAILED, and some resources (like DynamoDB tables with deletion protection) cannot be deleted. This approach can cause data loss or retain orphaned resources, and it does not address the cause of the rollback failure. ContinueUpdateRollback is the supported, non-destructive recovery method.