Hard Difficulty Questions

Use io2 EBS volumes because they provide the highest durable block storage performance.

Why it fails: io2 volumes attach to a single EC2 instance in one Availability Zone, so they cannot be mounted concurrently by multiple instances across zones. It is tempting because io2 delivers the highest durable block IOPS, which suits a single latency-sensitive database rather than this shared multi-instance workload.

Use Amazon EFS for the workspace so the temporary files survive instance replacement.

Why it fails: EFS is a shared POSIX file system that persists independently of instance lifecycle, so files already survive instance replacement without EFS. It is tempting because it offers durable shared storage, but the requirement is a high-performance local working directory, which instance store provides.

Use S3 as the working directory and read and write the intermediate files directly there.

Why it fails: S3 is object storage accessed over HTTP, so it cannot host a POSIX working directory that the application reads and writes as local files. It is tempting as durable, cheap shared storage, but the workload needs a mounted file system, which Amazon EFS provides.

Change both records to weighted routing with a 50/50 split so Route 53 can shift traffic gradually.

Why it fails: Weighted routing distributes traffic according to assigned weights (e.g., 50/50) and does not incorporate health checks, so Route 53 would continue to send a portion of traffic to the primary even if it is healthy, while also sending half to the secondary. This split does not implement the required active-passive failover model, because no automatic switch occurs when the primary becomes unhealthy. The symptom of 'still sends traffic to primary' would persist because weighted routing does not evaluate endpoint health when choosing answers. For health-based failover, Route 53 needs failover routing with a health check attached to the primary record.

Switch to latency-based routing so users are always directed to the lowest-latency Region.

Why it fails: Latency-based routing selects the region that offers the lowest network latency for the specific client, optimizing user experience rather than enforcing a business preference for a primary region. It does not use health checks to fail over; if the primary endpoint remains the lowest-latency target even while unhealthy, Route 53 will continue directing traffic there, matching the reported symptom. This policy cannot implement a strict active-passive model because it lacks a primary/secondary hierarchy and would not automatically shift traffic solely on health status. A failover routing policy with a health check on the primary record is required to achieve the desired behavior.

Use geolocation routing so clients in one Region are sent to the healthier endpoint.

Why it fails: Geolocation routing determines DNS responses based on the geographic location of the requesting client, not on the operational health of the target endpoints. Even if one region is 'healthier,' Route 53 would continue to serve the record mapped to each client's location, ignoring any health-check failures on the primary. This approach cannot express the requirement of preferring one region and failing over only when it is unhealthy, because geo policies do not include health evaluation in their decision logic. The observed behavior of traffic still going to the primary would remain unchanged unless the client’s location happens to map elsewhere.

Replace the instances with a larger instance family so each server has more headroom.

Why it fails: Vertical scaling raises per-instance capacity but leaves the fleet count unchanged, so peak concurrency still queues on the same number of servers. It is tempting because bigger instances feel like added headroom, and it would be correct if the exhibit showed CPU-bound instances rather than a fleet-size shortfall.

Use scheduled scaling to add instances only during the business hours peak window.

Why it fails: Scheduled scaling adds capacity at fixed clock times regardless of actual demand, so latency spikes outside the window remain unaddressed. It is tempting because it avoids permanent overprovisioning, and it would be correct for predictable diurnal load rather than the exhibit's variable peak traffic.

Replace the ALB with a Network Load Balancer to reduce request latency.

Why it fails: A Network Load Balancer operates at layer 4 and cannot terminate HTTP or route by path, so it removes the application-layer routing the workload depends on. It is tempting because NLB offers ultra-low latency, and it would be correct for TCP or UDP workloads needing extreme throughput.

Increase the Lambda timeout so cold starts have more time to complete.

Why it fails: Raising the Lambda timeout only extends the maximum time a function can run before being terminated; it does nothing to reduce the time required to initialize a new execution environment. A cold start comprises downloading the code, starting the runtime, and running initialization code—all of which happen before the handler executes, regardless of the configured timeout. In fact, a longer timeout could worsen user-perceived latency by allowing slow executions to continue, but it cannot prevent the initial invocation from suffering a multi-second init duration.

Move the function behind an Application Load Balancer to improve warm-up behavior.

Why it fails: Placing the function behind an Application Load Balancer merely changes how HTTP requests are routed to Lambda; the ALB itself does not perform any warm-up or pre-initialization of Lambda execution environments. Cold starts originate from the Lambda service when it must create a new sandbox and load the function package—a process that is independent of the front-end service. ALBs do offer connection keep-alive, but that only affects network-level connections, not the lifecycle of the underlying compute environment, so first requests would still experience the exact same initialization latency.

Increase the function memory to the maximum value and leave concurrency unchanged.

Why it fails: Increasing the function memory allocates proportionally more CPU and can make the execution phase faster, but it does not eliminate the initial environment setup time. The cold-start penalty is driven by initialization steps such as loading the code and starting the runtime, which must occur for every new execution environment no matter how much memory is configured. Even at the maximum memory setting, a fresh container still incurs an init duration before your handler starts, so without pre-warming or provisioned concurrency the spike would still see slow first responses.

S3 Glacier Flexible Retrieval: Data accessed infrequently but requires rapid access when needed; retrieval time milliseconds

Why it fails: S3 Glacier Flexible Retrieval is an archival storage class with data access times ranging from 1 minute to several hours, not milliseconds as the scenario requires. The scenario describing rapid access with millisecond retrieval matches S3 Standard-IA, so assigning Glacier Flexible Retrieval is incorrect because it cannot meet the sub-second latency requirement and would also introduce unnecessary retrieval costs.

S3 Glacier Deep Archive: Data that can be retrieved within 1-5 minutes

Why it fails: S3 Glacier Deep Archive is the lowest-cost storage class, intended for long-term archival data with retrieval times of 12–48 hours. The scenario specifies data that can be retrieved within 1–5 minutes, which is characteristic of Glacier Flexible Retrieval, not Deep Archive; hence this option is wrong as it would fail to satisfy the defined retrieval-time constraint.

Enable Origin Shield and keep the current cache behavior unchanged.

Why it fails: Origin Shield adds a regional caching layer that reduces origin load, but it does not change the cache key. If cookies and query strings are still forwarded, each viewer's request remains distinct and the hit ratio is unchanged. Origin Shield suits reducing origin fetches for cacheable content.

Move the static assets to individual presigned URLs for each viewer.

Why it fails: Presigned URLs are per-object, time-limited credentials; each viewer receives a distinct URL, so CloudFront treats every request as unique and caches nothing, lowering the hit ratio. Presigned URLs suit granting temporary access to private S3 objects, not public static asset delivery.

Increase the CloudFront default TTL to 24 hours while continuing to forward all cookies and query strings.

Why it fails: Forwarding all cookies and query strings makes the cache key unique per viewer, so objects are never reused and the hit ratio stays low regardless of TTL. Raising TTL helps only when the cache key is already stable. Long TTLs suit static assets with no per-user variance.

Internet gateway attached to the VPC

Why it fails: An internet gateway only enables public IPv4 connectivity for resources with public addresses; private subnets have no route to it, so S3 and Parameter Store traffic cannot traverse it. It is tempting because it is the standard egress path for public subnets, and would be correct if the instances held Elastic IPs and the requirement permitted internet routing.

NAT gateway in each Availability Zone

Why it fails: A NAT gateway routes traffic to the public internet via an internet gateway, which the stem explicitly forbids; it also cannot reach S3 or Parameter Store without public endpoints. It is tempting because NAT gateways are the usual answer for private-subnet egress, and would be correct if outbound internet access were permitted.

Instance store volumes

Why it fails: Instance store volumes are physically attached to a single EC2 host and have a lifespan tied to that instance; data is lost when the instance is stopped or terminated. They are local to one instance only, so other EC2 instances in different Availability Zones cannot access the same data. This ephemeral, single-instance nature makes it impossible to use instance store as shared file storage for a multi-instance patient portal.

An EBS volume attached to all instances

Why it fails: A standard EBS volume is a block-level storage device that can be attached to only one EC2 instance at a time, and it is zonal—confined to a single Availability Zone. While multi-attach EBS is available for specific io1/io2 volumes, it only works within the same AZ and requires a clustered file system, making it impractical for broad multi-AZ sharing. Attaching one EBS volume to all instances is not supported and would not provide the shared file storage the portal needs.

S3 mounted as a POSIX file system without a file gateway

Why it fails: S3 is an object storage service, not a file system, and it does not natively provide POSIX semantics such as file locking, in-place directory operations, or strong consistency for all operations. Tools like s3fs can mount S3 as a POSIX-like filesystem, but they rely on workarounds that introduce performance and consistency issues, and they require a gateway or third-party software. Without Amazon File Gateway, directly mounting S3 as a POSIX file system is not a reliable solution for shared file storage for multiple Linux EC2 instances.

Store the database password in user data

Why it fails: Storing the database password in user data is insecure because user data is available to any process running on the instance via the instance metadata service at http://169.254.169.254/latest/user-data. It is also returned in the EC2 console and API responses unless explicitly redacted, making it visible to anyone with read access. A plaintext secret in user data has no rotation mechanism, and if the instance is compromised, the attacker can trivially retrieve the password and reuse it against the RDS database.

Use a security group rule that allows only application instances

Why it fails: Using a security group rule to allow only application instances restricts network access to the RDS endpoint, but it is purely a network-layer control and does not authenticate the identity of the caller. Any process on an allowed instance can still attempt SQL login, and if the instance is compromised, the attacker inherits the allowed network path; the database would still require a static password to be stored somewhere, reintroducing the original secret-management problem. Security groups complement authentication but cannot replace IAM database authentication or a robust password mechanism.

Embed the database password in the AMI

Why it fails: Embedding the database password in the AMI bakes a long-lived secret into a reusable image artifact that is copied, versioned, and launched in multiple environments. Anyone with access to the AMI—or any snapshot derived from it—can extract the password, and unauthorized AMI sharing can leak the secret outside your account. Rotating the password would require rebuilding and redeploying the AMI, which is slow, error-prone, and leaves old instances still holding stale credentials; the secret also remains recoverable from the AMI even after you stop using it.

Disable CloudWatch Logs entirely and rely on application local files for 90 days.

Why it fails: Local files are not a durable or centralized substitute for CloudWatch Logs. This would increase operational risk, make retention harder to enforce, and reduce the team’s ability to search and audit logs across the fleet.

Increase the number of CloudWatch alarms so that metrics are collected less expensively.

Why it fails: Adding more CloudWatch alarms cannot lower metric collection costs because alarms are billed per alarm and simply evaluate the same underlying metric data. The recurring costs in the exhibit come from detailed monitoring (1-minute metrics) enabled on all 200 EC2 instances and from CloudWatch Logs retention set to indefinitely retain logs; neither of these charges is affected by the number of alarms. In fact, each standard alarm has its own monthly price, so increasing alarm count would increase the bill, not reduce it.

Export all logs to Amazon S3 immediately and keep detailed monitoring enabled on every instance.

Why it fails: Exporting logs to S3 can help with archival economics, but it does not reduce the detailed monitoring charge, which is one of the largest costs in the exhibit. Keeping detailed monitoring on every instance preserves the major recurring cost driver.

Enable DynamoDB Streams so the table can process writes more quickly.

Why it fails: DynamoDB Streams capture item-level changes after a write has already succeeded, making them a tool for event-driven workflows, triggers, or replication—not a mechanism to accelerate or absorb incoming write throughput. A hot partition arises because all requests for a given tenantId hash to the same partition, and that partition's write capacity is exhausted before the database even persists an item; enabling streams does nothing to redistribute that traffic or raise the partition's throughput limit. In fact, because stream records are produced by the same write path, a throttled write is not recorded in the stream at all, so the feature is entirely orthogonal to the latency and throttling problem described.

Switch the table to on-demand capacity mode and keep the same key design.

Why it fails: Switching to on-demand capacity mode automatically scales table-level throughput in response to total traffic, but that scaling applies across the entire table, not to an individual partition. DynamoDB still enforces a per-partition throughput ceiling (commonly 1000 WCU per partition for items up to 1 KB), and if the partition key design funnels all of tenant ACME's writes into a single partition, that partition will hit its ceiling and throttle requests even though the table as a whole has plenty of available capacity. A skewed key design is thus the root cause, and on-demand mode merely changes how the aggregate capacity is calculated—it cannot split the hot key's data across physical storage.

Add a global secondary index on eventTime and query the index instead of the base table.

Why it fails: A global secondary index on eventTime would use eventTime as the index partition key, meaning it would group items by time rather than by tenant, so querying that index would return events from every tenant in a given time range and force additional filtering to isolate a specific issuer's data. This both breaks the required tenant-scoped access pattern and introduces a different hot partition: any surge of writes at the same timestamp (or into the same time bucket) would concentrate on one index partition and throttle. Moreover, a GSI still stores items under a partition key that determines physical placement; without a well-distributed partition key such as tenantId as the sort key or a tenant+shard compound key, the index does not solve the underlying partition distribution problem.

Disable CloudTrail logging

Why it fails: Disabling CloudTrail logging is unrelated to EBS storage cost optimization because CloudTrail records API activity and does not manage or reduce the storage footprint of EBS volumes or snapshots. Doing so actually degrades governance and audit compliance, making it harder to identify who created or deleted orphaned volumes, which is the opposite of the careful, verifiable cleanup this scenario requires.

Stop all EC2 instances in the account

Why it fails: Stopping all EC2 instances halts instance compute charges, but it does not remove the underlying EBS volume storage fees: every volume remains allocated and billed per GB-month regardless of whether the instance is running or stopped. Moreover, this action is overbroad because not all instances necessarily have unnecessary EBS usage, and it fails to address separate snapshot costs that accumulate independently of instance state.

Wait for AWS to automatically fail over the read replica to become the new primary

Why it fails: AWS does not automatically fail over cross-region read replicas. While Multi-AZ deployments do provide automatic same-region failover to a standby, a cross-region replica is intended only for manual promotion during DR, which requires a deliberate API call and connection-string updates. Waiting for AWS to auto-failover will leave the primary and replica in a stalled state and applications unavailable.

Restore the primary database from the most recent automated snapshot in us-west-2

Why it fails: Restoring the primary from the most recent automated snapshot is inferior because automated snapshots are typically taken once or a few times per day, whereas the read replica continuously applies changes from the primary, leaving a much smaller recovery point objective (RPO). Additionally, automated snapshots are stored in the source region (us-east-1) by default; unless cross-region snapshot copy is enabled, the snapshot isn't even available in us-west-2. Restoring a snapshot would require copying it first and would still yield data as of the snapshot time, losing hours of committed transactions.

Create a new RDS instance in us-west-2 and manually restore data from application logs

Why it fails: Manually rebuilding from application logs is not a valid RDS disaster recovery path because application logs capture business events, not the transactional database changes needed to reconstruct table state. Even if you had a full backup, replaying logs would be an error-prone manual process and would still lose transactions committed after the log cutoff. The existing read replica already contains a near-current, MySQL/Postgres-compatible copy of the production database, so using it is far faster and more reliable.

Internet gateway attached to the VPC

Why it fails: An internet gateway is a horizontally scaled, redundant component that enables communication between a VPC and the public internet. It is not used by private subnets because private subnets have no route to an internet gateway, and even if one were added, traffic would traverse the public internet and not be private. Internet gateways also do not provide private connectivity to AWS services, so this option cannot satisfy the requirement.

NAT gateway in each Availability Zone

Why it fails: A NAT gateway allows instances in private subnets to initiate outbound internet traffic, but it does so by routing packets through the internet gateway to the public internet. For private access to Systems Manager, this would mean traffic leaving the AWS network and traveling over public infrastructure, violating the requirement for private connectivity. Additionally, NAT gateways incur hourly charges and data processing fees, and they are not designed as a private endpoint service.

Create a global secondary index with the same date key

Why it fails: Creating a global secondary index on the same low-cardinality date key does not fix the root problem because the GSI maintains its own partitions and is subject to the same 1,000-WCU-per-partition limit. Every write to the base table also propagates to the GSI synchronously, so the GSI's date-only key would simply become another hot partition, causing the same throttling errors and potentially doubling the write cost.

Reduce the table's write capacity

Why it fails: Reducing the table's write capacity is diametrically opposite to solving a throttling problem: it lowers the total provisioned write budget available across partitions, making the hot partition even more likely to exhaust its allowance. Throttling occurs because one key receives disproportionate traffic, not because total capacity is wasted; the fix is to redistribute writes, not to shrink the capacity granted to the entire table.

Move the table to S3 Glacier Instant Retrieval

Why it fails: S3 Glacier Instant Retrieval is an object-storage class designed for long-lived, infrequently retrieved archives, not a transactional database for high-throughput item writes. Moving a DynamoDB table to it would abandon DynamoDB's partition-aware write engine, and the service cannot serve low-latency, item-level read/update/delete operations via the DynamoDB API; it is also not a way to scale write throughput for a hot key.

Allow the Account B root principal to assume the role if the caller knows the role ARN.

Why it fails: This would work technically, but it is far too broad and does not protect against confused deputy abuse. Any principal in Account B that can reach the role could potentially use it, which violates the customer’s security requirement.

Attach a permissions boundary to the role so that the vendor cannot exceed the approved permissions.

Why it fails: A permissions boundary caps the maximum permissions the role can have once assumed, but it does not restrict which principals can call sts:AssumeRole. In a confused deputy attack, an unintended principal could still assume the role and perform any actions allowed by the boundary, so the boundary does not protect against the threat. The trust policy with a specific principal and sts:ExternalId condition is the control that gates who can assume the role, not a permissions boundary.

Require MFA for the role assumption because it ensures only the vendor’s production automation can use the role.

Why it fails: Requiring MFA is nonsensical for non-human automation and does not mitigate the confused deputy risk. MFA only verifies that the caller possesses a second factor, but if the trust policy allows broad principals, any of them with valid MFA could potentially assume the role. Moreover, the vendor's production automation credentials are not tied to a human identity, so MFA cannot be practically enforced; the correct pattern is to use a unique sts:ExternalId condition in the trust policy to prevent reuse across customers.

Increase the ASG desired capacity so that extra instances absorb the failed ones.

Why it fails: Raising the desired capacity only launches more instances; it does not alter the Auto Scaling group's health check type, which remains EC2 status checks that pass because the OS and hypervisor are healthy. The application's HTTP 500 response occurs at the endpoint level, so the extra instances would either also return 500 and remain in service, or at best mask the symptom without removing the failing instances. This action neither detects the application failure nor triggers replacement.

Replace the Application Load Balancer with a Network Load Balancer to improve failover speed.

Why it fails: A Network Load Balancer operates at Layer 4 and its health checks only verify TCP connectivity, not HTTP response codes such as the 500 shown by the application's health endpoint. Replacing the ALB with an NLB would therefore still report the instance as healthy, and the ASG's default EC2 health checks would also continue to pass. This change alters the traffic layer but leaves the actual detection and replacement mechanism unchanged.

Increase the HealthCheckGracePeriod to the maximum value so the instances have more time to stabilize.

Why it fails: The health check grace period applies only after an instance is launched, delaying the start of health checks so a new instance has time to stabilize; it has no effect on instances that are already running and returning HTTP 500s. The exhibit shows a steady-state failure, not a boot-time issue, so increasing this value to the 3600-second maximum would only postpone evaluation of new replacements, leaving the current unhealthy instances untouched. It also does not change the health check type from EC2 status checks to ELB, so application failures remain invisible.

S3 mounted as a POSIX file system without a file gateway

Why it fails: Amazon S3 is object storage, not a POSIX file system, and a direct FUSE mount such as s3fs only emulates file-system behavior with weak consistency, no file locking, and suboptimal performance for concurrent warehouse workloads. Amazon File Gateway is the intended bridge to expose S3 as NFS/SMB, but without it, mounting S3 as POSIX is not a valid shared file-storage solution.

Instance store volumes

Why it fails: Instance store volumes are ephemeral block devices physically attached to a single EC2 host; their data is lost when the instance is stopped, terminated, or fails, and they cannot be shared across multiple instances. While they offer high I/O performance, they lack durability and shared access, making them entirely unsuitable as a persistent shared file system for a warehouse integration service.

An EBS volume attached to all instances

Why it fails: A standard EBS volume is a zonal block-storage device bound to a single EC2 instance; it cannot be attached to 'all instances' for concurrent shared file access. EBS Multi-Attach exists only for specific io1/io2 volumes on Nitro instances and supports a few instances without full POSIX file-locking semantics, so it does not satisfy shared file storage for a warehouse integration service.

Attach an IAM policy in Account B that grants s3:* on the bucket and its objects.

Why it fails: An identity policy in Account B alone cannot authorize access to a bucket in another account. S3 also requires a matching resource policy or equivalent cross-account trust, and broad s3:* is unnecessary for least privilege.

Create an S3 gateway endpoint in Account B so the application can reach the bucket privately.

Why it fails: An S3 gateway endpoint is a networking feature that provides private connectivity from Account B's VPC to S3, but it does not grant any data-plane authorization. The AccessDenied error occurs because the bucket policy and KMS key policy in Account A do not yet permit the external role, not because traffic traverses the public internet. Even with the endpoint configured, S3 will still evaluate the bucket policy and KMS key policy and deny the request if the role lacks s3:GetObject and kms:Decrypt permissions, so this option does not address the root cause.

Add an SCP in Account A that allows the Account B role to bypass KMS encryption checks.

Why it fails: Service control policies (SCPs) are guardrails applied at the AWS Organizations account or OU level that limit the maximum permissions for principals within that account; they can only restrict, never grant, permissions to an external role. An SCP in Account A cannot authorize an Account B role to bypass KMS encryption checks because KMS decisions are always governed by the key policy in Account A, and SCPs do not override that policy. To fix the encryption failure, the Account B role must be explicitly added to the KMS key policy with kms:Decrypt, alongside the bucket policy for s3:GetObject, so this option is both conceptually and technically incorrect.

Restore the latest automated snapshot and accept data loss from the last backup window.

Why it fails: Restoring the latest automated snapshot would bring the database back to the exact point in time when that snapshot was captured, not to the target time of 2026-04-27 15:10 UTC. Since automated snapshots are taken daily (and manual snapshots only when you take them), if the incident occurred after the last snapshot, all transactions committed since that snapshot—including the valid rows that should be preserved—would be lost. While a snapshot restore is a valid recovery option, it cannot provide fine-grained point-in-time recovery, so it would not meet the requirement to restore to the specific pre-delete moment and would leave you short of the desired recovery point objective.

Promote a read replica because it will contain the deleted rows and can replace the primary immediately.

Why it fails: A read replica operates asynchronously, meaning it continually applies the same changes that reach the primary database after a brief replication lag. The accidental DELETE is a committed transactional change, so it will replicate to the replica just like any other statement; the replica will contain deleted rows only if it is frozen at a point before the delete replicated, which is not guaranteed and cannot be precisely controlled. Promoting the replica gives you a live database at whatever its current, post-delete state is, not a point-in-time pre-incident copy, so it cannot satisfy the requirement to restore to exactly 2026-04-27 15:10 UTC.

Enable Multi-AZ on the current database and wait for automatic failover to reverse the delete.

Why it fails: Multi-AZ on Amazon RDS provides high availability by maintaining a synchronous standby replica in a different Availability Zone that is automatically promoted during failover caused by infrastructure failures or maintenance. It is a physical copy of the database, not a logical undo log: any committed DELETE on the primary is also committed on the standby, so a failover would fail over to a database that already has the rows deleted. Enabling Multi-AZ and waiting for failover cannot roll back a logical data modification; it only restores service availability after a loss of the primary instance, not after an application error.