High Availability practice questions

Instance store volumes

Why it fails: Instance store volumes are physically attached to a single EC2 host and have a lifespan tied to that instance; data is lost when the instance is stopped or terminated. They are local to one instance only, so other EC2 instances in different Availability Zones cannot access the same data. This ephemeral, single-instance nature makes it impossible to use instance store as shared file storage for a multi-instance patient portal.

An EBS volume attached to all instances

Why it fails: A standard EBS volume is a block-level storage device that can be attached to only one EC2 instance at a time, and it is zonal—confined to a single Availability Zone. While multi-attach EBS is available for specific io1/io2 volumes, it only works within the same AZ and requires a clustered file system, making it impractical for broad multi-AZ sharing. Attaching one EBS volume to all instances is not supported and would not provide the shared file storage the portal needs.

S3 mounted as a POSIX file system without a file gateway

Why it fails: S3 is an object storage service, not a file system, and it does not natively provide POSIX semantics such as file locking, in-place directory operations, or strong consistency for all operations. Tools like s3fs can mount S3 as a POSIX-like filesystem, but they rely on workarounds that introduce performance and consistency issues, and they require a gateway or third-party software. Without Amazon File Gateway, directly mounting S3 as a POSIX file system is not a reliable solution for shared file storage for multiple Linux EC2 instances.

Amazon EBS

Why it fails: EBS volumes are block storage that attach to a single EC2 instance in one Availability Zone; while multi-attach EBS exists, it is limited to a few instances in the same AZ and does not function as a general-purpose, POSIX-compliant shared file system. Because the scenario requires multiple instances across different AZs to access the same files concurrently, EBS cannot provide the necessary shared, multi-AZ access.

Amazon S3 only

Why it fails: Amazon S3 is object storage accessed via a REST API, not a mountable POSIX file system; while it can hold shared data, it does not support file locks, arbitrary byte-range edits, or the low-latency file semantics expected by applications. To mount S3 on EC2, you need additional software such as S3FS or a storage gateway, which introduces overhead and still does not provide the same multi-AZ shared file system behavior as EFS.

Instance store

Why it fails: Instance store volumes are ephemeral block storage physically attached to the host server, providing high I/O but with no durability — data is lost when the instance is stopped, terminated, or the underlying hardware fails. Since the volumes are local to a single instance, they cannot be mounted by other instances, making them useless for shared storage across Availability Zones.

S3 Cross-Region Replication

Why it fails: S3 Cross-Region Replication (CRR) is designed to asynchronously copy objects from one S3 bucket in one Region to a bucket in another Region. It has no integration with RDS failover mechanics: RDS MySQL does not store its database files as S3 objects in a way that CRR would replicate, and CRR cannot detect an AZ outage or automatically promote a standby database. Even if you manually exported DB snapshots to S3 and replicated them, restoring would be a manual, time-consuming operation, not automatic high availability.

EBS snapshots every hour

Why it fails: Taking EBS snapshots every hour is a backup strategy, not a high-availability or automatic failover mechanism. For RDS, automated backups and manual snapshots are stored in S3 and can be used to restore to a point in time, but the restore process is manual and takes significant time, often involving launching a new DB instance. Even with hourly snapshots, you could lose up to an hour of changes, and the database would be unavailable for the entire duration of the restore, which fails to meet a requirement of 'remain 2 nines' or similar high-availability SLA.

Read replicas only

Why it fails: Read replicas alone do not provide automatic failover for the primary RDS instance. A read replica uses asynchronous replication, so it can lag behind the primary (often by seconds or more), and while you can manually promote a read replica to become a standalone primary, there is no built-in automatic detection of primary failure or automatic promotion. Read replicas are beneficial for offloading read traffic or for multi-region disaster recovery, but for automatic AZ-level failover of a single RDS MySQL database, you need Multi-AZ, not read replicas.

EC2 Placement Groups with the "spread" strategy

Why it fails: Spread placement groups are designed to reduce the risk of correlated failures by placing instances on distinct underlying hardware (for example, different racks). That separation can increase network latency compared to cluster placement groups, which are optimized for lowest latency.

Auto Scaling cooldown adjustments only

Why it fails: Cooldown controls how quickly Auto Scaling scales in or out after a scaling event. It does not influence where instances are physically placed on the underlying hardware, so it cannot reduce network latency variability between instances.

Switching the instances to a larger instance size without any placement group

Why it fails: Resizing to a larger instance size can increase the available CPU and per-instance network bandwidth, but it does nothing to control the physical distance or network path between the communicating instances. In a default VPC placement, instances may land on different hardware racks or even in different AZs, so network latency and jitter remain variable regardless of instance size. The higher bandwidth might mask throughput constraints, but it does not reduce the round-trip delay caused by traversing aggregation or core network switches, which is the actual source of latency variance.

Use a spread placement group to distribute instances across multiple underlying hardware to improve overall availability.

Why it fails: Spread placement groups are optimized for fault isolation across distinct underlying hardware. They intentionally reduce the likelihood of all instances sharing the same failure domain, which typically increases the variability in network paths rather than minimizing latency/jitter between instances.

Use no placement group and rely on the Auto Scaling group to balance instance placement automatically.

Why it fails: Auto Scaling helps with capacity and scaling, but it does not guarantee proximity of instances for low-latency inter-node traffic. Without a placement group, placement is determined by normal scheduling, which may not keep instances close together.

Use a partition placement group so each instance is assigned to separate failure domains for low variance.

Why it fails: Partition placement groups are designed to distribute instances across partitions for capacity scaling with fault isolation (each partition is its own failure domain). They are not intended to keep instances in a tightly coupled network layout, so they do not best satisfy jitter-sensitive, low-latency requirements.

Increase the desired capacity in the existing Availability Zone to handle all traffic during an outage.

Why it fails: Raising the desired capacity within the current single Availability Zone merely adds more instances that share the same failure domain. When that whole AZ becomes unavailable—due to power loss, network disruption, or other zone-level events—every one of those instances, regardless of count, goes down together. There is no independent infrastructure remaining to accept traffic, so this approach cannot preserve availability during an AZ outage.

Enable longer ALB health check intervals so failing instances are detected more slowly.

Why it fails: Lengthening health check intervals makes the ALB less responsive at detecting an already-failing instance, but it does not create any redundant compute capacity elsewhere. In fact, slower health checks delay the Auto Scaling group's ability to replace unhealthy instances, extending the period of reduced availability. This change only alters detection timing; it does not address the fundamental dependency on a single AZ for both the load balancer and the instances.

Switch from the ALB to an Internet Gateway so instances can fail over to the public internet.

Why it fails: An Internet Gateway (IGW) is a horizontally scaled, redundant network component that simply provides bidirectional routing between a VPC and the public internet. It does not perform load balancing, does not evaluate target health, and has no concept of application-level failover between compute instances. Replacing the ALB with an IGW would actually remove the only mechanism that can route requests to healthy instances across multiple AZs, thereby guaranteeing that an AZ outage takes down the application.

Weighted routing

Why it fails: Weighted routing assigns a relative weight to each record and proportionally distributes DNS answers among all records, for example 80% primary and 20% standby, which intentionally sends traffic to both endpoints at all times. Unlike failover routing, it does not encode an explicit active-passive relationship; even with health checks, weighted routing simply removes unhealthy records from the denominator and continues splitting traffic among the remaining records based on original weights. This makes it suitable for load distribution or canary releases, but it does not model the 'primary always preferred, standby only on failure' pattern described in the question.

Geolocation routing

Why it fails: Geolocation routing evaluates the geographic origin of the client's DNS resolver and maps that location to a designated record, such as sending US users to us-east-1 and European users to eu-west-1. This policy is location-driven, not health-driven; if the primary endpoint fails, users from that region will still receive the DNS record pointing to the unhealthy endpoint until an administrator manually changes the geographically assigned record. It therefore cannot automatically redirect all traffic to the standby environment during an outage.

Latency-based routing

Why it fails: Latency-based routing answers DNS queries by returning the endpoint that appears to have the lowest latency from the calling resolver's network, so users are directed to whichever site is fastest rather than to a pre-defined primary site. Even if health checks eliminate unhealthy endpoints, the decision when both sites are healthy is purely performance-based, which can send traffic to the standby region during normal operation. This fails the active-passive requirement where the primary region must always be preferred unless it is actually unhealthy.

The database remains unavailable until an administrator manually creates a new instance.

Why it fails: Multi-AZ RDS does not require manual intervention after a primary instance failure. The service automatically detects the loss of the primary, promotes the standby instance in the same Region, and updates the DNS endpoint to point to the new primary. Because the failover is automated, an administrator does not need to create a new instance manually, which would extend recovery time significantly and contradict the purpose of Multi-AZ deployment.

Traffic is routed to a read replica in another Region for immediate continuity.

Why it fails: A read replica in another Region is primarily used for read scaling and disaster recovery, not for automatic failover during a Multi-AZ outage. Unlike a Multi-AZ standby, which is a synchronous copy in the same Region, a cross-region read replica uses asynchronous replication and would require manual promotion plus a change to the application connection string. RDS does not automatically route traffic to a read replica; doing so would serve stale data and not provide the immediate continuity promised by Multi-AZ.

The failed primary continues serving traffic while the standby synchronizes in the background.

Why it fails: A failed primary instance cannot continue serving traffic because the failure condition means it is no longer operational or reachable. When the primary fails, RDS promotes the standby instance to be the new primary, and the old primary is either replaced or removed. Synchronization between the primary and standby occurs continuously before failure; it does not happen in the background while the primary is down, so the standby cannot serve while syncing because failover has already occurred.

Spread placement group, because it places each instance on distinct underlying hardware.

Why it fails: A spread placement group is optimized for availability by reducing correlated failure risk across individual instances. It intentionally separates instances across distinct hardware, which is useful for fault tolerance but not for minimizing east-west latency or jitter for a tightly coupled cluster.

Partition placement group, because it isolates nodes across rack partitions.

Why it fails: Partition placement groups help large distributed systems reduce the impact of correlated infrastructure failures by separating instances into logical partitions. They are a resilience feature, not a performance feature, and they do not provide the close network proximity needed for the lowest-latency message exchange.

Move the workload behind an Application Load Balancer so node-to-node traffic is balanced more efficiently.

Why it fails: An Application Load Balancer operates at Layer 7 and is designed to distribute external client requests across a fleet of instances or containers, not to mediate server-to-server (east-west) traffic. Placing an ALB between cluster nodes would insert an extra proxy hop, adding latency and jitter rather than reducing it, and it would terminate and re-establish TCP connections, which is catastrophic for a tightly coupled analytics cluster that relies on persistent, high-throughput streams. Furthermore, ALB does not alter the physical placement of instances, so it cannot shorten the underlying network path that determines east-west latency.

Use a spread placement group to maximize instance separation across hardware.

Why it fails: A spread placement group places each instance on its own distinct rack, with independent power and network infrastructure, to maximize fault isolation for a small number of critical instances. This strategy physically spreads instances apart, increasing the distance and number of network segments between them, which directly contradicts the goal of reducing message latency. Also, spread placement groups are limited to 7 instances per Availability Zone, so they cannot support large tightly coupled fleets while maintaining low latency.

Use a partition placement group to distribute instances across many partitions.

Why it fails: A partition placement group is designed for large distributed workloads that need to tolerate rack-level failures, such as HDFS or Kafka, by dividing instances into logical partitions each on separate hardware. It does not optimize for the lowest latency between instances — in fact, instances in different partitions are intentionally isolated across racks or data centers, which adds network hops. This question is about tight, frequent low-latency traffic between a fixed set of nodes, which a partition placement group would actually hinder, not help.

Use multiple Auto Scaling groups to spread traffic across more subnets.

Why it fails: Multiple Auto Scaling groups manage instance quantity and replacement policies; they have no effect on the physical placement of instances relative to each other. Spreading traffic across more subnets typically means spanning across Availability Zones, which introduces cross-AZ network latency and data transfer costs, making the system slower for inter-instance messaging. The workload described is bottlenecked by placement and network topology, not by scaling or load balancing, so adjusting Auto Scaling groups cannot address the need for the lowest possible latency.

Replacing every NAT gateway with an internet gateway attached to private subnets

Why it fails: An internet gateway (IGW) is a VPC-level resource, not a per-subnet device, and it cannot be 'attached' to private subnets. When a private subnet routes 0.0.0.0/0 to an IGW, instances would need public IP addresses to actually reach the internet; without them the route is ineffective, and with them the subnet is effectively public. An IGW permits bidirectional traffic, so inbound connections from the internet can reach those resources, completely undermining the isolation that private subnets and NAT gateways are designed to provide. Replacing NAT gateways with IGWs does not preserve outbound-only access and is not a valid cost optimization.

Disabling route tables

Why it fails: Disabling route tables is not a valid operation in AWS, and more importantly, route tables are the essential mechanism that tells each subnet where to send traffic. Every private subnet that uses a NAT gateway must have a route table with a 0.0.0.0/0 entry pointing to that gateway; without that route, the batch job's outbound internet traffic would have no path and would be dropped, breaking the analytics pipeline. Disabling or deleting route tables would also disrupt internal VPC routing, not just internet access, so it would cause a full network outage rather than any cost savings. The correct optimization is to simplify or consolidate routing, not to remove the routing layer itself.

Moving all workloads to public subnets

Why it fails: Moving all workloads to public subnets would require assigning public IP addresses and routing traffic directly through an internet gateway, which exposes the batch analytics instances to unsolicited inbound internet traffic. This fundamentally violates defense-in-depth principles because private subnets exist to keep sensitive data and processing logic inaccessible from the public internet, and many compliance frameworks mandate that data-processing workloads remain private. While this change could eliminate NAT gateway costs, it is not a cost-first best practice because the security risk and potential compliance breach far outweigh the savings; a better cost play is to right-size the NAT gateway count, not expose the workload.

S3 Cross-Region Replication

Why it fails: S3 Cross-Region Replication (CRR) automatically replicates objects between S3 buckets in different AWS Regions. It is designed for bucket-level object replication, not for database instances. RDS MySQL does not use S3 CRR for high availability or failover; any use of S3 would only be for backups or manual export/import, which cannot provide automatic database failover.

Read replicas only

Why it fails: Read replicas are independent MySQL instances that asynchronously replicate from the primary to offload read traffic; they are not designed for automatic failover. While a read replica can be promoted to a primary role manually during a disaster, that action is not automatic and may lose recent transactions due to asynchronous lag. For automatic high availability, you need Multi-AZ, not just read replicas.

EBS snapshots every hour

Why it fails: Taking EBS snapshots every hour provides point-in-time recovery for the underlying storage, but restoring a snapshot creates a new database instance from that backup. This is a manual recovery process that involves downtime; it does not provide automatic failover or continuous availability. Snapshots protect data durability but are not a high-availability feature.

Place the instances in a spread placement group across multiple Availability Zones.

Why it fails: Spread placement groups are designed to reduce correlated failure risk by separating instances, not to minimize communication latency. Spreading instances across multiple AZs adds network distance and is the opposite of what a latency-sensitive clustered workload wants. The design improves resilience, but it sacrifices the fast intra-node communication the application needs.

Place the instances in a partition placement group within one Availability Zone.

Why it fails: Partition placement groups are useful when you want to isolate groups of instances from each other for resilience at large scale. They do not provide the same tight network proximity as a cluster placement group. For a low-latency analytics cluster, partitioning adds unnecessary separation and is not the best choice for fast node-to-node messaging.

Deploy the instances behind an Application Load Balancer in multiple Availability Zones.

Why it fails: An Application Load Balancer helps distribute client traffic for HTTP and HTTPS applications, but it does not optimize the private network path between EC2 instances. Adding multiple Availability Zones also increases the distance between nodes. That helps availability, but it does not address the workload's requirement for very low inter-instance latency.

Amazon EC2 instance store volumes.

Why it fails: Instance store volumes are ephemeral and provide no durability beyond the life of the instance. They are not replicated across AZs and are not suitable for compliance logs that must be retained. Data is lost if the instance stops or terminates.

Amazon Elastic Block Store (Amazon EBS) volumes attached to each EC2 instance.

Why it fails: EBS volumes are replicated within a single Availability Zone, not across multiple AZs. They are not designed for shared access by multiple instances and do not provide the required durability. Storing logs on individual EBS volumes would also make aggregation difficult.

Amazon RDS for MySQL with a Multi-AZ deployment.

Why it fails: Amazon RDS is a relational database service, not an object store for logs. While Multi-AZ provides high availability, it is not designed for storing large volumes of log data and does not offer the same durability or cost-effectiveness as Amazon S3. It also requires schema management.

Place the instances in a spread placement group across multiple Availability Zones.

Why it fails: Spread placement groups are designed to reduce correlated failure risk, not to minimize latency between tightly coupled nodes. Spreading instances across Availability Zones also adds network distance and does not improve east-west communication. This would make the cluster more resilient in some cases, but it works against the stated performance goal.

Add an Application Load Balancer in front of the workers to reduce inter-node latency.

Why it fails: An Application Load Balancer helps distribute client traffic to application targets, but it does not reduce latency between worker nodes that are already talking to each other directly. It would introduce another hop for incoming traffic and does nothing to change physical placement of the instances.

Increase the EC2 instance size while keeping the current multi-AZ layout.

Why it fails: Larger instances may improve CPU or memory headroom, but they do not solve the core issue shown in the exhibit: network latency between workers. The RTT problem is caused by placement and topology, not lack of compute capacity. Keeping the current AZ spread preserves the latency penalty.

Use a spread placement group so each instance is placed on separate hardware.

Why it fails: A spread placement group places each EC2 instance on its own distinct hardware, maximizing availability for small fault domains, but it intentionally scatters instances across different racks and often switches. That physical separation increases hop distance and network latency between application nodes, which is the opposite of what a cluster placement group would provide for this low-latency messaging workload. It also limits the number of instances you can place in a single Availability Zone, making it impractical for a large, tightly coupled fleet.

Move the instances into different Availability Zones to improve fault tolerance.

Why it fails: Moving instances into different Availability Zones does improve fault tolerance, but this workload is latency-sensitive and relies on frequent, small inter-instance messages. Cross-AZ traffic traverses more physical network distance and typically adds round-trip latency, which directly undermines the application's performance. Furthermore, inter-AZ traffic incurs data-transfer costs, so it is also the wrong economic choice for a tightly coupled workload that should stay within one place.

Use a partition placement group to balance traffic across partitions.

Why it fails: Partition placement groups help isolate groups of instances from one another for availability and scale. They are not the best option when the primary goal is the lowest possible latency between all nodes in the same application.

Increase the Auto Scaling group desired capacity from 2 to 3 in the same subnet.

Why it fails: Increasing the Auto Scaling group's desired capacity to three in the same subnet keeps every EC2 instance within a single Availability Zone, which is the same failure domain shown in the exhibit. If us-west-2a suffers an outage, all three instances fail simultaneously because they share the same physical dependency. Horizontal scaling in one AZ does not add redundancy; it just increases the blast radius of an AZ failure.

Replace the Application Load Balancer with a Network Load Balancer.

Why it fails: Replacing the Application Load Balancer with a Network Load Balancer does not change the deployment's Availability Zone topology because both load balancer types are zonal—they are provisioned only in the subnets you explicitly attach. If the NLB still uses the original single-AZ subnets, it remains in the same failure domain, and an outage of us-west-2a continues to take down both the load balancer nodes and the target instances. A different Layer 4/7 load balancer type cannot provide cross-AZ resilience on its own; you still must attach subnets in a second AZ.

Increase the health check grace period so instances stay registered longer.

Why it fails: The health check grace period only delays when Auto Scaling starts replacing an instance after it fails its health checks. During a full Availability Zone outage, the Application Load Balancer cannot reach any target in that subnet, so extending the grace period merely postpones the inevitable replacement and leaves the service unavailable longer. It does not introduce a second Availability Zone, nor does it give the ASG a healthy location to fail over to.

Instance store volumes

Why it fails: Instance store volumes are ephemeral block storage physically attached to a single EC2 host. Their data persists only as long as the instance is running; stopping, terminating, or a host failure permanently erases the data. Because instance store is exclusive to one instance and tied to a specific Availability Zone, it cannot serve as shared file storage for multiple Linux EC2 instances across AZs.

An EBS volume attached to all instances

Why it fails: A standard EBS volume is a zonal block storage device designed for single-instance attachment; it cannot be attached to all instances simultaneously. Even with EBS Multi-Attach (supported on certain io1/io2 volumes on Nitro instances), the volume stays within one Availability Zone and requires a cluster-aware file system to avoid corruption. Simply attaching one EBS volume to every instance would cause attachment failures and data inconsistency, making it an invalid choice for shared file storage.

S3 mounted as a POSIX file system without a file gateway

Why it fails: S3 is object storage with a flat key-value namespace, not a POSIX file system, and natively lacks features like file locking, directory semantics, and consistent concurrent modification. Tools that mount S3 via FUSE (e.g., s3fs) are not production-safe for shared POSIX workloads because they introduce caching inconsistencies, latency, and limited metadata support. Without a file gateway to translate POSIX operations to S3 API calls, S3 cannot provide the required shared file storage semantics for Linux EC2 instances.

A single EC2 instance with detailed monitoring

Why it fails: Detailed monitoring only reduces the CloudWatch metric interval to 1 minute; it does nothing to keep the application available if the instance fails or the underlying hardware degrades. A single EC2 instance is a single point of failure, so any OS crash, software fault, or Host maintenance event will take the dashboard offline. To achieve resilience, you need multiple instances managed by an Auto Scaling group that can automatically replace unhealthy instances.

All instances in one larger subnet

Why it fails: A subnet is always contained within a single Availability Zone, so making the subnet larger simply concentrates all instances in one physical location. If that AZ fails, every instance in the larger subnet becomes unreachable, taking the dashboard down completely. Increasing instance count in one subnet does not add fault tolerance; you must distribute instances across distinct AZs to avoid a single point of failure.

A Network Load Balancer in one subnet

Why it fails: An NLB deployed in only one subnet lives entirely in one Availability Zone, and its endpoints are tied to that AZ. Even though the NLB can route traffic to instances in a single subnet, a failure or network partition affecting that AZ will make the load balancer itself unreachable. For resilience, the NLB must have load balancer nodes in multiple AZs and be configured with target instances in those AZs to survive an AZ-level outage.

Instance store volumes

Why it fails: Instance store volumes provide temporary block-level storage physically attached to the host computer for the EC2 instance. They are ephemeral, meaning data is lost when the instance is stopped, terminated, or fails, and they cannot be shared across multiple instances because each instance has its own dedicated instance store. As a result, they cannot serve as durable, shared file storage for a patient portal spanning multiple Linux EC2 instances.

An EBS volume attached to all instances

Why it fails: A standard Amazon EBS volume is a block-level storage device that is created in a single Availability Zone and can be attached to only one EC2 instance at a time (unless you use the special Multi-Attach feature on certain volume types, which is limited to a few instances in the same AZ and still does not provide a true file-sharing model). Trying to attach an EBS volume to all instances is neither possible nor appropriate for shared file storage, and it would lack the distributed file system semantics needed by multiple Linux EC2 instances.

S3 mounted as a POSIX file system without a file gateway

Why it fails: Amazon S3 is an object storage service, not a POSIX-compliant file system; while you can use tools like s3fs to mount an S3 bucket as a file system, this approach lacks native file locking, consistent directory semantics, and adequate performance for concurrent writes, and it is generally not recommended for production workloads. Without AWS Storage Gateway's File Gateway (which presents a file-based NFS/SMB interface over S3), S3 does not provide the shared file system behavior required for a patient portal.