Elastic Fabric Adapter (EFA) for HPC

Elastic Fabric Adapter (EFA) is a custom-built network interface for Amazon EC2 instances that accelerates High Performance Computing (HPC) and machine learning (ML) applications. Unlike standard Elastic Network Adapters (ENA) which handle TCP/UDP traffic through the kernel, EFA provides OS-bypass capabilities, allowing applications to communicate directly with the hardware. This reduces latency from tens of microseconds to single-digit microseconds and increases message rate from thousands to millions of messages per second.

The primary driver for EFA is the need for tightly coupled parallel computing. In HPC workloads like weather simulation, molecular dynamics, or computational fluid dynamics, thousands of compute nodes must exchange small messages (often less than 100 bytes) at very high frequency. Traditional TCP/IP incurs significant overhead due to context switching, data copying, and protocol processing. EFA eliminates these bottlenecks by providing a reliable, low-latency transport that bypasses the kernel entirely.

EFA leverages two key technologies: - RDMA (Remote Direct Memory Access): Allows one computer to directly access the memory of another computer over a high-speed network without involving the CPU, cache, or OS of either system. This eliminates data copying and reduces CPU overhead. - OS-bypass: User-space applications can post send/receive operations directly to the EFA hardware, bypassing the kernel network stack. This removes context switches and interrupt handling.

At the protocol level, EFA implements a custom transport protocol called EFA Transport Protocol (EFAtp). It operates over a lossless fabric using priority flow control (PFC) to ensure no packet drops. The protocol supports: - Reliable Datagram: Messages are delivered exactly once, in order, without the overhead of TCP connection management. - Unreliable Datagram: For applications that can tolerate occasional loss (e.g., some ML training). - Reliable Connection: For stateful communication requiring guaranteed delivery.

From an application perspective, EFA exposes a libfabric interface, which is a user-space library that provides a standardized API for fabric communication. libfabric supports multiple providers; the EFA provider (efadv) is specifically optimized for AWS hardware.

To enable EFA: 1. Launch an EC2 instance from an HPC-optimized AMI (e.g., AWS ParallelCluster AMI) or a custom AMI with the EFA kernel module and libfabric installed. 2. Select an instance type that supports EFA (e.g., p4d.24xlarge). 3. In the launch configuration, under Network interfaces, attach an EFA device. You can attach up to 4 EFAs per instance on supported types. 4. Ensure the instance is in a cluster placement group. 5. Install the EFA software package (e.g., via aws-efa-installer).

Verification commands:

# Check if EFA device is present
lspci | grep -i efa

# Check EFA driver version
modinfo efa | grep version

# Run libfabric info test
fi_info -p efa -t FI_EP_DGRAM

# Check EFA health status
/opt/amazon/efa/bin/fi_info -p efa -t FI_EP_DGRAM | grep -E "(name|domain|version)"

EFA achieves its performance through: - Kernel bypass: Application writes directly to hardware queue pairs. - Zero-copy: Data is transferred from application memory to network without intermediate buffers. - Reliable transport with hardware retransmission: The EFA hardware handles retransmissions at the link layer, not the OS. - Adaptive routing: Traffic is dynamically load-balanced across multiple paths within the placement group.

However, EFA performance degrades if instances are spread across different placement groups or Availability Zones. Latency increases by 50-100 microseconds across AZs. Also, EFA does not support multicast; use MPI collective operations instead.

EFA itself is free; you only pay for the EC2 instances and data transfer. However, HPC-optimized instances are expensive (p4d.24xlarge at ~$32/hour). Use Spot Instances for cost savings if workloads are fault-tolerant.

An aerospace company runs ANSYS Fluent simulations on a cluster of 100 p4d.24xlarge instances. Each instance has 4 EFAs, providing 400 Gbps aggregate bandwidth per node. The CFD mesh has 500 million cells, requiring frequent halo exchanges between neighboring nodes. Using EFA, the simulation completes in 4 hours instead of 20 hours with ENA. The cluster is placed in a single cluster placement group in us-east-1a. The company uses AWS ParallelCluster to automate instance provisioning and MPI configuration. Key performance metric: MPI_Allreduce latency for 8-byte messages is ~2 microseconds between any two nodes.

A bank runs Monte Carlo simulations for portfolio risk using a custom MPI application. They use 50 hpc7a.48xlarge instances (AMD EPYC) with EFA. The workload is memory-bound, not compute-bound, so EFA's low latency is critical for synchronizing random number generators across nodes. Without EFA, the simulation would require checkpointing every 10 minutes to recover from stragglers; with EFA, synchronization overhead is negligible. The bank uses Spot Instances to reduce costs, but must handle interruptions via checkpointing. EFA's lack of encryption is acceptable because the VPC is private and data is encrypted at rest.

A tech company trains a large language model using 256 p5.48xlarge instances (NVIDIA H100 GPUs). They use NVIDIA NCCL for GPU-to-GPU communication. EFA supports GPUDirect RDMA, allowing GPUs to directly communicate across nodes without CPU involvement. The training throughput is 100 TFLOPS per GPU, and EFA provides 3200 Gbps bisection bandwidth per node. The cluster uses a non-blocking fat-tree topology within a single placement group. The company monitors EFA health using CloudWatch metrics (e.g., EFA_ReadBytes, EFA_WriteBytes). A common issue is misconfigured NCCL environment variables, leading to fallback to TCP (ENA), which reduces throughput by 10x.