FSx for Lustre for HPC

Amazon FSx for Lustre is a fully managed, high-performance file system that provides sub-millisecond latencies, millions of IOPS, and hundreds of gigabytes per second of throughput. It is built on the open-source Lustre file system, which is widely used in HPC environments. FSx for Lustre is designed for workloads that require fast access to data, such as genomics, financial modeling, autonomous vehicle simulations, and media processing. It integrates natively with Amazon S3, allowing you to process data stored in S3 with the performance of a parallel file system.

FSx for Lustre uses a distributed architecture based on the Lustre parallel file system. The key components are:

When a client wants to read a file, it first contacts the MDS to get the layout of the file (which OSTs hold which stripes). The client then reads the data directly from the OSTs in parallel. This design eliminates bottlenecks and scales linearly with the number of clients and OSTs.

- Deployment Types: - Scratch File System: Designed for temporary, bursty workloads. It is highly optimized for short-term processing and provides up to 200 MB/s per TiB of storage throughput. Scratch file systems are not replicated and are vulnerable to hardware failures. They are ideal for temporary HPC jobs. - Persistent File System: Designed for long-term, production workloads. It provides higher durability (data is replicated within the same AZ) and supports maintenance windows. Persistent file systems offer consistent performance with up to 200 MB/s per TiB for the 200 MB/s/TiB tier, or up to 1000 MB/s per TiB for the 1000 MB/s/TiB tier. - Throughput: The baseline throughput is 200 MB/s per TiB for both Scratch and Persistent (200 MB/s/TiB tier). Persistent also offers a 1000 MB/s/TiB tier. Throughput scales linearly with storage capacity. - IOPS: Up to 1,000,000 IOPS for larger file systems. - Latency: Sub-millisecond for metadata operations and low single-digit milliseconds for data I/O. - Storage Capacity: Ranges from 1.2 TiB to 10s of PiB. - Network: Uses Elastic Fabric Adapter (EFA) or TCP for client communication. EFA provides lower latency and higher throughput for tightly coupled HPC workloads. - S3 Integration: FSx for Lustre can be linked to an S3 bucket. Data is lazily imported from S3 on first access (i.e., when a file is read, it is fetched from S3 and stored on the file system). Exports can be configured to automatically write data back to S3 at a configurable interval (default 1 hour, but can be set to 0 for no automatic export). - Data Compression: Lustre supports lzfse compression to reduce storage usage, but this is not directly managed by FSx for Lustre.

To create an FSx for Lustre file system using the AWS CLI:

aws fsx create-file-system \
    --file-system-type LUSTRE \
    --storage-capacity 1200 \
    --subnet-ids subnet-12345678 \
    --lustre-configuration DeploymentType=PERSISTENT_1,PerUnitStorageThroughput=200

To mount the file system on an EC2 instance (Lustre client must be installed):

sudo mount -t lustre -o noatime,flock fs-0123456789abcdef0.fsx.us-east-1.amazonaws.com@tcp:/mountname /mnt/fsx

To verify the mount:

df -h /mnt/fsx

To check Lustre client statistics:

lctl get_param *.*.stats

FSx for Lustre performance scales linearly with storage capacity. For example, a 10 TiB Persistent file system with the 200 MB/s/TiB tier provides 2,000 MB/s of throughput. With the 1000 MB/s/TiB tier, it provides 10,000 MB/s. IOPS also scale: a 5 TiB file system can achieve up to 500,000 IOPS. The Lustre client can stripe files across multiple OSTs to maximize parallelism. The number of OSTs is automatically determined based on storage capacity (roughly one OST per 2.4 TiB).

A large investment bank runs Monte Carlo simulations for risk analysis. They have petabytes of historical market data stored in Amazon S3. The simulations require random access to thousands of files, with each simulation job needing high IOPS. The bank deploys an FSx for Lustre Persistent file system with 10 TiB storage and the 1000 MB/s/TiB tier, linked to their S3 data lake. They launch a cluster of EC2 instances (c5n.18xlarge) with EFA, each mounting the file system. The Lustre client caches metadata locally, reducing MDS load. The simulations complete in hours instead of days. Common issues: misconfigured security groups blocking Lustre ports, or not using EFA for network-intensive jobs, leading to lower throughput.

An autonomous vehicle company runs simulation pipelines that process sensor data (LiDAR, camera) and train ML models. Data is stored in S3 and processed in parallel across hundreds of GPU instances. They use FSx for Lustre Scratch file systems because simulations are ephemeral. Each simulation run creates a new Scratch file system, imports the relevant data from S3, runs the simulation, exports results back to S3, and then deletes the file system. This minimizes cost and leverages the high burst throughput of Scratch. Pitfall: forgetting to export data before deleting the file system, resulting in data loss. They mitigate this by setting a short export interval (e.g., 5 minutes) and using lifecycle hooks.

A genomics research lab uses AWS Batch to run analysis workflows (e.g., BWA, GATK) on whole-genome sequencing data. Each workflow processes 100-200 GB files. They use a Persistent FSx for Lustre file system with 200 MB/s/TiB tier, sized at 50 TiB. The file system is mounted by hundreds of spot instances. The lab configures automatic export to S3 every 30 minutes to persist intermediate results. A common misconfiguration is not setting the export policy, causing results to be lost if instances are terminated. Another issue is exceeding the file system's throughput limits by running too many concurrent jobs; they solve this by increasing storage capacity or using the higher throughput tier.