FSx for Lustre for HPC Workloads

Amazon FSx for Lustre is a fully managed, high-performance file system that provides POSIX-compliant, parallel file system capabilities. It is built on the open-source Lustre file system, which is widely used in HPC environments for its ability to handle massive scale and high throughput. FSx for Lustre is designed to process large datasets at high speed, making it ideal for workloads like genomic analysis, financial modeling, and autonomous vehicle simulation.

FSx for Lustre separates metadata and data onto different components: - Metadata Servers (MDS): Manage the file system namespace, including file and directory metadata (names, permissions, timestamps). They are highly available in persistent deployments. - Object Storage Targets (OSTs): Store the actual file data. Data is striped across multiple OSTs to increase aggregate throughput. Each OST is a storage server backed by NVMe SSD or HDD storage. - Lustre Clients: Mount the file system and access data via the Lustre network protocol. Clients communicate with MDS for metadata operations and with OSTs for data operations.

When a client reads a file, it first contacts the MDS to get the file's layout (which OSTs hold the data and at what offsets). Then it directly reads from the OSTs in parallel. This parallelism is key to achieving high throughput.

- Deployment Types: - Scratch: Optimized for short-term, bursty workloads. No data replication; if an OST fails, data is lost. Provides up to 200 GB/s throughput and 1 TB/s for larger file systems. Default storage: SSD. - Persistent: Designed for long-term, durable workloads. Data is replicated within a single Availability Zone (AZ) to protect against OST failure. Supports SSD and HDD storage options. Provides up to 8 GB/s per TB of storage. - Storage Capacity: From 1.2 TB to hundreds of PB. Scratch: 1.2 TB – 200 TB (SSD). Persistent: 1.2 TB – 100 TB (SSD), 1.2 TB – 10 TB (HDD). - Throughput: Scratch: 200 MB/s per TB of storage (baseline), burst up to 1 GB/s per TB. Persistent SSD: 200 MB/s per TB (base), 8 GB/s per TB (provisioned). Persistent HDD: 40 MB/s per TB (base), 250 MB/s per TB (provisioned). - Network: Uses Amazon VPC; clients must be in the same VPC or connect via VPC peering or AWS Direct Connect. Supports TCP and RDMA (for high-performance instances). - S3 Integration: Can import data from S3 at file system creation and export results back to S3. Data is lazy-loaded on first access. Supports import/export policies: NEW, NEW_CHANGED, and FROM_S3 (for import). - Backup: Automatic daily backups (with retention up to 35 days) and manual snapshots. Persistent file systems only. - Security: Encryption at rest (using AWS KMS) and in transit (TLS). Access controlled via VPC security groups and AWS IAM.

Creating a file system via AWS CLI:

aws fsx create-file-system \
    --file-system-type LUSTRE \
    --storage-capacity 1200 \
    --subnet-ids subnet-12345678 \
    --lustre-configuration DeploymentType=SCRATCH_2,DataCompressionType=LZ4,ImportPath=s3://my-bucket,ExportPath=s3://my-bucket/export

Mounting on an EC2 instance:

sudo mount -t lustre -o noatime,flock fs-0123456789abcdef.fsx.us-east-1.amazonaws.com@tcp:/mountname /mnt/lustre

Verifying mount:

df -h
lfs df -h /mnt/lustre

A pharmaceutical company runs genomic sequencing analysis using GATK and other bioinformatics tools. The input datasets are stored in Amazon S3 (tens of terabytes) and need to be processed by a cluster of EC2 instances. The company deploys a persistent FSx for Lustre file system (SSD, 10 TB) with an import path pointing to the S3 bucket. As jobs start, files are lazily imported from S3 into Lustre, allowing fast parallel reads. The file system is mounted on all compute instances. The company provisions 2 GB/s of throughput to match the aggregate read demand. Without Lustre, reading from S3 directly would be bottlenecked by S3's object-level throughput limits. The key configuration is to set the stripe count to 4 for large reference files to maximize parallelism. A common mistake is not increasing the stripe count, leading to single-OST contention. The company also sets up automatic export to S3 for results, using a lifecycle policy to delete temporary files after 7 days.

An autonomous vehicle company runs large-scale simulations using NVIDIA DRIVE Sim. Each simulation generates petabytes of sensor data that must be read and written at high speed. They use a scratch FSx for Lustre file system (100 TB, SSD) to achieve up to 100 GB/s throughput. The scratch deployment is chosen because simulations are transient and can be rerun if data is lost. The file system is mounted on GPU instances (p4d) using RDMA for low-latency access. They use LZ4 compression to reduce storage footprint. A common issue is running out of burst credits during long simulations; they mitigate by over-provisioning storage capacity (which increases baseline throughput) or by using persistent deployment with provisioned throughput. They also monitor FreeStorageCapacity and BurstCreditBalance in CloudWatch to avoid performance degradation.

A media company transcodes thousands of video files daily using AWS Batch. Source videos are in S3, and intermediate files require fast shared storage. They create a persistent HDD file system (5 TB) to keep costs low while still providing 200 MB/s base throughput. They import only the files needed for each job to avoid importing the entire S3 bucket. The file system is mounted on all Batch compute environments. A common misconfiguration is using the wrong security group rules, causing mount failures. They also learn that HDD file systems have higher latency for small files, so they use SSD for metadata-heavy workloads. They set up automatic backups to prevent data loss from accidental deletion.