Azure Data Lake Storage Gen2

Azure Data Lake Storage Gen2 (ADLS Gen2) is a cloud storage service built on top of Azure Blob Storage. It provides a hierarchical namespace that allows you to organize data into directories and subdirectories, much like a traditional file system. This enables you to use file system semantics (e.g., rename, move, delete directories) while leveraging the massive scalability and cost-effectiveness of Blob Storage. ADLS Gen2 is designed for big data analytics workloads, where data is often organized in a folder hierarchy and accessed by multiple compute engines simultaneously.

Before ADLS Gen2, Azure had two separate storage offerings for analytics: Azure Data Lake Storage Gen1 (a dedicated storage service) and Azure Blob Storage (a general-purpose object store). Gen1 offered a hierarchical namespace and high performance for analytics but was limited in scalability and integration with other Azure services. Blob Storage was highly scalable and cost-effective but lacked a hierarchical namespace, making it difficult to organize data for analytics. ADLS Gen2 was created to merge the best of both: the hierarchical namespace and performance optimizations from Gen1 with the scalability, durability, and cost tiers of Blob Storage.

ADLS Gen2 is essentially Blob Storage with a hierarchical namespace enabled. The hierarchical namespace is implemented as a metadata layer that maps directory paths to blob prefixes. When you create a directory, ADLS Gen2 creates a zero-length blob (a "directory marker") that represents that directory. However, the actual data is stored as blobs (objects) in a flat namespace underneath. The hierarchical namespace provides a logical view of directories and subdirectories, but the physical storage remains flat. This allows ADLS Gen2 to support both object storage semantics (via Blob APIs) and file system semantics (via the ABFS driver).

#### ABFS Driver

The Azure Blob File System (ABFS) driver is a Hadoop-compatible file system driver that enables compute engines like Azure Databricks and HDInsight to access ADLS Gen2 using the abfs:// protocol. The driver translates file system operations (e.g., ls, mkdir, rename) into Blob Storage REST API calls. For example, a ls on a directory triggers a list blobs operation with a prefix filter. The driver also optimizes performance by using multiple concurrent connections and caching.

To create an ADLS Gen2 storage account using Azure CLI:

az storage account create \
    --name mystorageaccount \
    --resource-group myResourceGroup \
    --location eastus \
    --sku Standard_LRS \
    --kind StorageV2 \
    --enable-hierarchical-namespace true

To verify hierarchical namespace is enabled:

az storage account show \
    --name mystorageaccount \
    --resource-group myResourceGroup \
    --query isHnsEnabled

To create a directory:

az storage fs directory create \
    --name mydir \
    --file-system mycontainer \
    --account-name mystorageaccount

To list files and directories:

az storage fs file list \
    --file-system mycontainer \
    --path mydir \
    --account-name mystorageaccount

ADLS Gen2 integrates seamlessly with: - Azure Synapse Analytics: Can query data directly in ADLS Gen2 using serverless SQL pools or dedicated SQL pools. - Azure Databricks: Uses the ABFS driver to read/write data with Spark. - Azure HDInsight: Hadoop clusters can access ADLS Gen2 as the default file system. - Azure Data Factory: Can copy data to/from ADLS Gen2. - Power BI: Can connect to data stored in ADLS Gen2 using DirectQuery or import.

ADLS Gen2 provides high throughput for analytics workloads. It supports up to 20,000 IOPS per partition (blob) for standard accounts and 100,000 IOPS for premium accounts. The hierarchical namespace reduces the number of API calls needed for directory operations (e.g., listing a directory with many subdirectories is faster than listing all blobs with a prefix). However, enabling the hierarchical namespace can increase the cost of certain operations (e.g., renaming a directory requires updating metadata for all blobs in that directory).

ADLS Gen2 supports two layers of security: 1. Azure RBAC: Assign roles like Storage Blob Data Contributor, Storage Blob Data Reader, or Storage Blob Data Owner at the storage account, container, or directory level. 2. POSIX ACLs: Fine-grained permissions on directories and files. ACLs can be default (inherited by new children) or access (applied to the current object).

Both must be evaluated: RBAC determines whether a user can access the storage account at all, while ACLs determine what they can do within the hierarchy. The effective permissions are the intersection of RBAC and ACLs.

A large e-commerce company ingests terabytes of server logs daily. They use Azure Data Lake Storage Gen2 as the landing zone for raw logs. The logs are organized by date and hour in a directory structure like /logs/raw/2023/10/15/14/. A Databricks job reads the logs from the last hour, processes them, and writes the aggregated results to a different directory /logs/aggregated/. The hierarchical namespace makes it easy to manage the directory structure and apply lifecycle policies to move raw logs older than 30 days to the Cool tier and older than 90 days to the Archive tier. In production, they found that enabling hierarchical namespace reduced the time to list files for a given hour by 50% compared to a flat namespace with prefix listing. However, they encountered a performance issue when renaming a directory containing millions of small log files: the rename operation took several minutes because each file's metadata had to be updated. They mitigated this by using a separate container for each day and renaming containers instead of directories.

A healthcare provider uses ADLS Gen2 to store patient data, including medical images (DICOM files) and clinical notes. They need fine-grained access control to comply with HIPAA. They use RBAC to grant broad access to data scientists and ACLs to restrict access to specific patients' directories. For example, a data scientist in the cardiology department has read access to the /patients/cardiology/ directory but not to /patients/neurology/. They also use Azure Private Endpoints to ensure data never traverses the public internet. A common misconfiguration they discovered: when setting ACLs, they forgot to set the execute permission on parent directories, causing access denied errors for users trying to list files. The fix was to ensure all parent directories had at least execute permission for the group.

A manufacturing company uses Azure IoT Hub to stream sensor data from factory equipment. The data is written directly to ADLS Gen2 using the Event Hubs Capture feature, which writes Avro files to a container. The data is organized by device ID and timestamp. The hierarchical namespace allows them to query data for a specific device and time range efficiently using Azure Synapse serverless SQL. They configured a lifecycle management policy to move data older than 7 days to the Cool tier and delete data older than 1 year. One issue they faced: the archive tier has a 15-minute retrieval latency, and they accidentally archived data that was still being queried by a real-time dashboard. They resolved this by setting a rule to not archive data that was accessed in the last 30 days.