Azure Data Lake Storage Gen2

Azure Data Lake Storage Gen2 is a set of capabilities dedicated to big data analytics, built on top of Azure Blob Storage. It is not a separate storage account type; rather, it is Blob Storage with the hierarchical namespace feature enabled. This feature introduces a file system abstraction that allows you to organize data into directories and subdirectories, just like a traditional file system. Under the hood, the data is still stored as blobs in a flat namespace, but the service maintains a directory tree that maps paths to blobs. This enables operations like renaming directories, listing contents of a directory, and setting permissions at the directory level, all without scanning all blobs.

Traditional Blob Storage is optimized for simple object storage where each blob has a unique name. It works well for content delivery, backups, and general-purpose storage. However, for big data analytics workloads (like those using Apache Spark, Hive, or Azure Databricks), applications expect a file system interface. They need to navigate directories, list files in a folder, and set permissions on directories. Without a hierarchical namespace, these operations become slow and expensive because the application must scan all blobs or maintain external metadata. ADLS Gen2 solves this by providing a native file system interface on top of blob storage, offering the best of both worlds: the scalability and cost of blob storage with the usability of a file system.

Internally, ADLS Gen2 uses a two-layer architecture: - Storage Layer: The actual data is stored as block blobs in a flat namespace. Each blob corresponds to a file, and its name is a unique identifier that includes the full path (e.g., /mydata/2024/01/log.txt). However, unlike standard Blob Storage, the blob name is not just a flat string; it is parsed by the service to simulate a directory structure. - Hierarchical Namespace Layer: This layer maintains a directory tree in a separate metadata store. When you create a directory, the service creates a zero-length blob that represents the directory. When you list files in a directory, the service queries the metadata store to find all blobs whose path starts with that directory prefix. This is much faster than scanning all blobs because the metadata store is indexed.

Key operations: - Create Directory: Creates a zero-length blob with a special marker (e.g., $directory$). The blob has no content but acts as a placeholder. - Rename Directory: Instead of renaming each blob individually, the service updates the metadata to point the old path to the new path. This is an atomic operation at the metadata level, making it very fast even for directories containing millions of files. - List Directory: The service queries the metadata store for all blobs whose path starts with the directory prefix and returns the list. This is O(n) where n is the number of blobs in that directory, not the total number of blobs. - Set Permissions: ACLs are stored as metadata on the directory or file blob. When a request is made, the service checks the ACLs along the path (from root to the target) to determine access.

To create a storage account with hierarchical namespace enabled using Azure CLI:

az storage account create \
    --name mystorageaccount \
    --resource-group myResourceGroup \
    --location eastus \
    --sku Standard_LRS \
    --kind StorageV2 \
    --hierarchical-namespace true

To verify hierarchical namespace status:

az storage account show --name mystorageaccount --query isHnsEnabled

To create a container (filesystem) and a directory:

az storage fs create --name mycontainer --account-name mystorageaccount
az storage fs directory create --name mydir --file-system mycontainer --account-name mystorageaccount

To set ACLs on a directory:

az storage fs access set --path mydir --file-system mycontainer --permissions rwxr-xr-x --account-name mystorageaccount

To list ACLs:

az storage fs access show --path mydir --file-system mycontainer --account-name mystorageaccount

To upload a file:

az storage fs file upload --source localfile.txt --path mydir/file.txt --file-system mycontainer --account-name mystorageaccount

- Authentication: Supports Azure AD (OAuth2) and shared key (account key). For production, Azure AD is recommended. - Authorization: RBAC for control plane (storage account management) and ACLs for data plane. The built-in roles for data plane are: - Storage Blob Data Owner: Full access including ACL management. - Storage Blob Data Contributor: Read/write/delete access. - Storage Blob Data Reader: Read-only access. - ACL Entries: Each ACL entry consists of a security principal (user, group, or service principal), a permission type (read, write, execute), and a scope (access ACL or default ACL). Maximum number of ACL entries per file or directory is 32 (including the four built-in entries: owner, owning group, mask, and other). - Superuser: The storage account owner (the Azure AD user who created the account) has superuser privileges and can bypass ACLs.

ADLS Gen2 supports lifecycle management policies to tier data to cool or archive tiers based on last modification time. You can define rules at the container or directory level using Azure CLI or portal. Note that once a blob is moved to archive tier, it must be rehydrated before reading, which can take hours.

Azure Data Lake Storage Gen2 is a powerful enhancement to Blob Storage that brings file system semantics to the cloud. For the AZ-104 exam, focus on its architecture, configuration, security model, and limitations. Understanding these will help you answer questions about storage for big data analytics, data lake architectures, and hybrid scenarios.

A large e-commerce company uses ADLS Gen2 as the central data lake for all customer transactions, clickstream logs, and inventory data. Data arrives in real-time from Azure Event Hubs and is stored in ADLS Gen2 in a directory structure like /raw/events/{date}/{hour}/. The company uses Azure Databricks to process this data, transforming it into curated datasets stored in /curated/customers/ and /curated/sales/. The hierarchical namespace allows Databricks to efficiently list only the new files in a directory, rather than scanning all blobs. This reduces job startup time from minutes to seconds. They also set ACLs so that only the data engineering team has write access to /raw/, while analysts have read access to /curated/. Performance scales to hundreds of terabytes, and they use lifecycle management to move data older than 30 days to cool tier, reducing costs. Misconfiguration example: If the hierarchical namespace was not enabled at creation, they would have to migrate data to a new account, causing downtime.

A SaaS provider hosts multiple customers in a single ADLS Gen2 account. Each customer's data is stored under a separate directory: /customer1/, /customer2/, etc. They use ACLs to isolate access: each customer's service principal has read/write access only to its own directory. RBAC is used for administrative tasks. The hierarchical namespace allows easy listing of all files for a customer without leaking data. They also use Azure Storage Firewall and Private Endpoints to ensure that only their VNet can access the storage account. Common pitfall: forgetting that ACLs are evaluated before RBAC; if a deny ACL is set inadvertently, even an administrator with Storage Blob Data Owner role cannot access the data. They mitigate this by using audit logs to detect unauthorized access attempts.

A financial services firm uses ADLS Gen2 for on-premises analytics with Azure Stack Hub. They replicate data from on-premises Hadoop clusters to ADLS Gen2 in Azure for long-term retention and disaster recovery. The hierarchical namespace ensures compatibility with on-premises Hadoop file systems (HDFS), allowing seamless migration. They use Azure Data Factory to orchestrate incremental data loads. Performance consideration: network latency between on-premises and Azure can impact throughput, so they use Azure ExpressRoute for dedicated connectivity. Misconfiguration: If they use standard Blob Storage instead of ADLS Gen2, their Hadoop jobs would fail because they expect HDFS-like directory operations. They learned this the hard way after a migration project delay.