Delta Lake Format in Azure

Delta Lake is an open-source storage layer that sits on top of existing data lake storage (like Azure Data Lake Storage Gen2) and provides ACID (Atomicity, Consistency, Isolation, Durability) transactions, scalable metadata handling, and unified streaming and batch data processing. It was created to solve the problem of "dirty data" in data lakes, where concurrent writes could lead to partial writes, inconsistent reads, and data corruption. Traditional file formats like Parquet or ORC do not provide transaction guarantees; if a job fails mid-write, the data can be left in an inconsistent state. Delta Lake uses a transaction log to track every change, ensuring that reads always see a consistent snapshot.

Delta Lake stores data as a directory of Parquet files (by default) and a transaction log (stored in a _delta_log subdirectory). The transaction log is a series of JSON files (e.g., 00000000000000000001.json, 00000000000000000002.json) that record every atomic change: add file, remove file, update metadata, etc. Each JSON file contains a list of actions that were committed atomically. The log is the source of truth for the current state of the table.

When a write operation (INSERT, UPDATE, DELETE, MERGE) occurs, Delta Lake performs the following steps:

Key default values and parameters: - Checkpoint interval: Every 10 commits, Delta Lake writes a checkpoint (a Parquet file summarizing the state) to speed up future reads. This is configurable via spark.databricks.delta.checkpointInterval. - Retention duration for old log files: By default, Delta Lake retains the last 30 days of transaction log files and the last 7 days of old data files for time travel. Configurable via delta.logRetentionDuration and delta.deletedFileRetentionDuration. - Data file size: Delta Lake aims for files around 256 MB to 1 GB. The delta.targetFileSize config can adjust this.

ACID Transactions: - Atomicity: Each commit is all-or-nothing. If the commit fails, the table remains unchanged. - Consistency: The transaction log ensures that all reads see a consistent view. - Isolation: Concurrent writes are isolated; readers see either the state before or after a commit, never a partial write. - Durability: Once a commit is written, it is persisted in cloud storage.

Time Travel (Data Versioning): Delta Lake allows you to query previous versions of a table by specifying a version number or timestamp. This is possible because the transaction log retains the history. For example:

SELECT * FROM my_table VERSION AS OF 5;
-- or
SELECT * FROM my_table TIMESTAMP AS OF '2023-01-01';

Retention of old data is controlled by delta.deletedFileRetentionDuration (default 7 days) and delta.logRetentionDuration (default 30 days). After these periods, old files are vacuumed (permanently deleted) by the VACUUM command.

Schema Enforcement and Evolution: Delta Lake automatically validates that data being written matches the table's schema. If there is a mismatch, the write fails. However, you can enable schema evolution by setting mergeSchema to true in the write options. For example:

df.write.option("mergeSchema", "true").mode("append").saveAsTable("my_table")

Schema evolution can add new columns but cannot drop or rename columns without explicit ALTER TABLE commands.

Unified Batch and Streaming: Delta Lake supports both batch and streaming writes. You can use Structured Streaming to write data into a Delta table, and the table can be queried in real-time. Delta Lake provides exactly-once semantics for streaming writes using the transaction log.

To create a Delta table:

df.write.format("delta").save("/mnt/delta/my_table")

To read a Delta table:

df = spark.read.format("delta").load("/mnt/delta/my_table")

To view the history of a table:

DESCRIBE HISTORY my_table;

This returns the version, timestamp, operation, and other details.

To vacuum old files:

VACUUM my_table RETAIN 168 HOURS;

Default retention is 7 days (168 hours). Vacuuming removes files older than the retention period.

Delta Lake is natively supported in Azure Databricks and Azure Synapse Analytics (serverless SQL pools and Spark pools). On Azure, Delta tables are typically stored in Azure Data Lake Storage Gen2 (ADLS Gen2). The hierarchical namespace of ADLS Gen2 improves performance for Delta Lake operations like rename and list.

Delta Lake also integrates with Azure Machine Learning for feature stores, and with Power BI via DirectQuery or import (though DirectQuery may have limitations).

A financial services company processes millions of transactions per day. They use Azure Event Hubs to ingest streaming data into a Delta table stored on ADLS Gen2. Delta Lake provides exactly-once semantics, ensuring no duplicates. The fraud detection team runs batch queries and streaming queries on the same Delta table, benefiting from unified batch and streaming. They use OPTIMIZE and ZORDER BY (account_id) to speed up queries that filter by account. The retention of 30 days allows them to replay historical transactions for model training. Misconfiguration: If the retention is set too low, they lose ability to time travel for audits. If auto-optimize is not enabled, small files accumulate, degrading read performance.

A retail company uses Azure Synapse Analytics to build a data lakehouse. They store sales data in Delta tables in ADLS Gen2. Power BI users query these tables directly via Synapse serverless SQL pools. Delta Lake's schema enforcement prevents data quality issues from breaking reports. The company uses MERGE operations to upsert daily sales updates. The transaction log allows them to roll back if an incorrect update is applied. Scale: They manage over 10 TB of data with millions of files. They run OPTIMIZE weekly to keep file sizes optimal. Common pitfall: Setting delta.deletedFileRetentionDuration too low (e.g., 1 day) and then needing to restore data from a version older than 1 day; they lose the ability to time travel.

A healthcare provider stores patient records in Delta tables on Azure. Regulations require that all changes be auditable for 7 years. They set delta.logRetentionDuration to 2555 days (7 years) and delta.deletedFileRetentionDuration to 2555 days. They use DESCRIBE HISTORY to track who changed what. They also enable Delta Lake's change data feed to capture row-level changes for downstream systems. Misconfiguration: If they forget to adjust retention defaults, old logs are vacuumed, violating compliance. They also must ensure that the underlying storage (ADLS Gen2) has versioning enabled for extra redundancy.