Databricks Delta Live Tables

Databricks Delta Live Tables (DLT) is a declarative framework for building and managing production-grade data pipelines on the Databricks Lakehouse. With DLT, you define what transformations to apply to your data, and DLT automatically handles task orchestration, cluster management, error handling, monitoring, and data quality enforcement. It is built on top of Delta Lake, Apache Spark, and the Databricks Runtime.

Traditional ETL pipelines require significant manual effort: you write scripts for each transformation, manage dependencies between tasks, configure clusters, handle failures, and monitor job health. As data volumes grow, this becomes brittle and hard to maintain. DLT abstracts away these complexities, allowing data engineers to focus on the logic of data transformation rather than infrastructure. It provides: - Declarative pipeline definition: You write Python or SQL code that declares what the output datasets should look like, not how to compute them step by step. - Automatic dependency resolution: DLT analyzes your code to determine the order of operations and builds a directed acyclic graph (DAG) of tasks. - Incremental processing: DLT automatically processes only new data since the last run, using Delta Lake's change data capture (CDC) capabilities. - Built-in data quality: Using 'expectations', you can define constraints on your data (e.g., NOT NULL, unique, custom SQL expressions). DLT can drop bad records, quarantine them, or fail the pipeline based on your policy. - Monitoring and alerting: DLT provides a rich UI and APIs for pipeline health, data quality metrics, and lineage.

DLT pipelines are defined in notebooks or Python files using the dlt module (for Python) or SQL syntax. When you create a pipeline, Databricks analyzes your code to build a DAG of transformations. Each transformation produces a dataset (a Delta table or view). The pipeline can be triggered on a schedule or continuously.

Pipeline Definition

A DLT pipeline consists of: - Source: Raw data from cloud storage (e.g., Azure Data Lake Storage Gen2, Blob Storage) or streaming sources (e.g., Event Hubs, Kafka). - Transformations: Python or SQL functions decorated with @dlt.view or @dlt.table that define how to derive new datasets. - Target: The output Delta tables that are stored in the Databricks File System (DBFS) or external locations. - Expectations: Data quality rules applied to datasets.

Execution Model

DLT uses a continuous processing mode or triggered mode. In triggered mode, the pipeline runs on a schedule (e.g., every hour) and processes all new data since the last run. In continuous mode, the pipeline runs indefinitely, processing data as it arrives with low latency (seconds to minutes).

Under the hood, DLT uses Structured Streaming for incremental processing. Each dataset is backed by a Delta table, and DLT tracks which data has already been processed using Delta Lake transaction logs. When a new batch of data arrives, DLT reads only the new files (or streaming records) and applies the transformations, updating the target tables atomically.

Data Quality with Expectations

Expectations are defined as constraints on a dataset. They can be: - `expect`: Records that fail the constraint are written to a quarantine table but do not stop the pipeline. - `expect_or_drop`: Records that fail are dropped from the output but written to a quarantine table. - `expect_or_fail`: If any record fails, the pipeline fails immediately.

Example in Python:

@dlt.expect("valid_id", "id IS NOT NULL")
@dlt.expect_or_drop("positive_revenue", "revenue > 0")
@dlt.table
valid_sales:
  return ...

Monitoring

DLT provides a built-in monitoring dashboard that shows:

To create a DLT pipeline via the Databricks CLI:

databricks pipelines create --json '{
  "name": "my_pipeline",
  "clusters": [{"num_workers": 4}],
  "libraries": [{"notebook": {"path": "/Users/user@example.com/my_pipeline_notebook"}}],
  "continuous": false
}'

To start a pipeline:

databricks pipelines start --pipeline-id <id>

To view pipeline status:

databricks pipelines get --pipeline-id <id>

In the Databricks UI, you can also view pipeline details under the 'Delta Live Tables' tab.

import dlt
from pyspark.sql.functions import col

# Define a streaming source
@dlt.table
def raw_orders():
  return (
    spark.readStream.format("cloudFiles")
      .option("cloudFiles.format", "json")
      .load("/path/to/raw/orders")
  )

# Transformation with data quality
@dlt.expect("valid_order_id", "order_id IS NOT NULL")
@dlt.expect_or_drop("positive_amount", "amount > 0")
@dlt.table
def cleaned_orders():
  return (
    dlt.read_stream("raw_orders")
      .select(
        col("order_id"),
        col("customer_id"),
        col("amount"),
        col("timestamp")
      )
  )

# Aggregation
@dlt.table
def daily_sales():
  return (
    dlt.read_stream("cleaned_orders")
      .groupBy("customer_id", "order_date")
      .agg({"amount": "sum"})
  )

This pipeline automatically handles schema inference, incremental processing, and data quality checks.

A financial services company processes millions of credit card transactions per day. They need to detect fraudulent transactions in near real-time and feed the results into a dashboard for analysts. They use DLT with continuous processing to ingest streaming data from Azure Event Hubs. The pipeline:

In production, this pipeline processes 50,000 events per second with sub-5-minute latency. The DLT monitoring dashboard shows throughput, data quality pass rates, and pipeline health. Common misconfigurations: not setting maxFilesPerTrigger or maxBytesPerTrigger can cause memory issues during streaming ingestion; forgetting to set cloudFiles.schemaLocation can lead to schema drift failures.

A retail company uses a medallion architecture (bronze, silver, gold) on Azure Data Lake Storage Gen2. They use DLT to automate the transformation of raw sales logs into curated analytical tables. The bronze layer ingests raw data from multiple sources (POS systems, web logs, CRM exports). DLT pipelines run in triggered mode every hour. In the silver layer, data is cleansed, deduplicated, and enriched with product and customer dimensions. Expectations enforce referential integrity: product_id IN (SELECT id FROM products). In the gold layer, aggregate tables are built for Power BI reports.

Key benefits: DLT automatically handles schema evolution (e.g., new columns added to source) via Delta Lake's schema enforcement and evolution. The team can add new transformations without rewriting the entire pipeline. They use @dlt.expect_or_fail on critical tables to prevent data quality issues from propagating. A common mistake: setting pipelines.autoOptimize.managed = false can lead to many small files and poor query performance. They use OPTIMIZE and ZORDER on gold tables periodically.