Azure Databricks Architecture

Azure Databricks is a fully managed, cloud-based big data analytics platform based on Apache Spark. It provides an interactive workspace that enables data engineers, data scientists, and analysts to collaborate on building data pipelines, performing exploratory data analysis, and training machine learning models. The platform abstracts away the complexity of managing Spark clusters, allowing users to focus on writing code in Python, Scala, SQL, or R.

Azure Databricks exists to solve the challenge of processing massive datasets that exceed the capacity of a single machine. By distributing data and computation across a cluster of virtual machines (VMs), it can process terabytes or petabytes of data in parallel. The exam expects you to understand the underlying architecture because designing cost-effective and performant data solutions requires knowing how clusters are provisioned, how data is partitioned, and how costs are incurred.

At the heart of Azure Databricks is Apache Spark, which follows a master-slave architecture with a driver node (master) and multiple worker nodes (slaves). When you run a notebook command, the driver converts it into a logical plan, optimizes it into a physical plan, and then executes it as a series of stages containing tasks.

Driver Node: The driver is the central coordinator. It runs the user's main program, creates the SparkContext, schedules tasks, and manages the overall execution. The driver holds the metadata about the data (the catalog) and the execution plan. It communicates with the cluster manager (Azure Databricks' auto-scaling manager or a user-defined cluster) to acquire resources.

Worker Nodes: Worker nodes execute the tasks assigned by the driver. Each worker runs one or more executors, which are JVM processes that cache data and run computations. Executors store data in memory or disk and report back to the driver. The number of executors per worker depends on the VM size and configuration.

Cluster Manager: Azure Databricks uses its own cluster manager, which can be either a Standard (fixed-size) cluster or a High Concurrency (auto-scaling) cluster. The cluster manager allocates resources to the driver and workers based on the configuration.

- Cluster Types: - Standard Cluster: Fixed number of worker nodes. Suitable for single-user development. Not recommended for production because it cannot share resources. - High Concurrency Cluster: Auto-scaling (0 to max workers) and supports multiple users with isolation. Uses a separate driver per user. This is the recommended cluster type for production workloads. - Single Node Cluster: A driver-only cluster with no workers. Used for lightweight tasks or notebooks that do not require distributed processing.

- Default Timeouts: - Auto-termination: Clusters can be set to auto-terminate after a period of inactivity (default 120 minutes). This saves costs. The timer resets each time a command is run. - Spark Dynamic Allocation: By default, Spark dynamically adjusts the number of executors based on workload. The minimum and maximum executors can be configured (default min 0, max 10).

- Instance Types: - Memory Optimized: For data caching and machine learning. Examples: Standard_E8ds_v5 (8 vCPUs, 64 GB RAM). - Compute Optimized: For CPU-intensive tasks. Examples: Standard_F8s_v2 (8 vCPUs, 16 GB RAM). - General Purpose: Balanced. Examples: Standard_D8ds_v5.

You can configure clusters via the Azure Portal, Azure CLI, or the Databricks CLI. Here are key commands:

Azure CLI to create a workspace:

az databricks workspace create --resource-group myRG --name myWorkspace --location eastus --sku standard

Databricks CLI to create a cluster:

databricks clusters create --json '{
  "cluster_name": "my-cluster",
  "spark_version": "12.2.x-scala2.12",
  "node_type_id": "Standard_D8ds_v5",
  "num_workers": 5,
  "autotermination_minutes": 120
}'

Verification: - Check cluster status: databricks clusters list - View Spark UI: Navigate to the Databricks workspace, click on the cluster, then click "Spark UI". This shows stages, tasks, and executor statistics.

Azure Databricks integrates deeply with other Azure services:

Spark splits data into partitions (default block size 128 MB for Parquet). Partitions are distributed across executors. When a transformation requires data from multiple partitions (e.g., a join or groupBy), Spark performs a shuffle, which moves data across the network. Shuffles are expensive and should be minimized. You can control parallelism by setting spark.sql.shuffle.partitions (default 200).

Spark achieves fault tolerance through lineage. Each RDD (Resilient Distributed Dataset) tracks the transformations used to build it. If a partition is lost, Spark can recompute it from the original source using the lineage graph. This avoids data replication. However, recomputation can be slow; you can persist intermediate results using cache() or checkpoint().

Azure Databricks can be deployed in your own virtual network (VNet injection) for network isolation. This allows you to use Network Security Groups (NSGs) and service endpoints. You can also use Private Link for private connectivity. Authentication can be via Azure Active Directory (AAD) or personal access tokens. For data encryption, Databricks uses at-rest encryption (Azure Storage encryption) and in-transit encryption (TLS).

A large retail company ingests clickstream data from millions of users into Azure Event Hubs. They use Azure Databricks to run a streaming ETL pipeline. The pipeline reads from Event Hubs using Structured Streaming, performs real-time aggregations (e.g., counts of page views per product), and writes the results to a Delta Lake table on ADLS Gen2. The Delta table is then used by Power BI for real-time dashboards. The cluster is configured with auto-scaling (2 to 20 workers) to handle traffic spikes. A common issue is backpressure: if the stream rate exceeds the cluster's processing capacity, the Event Hubs consumer lag increases. To mitigate, they increase the number of partitions in Event Hubs and the Spark shuffle partitions. Misconfiguration of checkpointing can cause data loss on restart; they set the checkpoint location to a dedicated directory in ADLS.

A financial services company trains fraud detection models on historical transaction data stored in ADLS Gen2. Data scientists use Azure Databricks notebooks with AutoML to train models. They use a High Concurrency cluster with GPU-enabled VMs (e.g., Standard_NC6s_v3) for deep learning. The cluster uses Spot VMs for cost savings on non-critical training jobs. A common pitfall: Spot VMs can be evicted, causing job failure. They configure the cluster to use a mix of Spot and On-Demand VMs, and they enable Spark's speculative execution to retry failed tasks. They also use MLflow to track experiments and model versions. Performance considerations: they optimize data loading by using Delta Lake and partitioning by date.

A healthcare organization builds a data lakehouse using Azure Databricks and Unity Catalog. They have multiple teams (data engineers, data scientists, analysts) who need access to curated datasets. They deploy a Premium-tier workspace with Unity Catalog enabled. They define a three-level namespace: catalog.schema.table. They use RBAC to grant permissions at the catalog or schema level. They also implement data lineage tracking. A common misconfiguration: forgetting to set the metastore and catalog permissions correctly, leading to 'table not found' errors. They also set up data quality checks using Delta Live Tables (DLT) to ensure data reliability. The cluster is configured with a high concurrency mode to allow multiple users to query simultaneously without interference.