Databricks Notebooks and Clusters

Azure Databricks is a unified analytics platform built on Apache Spark. It provides two primary abstractions: notebooks and clusters. Notebooks are web-based interfaces that contain executable code, visualizations, and narrative text. They support multiple languages: Python, Scala, SQL, and R. Clusters are sets of virtual machines that execute the code in notebooks. The compute is separated from the code: notebooks are persisted in the workspace, while clusters are ephemeral resources that can be started, stopped, and scaled independently.

A notebook is composed of cells. Each cell can be a code cell or a text cell (Markdown). When you run a cell, the code is sent to the attached cluster's driver node. The driver interprets the code, creates a Spark job, and distributes tasks to worker nodes. Results are returned to the notebook for display. Notebooks support interactive development: you can run cells in any order, modify previous cells, and re-run them. The notebook state (variables, data frames) is maintained in the cluster's memory across cells within the same session. However, if you detach and reattach the notebook to a different cluster, the state is lost.

Key notebook features tested on DP-900: - Languages: Python, Scala, SQL, R. Notebooks can mix languages using magic commands (%python, %scala, %sql, %r). - Visualizations: Built-in charting (bar, line, scatter, etc.) and integration with libraries like matplotlib and ggplot. - Collaboration: Multiple users can edit a notebook simultaneously (real-time co-authoring). - Version history: Databricks automatically saves notebook versions; you can view and restore previous versions. - Scheduling: Notebooks can be scheduled as jobs using Databricks Jobs. - Export/Import: Notebooks can be exported as HTML, DBC (Databricks archive), or source code (Python, Scala, etc.).

A cluster consists of a driver node and one or more worker nodes. The driver node is the master that coordinates work: it converts notebook code into Spark tasks, schedules them on workers, and aggregates results. Worker nodes execute the tasks and store intermediate data in memory or local disk. Clusters can be created interactively (interactive clusters) or automatically for jobs (job clusters).

Cluster types: - Standard cluster: General-purpose, suitable for interactive workloads. Can be used by multiple users. - High-Concurrency cluster: Managed by Databricks to support multiple users with automatic resource sharing. Uses a shared driver and separate executors per user session. - Single Node cluster: Only a driver node, no workers. Used for lightweight tasks or development with small data.

Cluster modes: - Interactive (or "all-purpose") cluster: Created manually and used for ad-hoc analysis. Runs continuously until terminated. Multiple users can attach notebooks. - Job cluster: Automatically created when a job runs and terminated after job completion. Single-tenant (only the job uses it). Cost-effective for scheduled or automated workloads.

Cluster lifecycle: 1. Creation: User specifies cluster configuration (Spark version, node type, number of workers, autoscaling, etc.). 2. Starting: Cluster transitions from Pending to Running state. Driver and workers are provisioned. 3. Running: Cluster is available for notebook attachment or job execution. 4. Auto-scaling (if enabled): Cluster adjusts number of workers between min and max based on workload. 5. Termination: Cluster is stopped. All memory and local storage are released. Configuration is preserved for later restart.

Key configuration parameters: - Databricks Runtime Version: Includes Apache Spark version and additional optimizations. E.g., "Runtime 12.2 LTS" includes Spark 3.3.2. - Node Type: Defines VM SKU (e.g., Standard_DS3_v2) with specific vCPUs, memory, and local SSD. - Min and Max Workers: For autoscaling. If min == max, cluster is fixed size. - Auto-termination: Idle timeout in minutes (default 120 minutes). Cluster auto-terminates if no activity. - Spark Config: Custom Spark properties (e.g., spark.sql.shuffle.partitions). - Environment Variables: Set for the cluster. - Init Scripts: Shell scripts run on each node during startup. - Access Mode: Single user, no isolation (shared), or per-user isolation (high-concurrency).

A notebook must be attached to a cluster to execute code. Attachment is a logical connection: the notebook sends code to the cluster's driver via the Databricks web app API. The driver compiles the code, creates Spark jobs, and returns results. Multiple notebooks can attach to the same cluster; they share the cluster's resources. However, they do not share session state (variables) unless explicitly persisted (e.g., using temp views or global temp views).

When you detach a notebook from a cluster, the cluster continues running but the notebook loses its state. Reattaching to the same cluster does not restore state.

Cluster policies are rules that constrain cluster creation. They allow administrators to control cost and security. For example, a policy can limit node types, enforce autoscaling, or restrict Spark configurations. Policies are applied at the workspace level and can be assigned to users or groups.

Databricks clusters can access Azure data sources: - Azure Blob Storage / ADLS Gen2: Mount points or direct access using service principals. - Azure SQL Database: JDBC connection from notebooks. - Azure Cosmos DB: Spark connector. - Azure Event Hubs: Streaming data ingestion. - Azure Synapse Analytics: Read/write using Synapse connector.

For DP-900, you should know that Databricks can read data from Azure Storage, Azure SQL Database, and Azure Synapse Analytics, and that clusters can be configured with managed identities for secure access.

To create a cluster using the Databricks CLI:

databricks clusters create --json '{
  "cluster_name": "my-cluster",
  "spark_version": "12.2.x-scala2.12",
  "node_type_id": "Standard_DS3_v2",
  "num_workers": 5,
  "autotermination_minutes": 60
}'

To list clusters:

databricks clusters list

To attach a notebook to a cluster via API (simplified):

notebook_path = "/Users/user@example.com/MyNotebook"
cluster_id = "1234-567890-abcd1234"
dbutils.notebook.run(notebook_path, 3600, {"param": "value"})

Databricks is used for data engineering (ETL), data science (ML), and data analysis (SQL). Notebooks provide the interactive interface for these tasks. Clusters provide the compute. On DP-900, you may be asked to identify which component is used for which purpose: notebooks for code development and visualization; clusters for execution. You may also need to choose the appropriate cluster type for a given scenario (e.g., job cluster for scheduled ETL, interactive cluster for ad-hoc analysis).

A large retail company ingests daily sales data from thousands of stores into Azure Data Lake Storage Gen2. They use Databricks to transform raw data into curated tables. They create a job cluster with 50 workers of type Standard_DS4_v2 (16 vCPUs, 32 GB RAM each) running Databricks Runtime 10.4 LTS. The job cluster is configured with autoscaling (min 10, max 50) and auto-termination of 30 minutes. The ETL notebook reads Parquet files from ADLS, performs aggregations, and writes to Delta tables. The job is scheduled to run nightly via Databricks Jobs. If the job fails due to a transient Azure issue, the job retries up to three times. The cluster is terminated after the job completes, saving costs. Common misconfiguration: setting autoscaling min too low can cause slow performance during data spikes; setting max too high can increase cost if not needed.

A financial services firm has a team of data scientists building machine learning models to detect fraud. They use an interactive cluster with high-concurrency mode to allow multiple data scientists to work simultaneously. The cluster uses 10 workers of type Standard_DS3_v2 (4 vCPUs, 14 GB RAM) and is configured to auto-terminate after 4 hours of inactivity. Each data scientist attaches their own notebook to the same cluster. They share data via global temp views. The cluster is left running during the workday and terminated manually at night. A common issue: one user's heavy workload (e.g., training a deep learning model) can consume all cluster resources, causing others' jobs to slow down. To mitigate, they use cluster pools or separate clusters for heavy workloads.

A healthcare organization uses Databricks SQL (SQL warehouses) to provide business analysts with a SQL interface. They create a SQL warehouse (which is a type of cluster optimized for SQL workloads) with a small size (2X-Small, 2 workers) for development and a medium size (2X-Large, 16 workers) for production. Analysts run SQL queries from the Databricks SQL editor or from Power BI via JDBC. The SQL warehouse auto-scales within its size range. Misconfiguration: setting the warehouse size too small can cause query timeouts; setting it too large wastes money. Also, if the warehouse is stopped, queries fail until it is restarted. They schedule the warehouse to start at 8 AM and stop at 6 PM weekdays using Auto Stop settings.