Databricks-Spark-Assoc Flashcards — Free Databricks Certified Associate Developer for Apache Spark Study Cards

Use window(timestamp, '10 minutes') with withWatermark('timestamp', '2 minutes').

Event-time windowing with watermarks is essential for handling late-arriving data in stream processing. By defining a window duration of 10 minutes and a watermark delay of 2 minutes, Spark maintains state for late events while discarding data older than the watermark threshold. This ensures the output remains accurate even when network latency or ingestion bottlenecks occur, preventing unbounded state growth in the processing engine.

df.repartition('user_id')

The repartition method creates a new set of partitions across the cluster network, which helps distribute data evenly to prevent skew. This is a critical transformation in Spark to optimize shuffle performance for downstream queries and aggregations by ensuring balanced workloads across executors.

Enable 'autoOptimize' and 'optimizeWrite' at the Delta table level.

Optimizing file sizes is crucial for read performance and metadata management in Delta Lake. Using the OPTIMIZE command or enabling Auto Optimize are the standard patterns to address the small file problem. These techniques consolidate fragmented data into larger, performant files, reducing the overhead on the query engine and preventing performance degradation during subsequent read operations. This is a fundamental skill for maintaining healthy, scalable data lakes on Databricks.

The Spark Driver

The Driver process is the central coordinator in Spark. It runs the main() method, creates the SparkContext, and performs RDD graph scheduling and task distribution. Understanding the Driver's role is crucial because it is the primary bottleneck for metadata operations and task scheduling in a Spark cluster, and failure here results in the loss of the application's state and active execution context.

Add a random prefix to the join key of the skewed table and replicate the join key of the other table.

Salting the join key by appending a random integer helps distribute the skewed keys across multiple partitions. This prevents a single executor from handling the bulk of the data, which is the primary cause of long-running tasks in skewed joins. Understanding how to repartition data based on a salted key is critical for developers tasked with optimizing performance in distributed systems where key distribution is inherently uneven across the cluster nodes.

Iterating through rows via Python lambdas forces high data serialization overhead between JVM and Python workers, destroying vectorized execution benefits.

Standard pandas `.apply()` functions often execute row-by-row python processing rather than leveraging native Catalyst query optimizations. When using non-vectorized operations in Spark without explicit type hints, the engine must serialize data between JVM and Python workers repeatedly. This causes high serialization overhead, defeats distributed columnar optimization, and ultimately results in severe performance degradation compared to vectorized Spark expressions.

The Spark Connect server running on the remote cluster driver node

Spark Connect introduces a decoupled client-server architecture where the client application sends dataframe plan representations via gRPC to a server component running on the driver node. This server translates the plan and executes it, which significantly reduces local memory overhead on the client machine and isolates client dependencies from the cluster environment.