Databricks-Spark-Assoc Troubleshooting and Tuning DataFrame Apps Practice Test 2 — 15 Questions

Question 1

A Spark DataFrame job on Databricks reads a large Parquet dataset and performs a groupBy aggregation. The job is slow, and the Spark UI shows that most tasks are reading the same partitions repeatedly. Which configuration change would most directly improve performance?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Cache the DataFrame in memory using .cache() before the aggregation.. Caching the DataFrame before the aggregation eliminates redundant reads of the same partitions by materializing the data after the first computation. This directly addresses the repeated partition reads shown in the Spark UI. While other options can improve performance in different scenarios, caching is the most targeted fix for this specific symptom.

Answer

Enable adaptive query execution by setting spark.sql.adaptive.enabled to true.

Answer

Increase the driver memory to 8 GB.

Answer

Set spark.sql.shuffle.partitions to 2000.

Question 2

A PySpark job on Databricks processes a 200 GB Delta table using df.groupBy('customer_id').agg(sum('amount')). The job runs for 3 hours, but Spark UI shows that 95% of tasks finish in 2 minutes while a few tasks run for over 2 hours. The shuffle read size is evenly distributed. Which Spark configuration change is most likely to resolve the long-running tasks?

Accepted Answer

Set spark.sql.adaptive.enabled=true and spark.sql.adaptive.skewJoin.enabled=true to allow AQE to handle skewed partitions automatically.. When a few tasks run far longer than the majority and shuffle read sizes are even, the root cause is typically skewed grouping keys. Adaptive Query Execution with skew join support can detect and split oversized partitions during execution. This dynamic optimization rebalances the work, reducing the tail latency caused by hot keys, and is the most direct fix for the observed behavior.

Answer

Increase spark.executor.memory to 32g to give each executor more memory for the large partitions.

Answer

Set spark.sql.autoBroadcastJoinThreshold=-1 to disable broadcasting and force a sort-merge join.

Answer

Increase spark.sql.shuffle.partitions from 200 to 2000 to create more partitions.

Question 3

A PySpark DataFrame job on Databricks repeatedly runs out of memory during a groupBy aggregation on a 500 GB Parquet dataset. The Spark UI shows that the task with the largest shuffle read processes 20 times more records than the median task, and the job fails with java.lang.OutOfMemoryError on the executor. You want to resolve the failure without changing the aggregation logic. Which action should you take?

Accepted Answer

Enable adaptive query execution and set spark.sql.adaptive.skewJoin.enabled to true so Spark splits the skewed partition before the aggregation.. The failure is a classic aggregation skew problem: one shuffle partition is far larger than the others, so a single task exhausts executor memory. Adaptive query execution with skew join handling detects and splits those oversized partitions at runtime, letting the same aggregation complete with balanced task sizes and without rewriting the transformation logic.

Answer

Call repartition(200) on the DataFrame immediately before the groupBy to evenly distribute the records.

Answer

Increase spark.sql.shuffle.partitions from 200 to 10000 so every task receives fewer records.

Answer

Set spark.executor.memory to 64g and spark.executor.cores to 1 so each executor has maximum heap for the aggregation.

Question 4

A nightly PySpark job joins a 900 GB transactions DataFrame with a 3 GB customers DataFrame. The join key is heavily skewed: one customer ID accounts for roughly 35 percent of all transactions, and a single task runs for hours while the rest of the stage finishes quickly. You cannot change the source data or the join key. Which approach most directly resolves the skew?

Accepted Answer

Enable adaptive query execution with skew join handling by setting `spark.sql.adaptive.skewJoin.enabled` to true.. Skew join handling in Adaptive Query Execution is designed for exactly this case: a post-shuffle stage where a few partitions vastly exceed the median. It splits oversized partitions and replicates the corresponding build-side rows, so the hot key's aggregation or join work is parallelized across several tasks while the data and key stay untouched.

Answer

Broadcast the 3 GB customers DataFrame with `broadcast(df_customers)` to eliminate the shuffle.

Answer

Call `df_transactions.coalesce(8)` before the join to reduce the number of input partitions.

Answer

Increase `spark.sql.shuffle.partitions` to 4000 so the hot partition is split across more tasks.

Question 5

A Spark Structured Streaming job on Databricks reads from a Kafka topic and writes to a Delta table using foreachBatch. The job processes 10,000 records per second, but after several hours, the batch duration steadily increases and eventually the job falls behind. The Spark UI shows that the number of active batches is growing and shuffle spill is high. Which action should you take to stabilize the streaming job?

Accepted Answer

Configure a trigger interval that allows the job to keep up, such as using Trigger.AvailableNow or increasing the processing time between micro-batches.. A streaming job that cannot keep up with its input rate will accumulate batches, leading to increasing latency and resource contention. Adjusting the trigger to process data in larger chunks or at a controlled pace allows the engine to catch up. Using Trigger.AvailableNow processes all available data in one batch, which can drain a backlog efficiently, while a longer processing interval gives each batch more time to complete.

Answer

Enable Kafka consumer group rebalancing by setting a unique group.id per executor.

Answer

Set spark.streaming.kafka.maxRatePerPartition to a very high value to consume faster.

Answer

Increase spark.sql.shuffle.partitions to match the number of Kafka partitions.

Question 6

You are tuning a Spark SQL query that performs a groupBy on a high-cardinality key. The job is spilling heavily to disk during the aggregation. Which configuration change is most likely to reduce spilling?

Accepted Answer

Increase spark.sql.shuffle.partitions.. Spilling during aggregation occurs when a task's memory is insufficient to hold the aggregation hash map. Increasing the number of shuffle partitions distributes the keys across more tasks, reducing the per-task memory footprint. This directly addresses the high-cardinality groupBy scenario, whereas the other options either increase memory pressure or are irrelevant to aggregation.

Answer

Decrease spark.sql.shuffle.partitions.

Answer

Set spark.sql.autoBroadcastJoinThreshold to -1.

Answer

Enable spark.sql.adaptive.enabled and set spark.sql.adaptive.coalescePartitions.enabled to true.

Question 7

A streaming PySpark job on Databricks uses `foreachBatch` to upsert micro-batches into a Delta table. After a few hours the Spark UI shows a steadily growing number of tasks per batch and increasing batch duration, even though the input rate is constant. The state store directory has grown to several hundred gigabytes. What is the most likely cause?

Accepted Answer

The job is not using a watermark, so the state store retains every key ever seen and never expires old entries.. Unbounded state is the classic cause of a streaming job that degrades steadily while input stays flat. A watermark with an event-time column lets Spark evict state older than the watermark, keeping the state store bounded so per-batch work and task counts stabilize instead of climbing.

Answer

The Delta target table lacks a Z-order index, so each upsert must scan the entire table.

Answer

`foreachBatch` buffers every micro-batch in memory until the query is stopped, causing state to accumulate.

Answer

Checkpointing is disabled, so the query replans the entire stream on every trigger.

Question 8

A Spark job on Databricks is experiencing out-of-memory errors on the executors during a groupBy aggregation on a large skewed dataset. The Spark UI shows that a few tasks are processing significantly more data than others. Which two actions should you take to mitigate the skew and improve performance? (Choose two.)

Accepted Answer

Enable adaptive query execution (AQE) with skew join optimization.. Enabling AQE with skew join optimization allows Spark to automatically split skewed partitions during shuffle, balancing the workload. Salting the skewed key manually achieves a similar effect by distributing the hot key's records across multiple partitions, enabling partial aggregation before a final combine. Both methods directly mitigate skew, whereas increasing partitions or executor memory does not address the concentration of data in one task.

Answer

Repartition the DataFrame by the grouping key before the aggregation.

Answer

Increase executor memory to 64 GB per executor.

Answer

Increase spark.sql.shuffle.partitions to a very high number, such as 10000.

Question 9

A PySpark DataFrame job on Databricks fails with `java.lang.OutOfMemoryError: GC overhead limit exceeded` on the executors during a wide transformation, and the Spark UI shows many tasks with very large shuffle spill and record counts. Which two changes are most likely to reduce executor memory pressure for this job? (Choose two.)

Accepted Answer

Increase `spark.executor.memory` and let `spark.memory.fraction` govern the unified memory pool.. Executor out-of-memory during a wide transformation is usually driven by too much data per task and too little heap. Raising the shuffle partition count shrinks each task's working set, and increasing executor memory enlarges the unified pool that execution uses, so together they lower the peak demand that triggers the garbage-collection limit error.

Answer

Set `spark.executor.cores` to a high value so each executor runs many concurrent tasks.

Answer

Enable `spark.sql.adaptive.enabled` so the optimizer can pick the join strategy at runtime.

Answer

Call `df.cache()` on the input DataFrame before the wide transformation.

Question 10

You are troubleshooting a Spark job that is running slowly due to data skew in a join. The skewed key is known to be a few values that dominate the data. Which TWO strategies can mitigate the skew? (Choose two.)

Accepted Answer

Salting the skewed key by appending a random suffix to the key in the large table and replicating the small table rows with all possible suffixes.. Salting the skewed key distributes hot keys across multiple tasks, while AQE's skew join handling automatically splits skewed partitions. These two strategies directly address the imbalance caused by a few dominant keys. The other options either do not target the skew or are impractical for large data.

Answer

Broadcasting the large table to all executors.

Answer

Increasing the number of shuffle partitions to a very high value, such as 10000.

Answer

Using a broadcast join for the small table if it fits within the broadcast threshold.

Question 11

A Spark job is experiencing data skew during a join operation on a key column. Which strategy is most effective for mitigating this issue without changing the business logic?

Accepted Answer

Add a random prefix to the join key of the skewed table and replicate the join key of the other table.. Salting the join key by appending a random integer helps distribute the skewed keys across multiple partitions. This prevents a single executor from handling the bulk of the data, which is the primary cause of long-running tasks in skewed joins. Understanding how to repartition data based on a salted key is critical for developers tasked with optimizing performance in distributed systems where key distribution is inherently uneven across the cluster nodes.

Answer

Increase the spark.sql.shuffle.partitions configuration value significantly.

Answer

Broadcast the larger table to all executor nodes.

Answer

Cache the skewed table in memory before performing the join.

Question 12

Which TWO of the following techniques effectively reduce the shuffle volume in a Databricks Spark job?

Accepted Answer

Apply filter transformations as early as possible in the DataFrame lineage.. Reducing shuffle volume is essential for performance, as shuffling moves data across the network, which is the most expensive operation in Spark. Techniques like predicate pushdown and column pruning minimize the amount of data read and processed before the shuffle phase occurs. Mastering these optimizations ensures that jobs remain scalable as datasets grow, preventing network congestion and I/O saturation during complex transformations or aggregate operations on large distributed DataFrames.

Answer

Increase the memory allocated to the Spark driver.

Answer

Enable dynamic allocation of executors.

Answer

Set the spark.sql.shuffle.partitions to 1.

Question 13

Refer to the exhibit. Which action is the most likely cause of the error shown in the Spark job logs?

Accepted Answer

The driver is attempting to aggregate a massive dataset into its memory.. The collect() action attempts to pull the entire DataFrame into the driver node's memory. When the dataset size exceeds the heap space allocated to the driver, a Java heap space error occurs. Developers must avoid collecting large datasets and instead use take() or head() for previews, or write results to cloud storage to maintain application stability when working with big data at scale.

Answer

The executor nodes have insufficient memory for the shuffle operation.

Answer

The cluster configuration uses too many small partitions.

Answer

The broadcast join threshold is set too low for the current job.

Question 14

A developer needs to optimize a Spark application that performs repetitive filtering and grouping on the same large DataFrame. Which feature should they implement to improve performance?

Accepted Answer

Use the cache() or persist() method on the DataFrame.. Caching (or persisting) the DataFrame keeps it in memory or on disk for subsequent actions. This is essential for iterative algorithms or workloads where the same data is reused, as it avoids recomputing the lineage from scratch. Understanding the trade-offs between memory and disk persistence is fundamental for building performant Databricks pipelines that maximize resource reuse and minimize redundant computation time.

Answer

Increase the number of cores per executor.

Answer

Switch the file format from Parquet to CSV.

Answer

Enable speculative execution.

Question 15

Which THREE factors should a developer consider when choosing a partition count for a shuffle operation?

Accepted Answer

The total size of the data being shuffled.. Selecting the right partition count is a balancing act between parallelism and overhead. Too few partitions lead to underutilization of the cluster, while too many partitions create excessive overhead for the scheduler. Developers must balance the data volume, the available cluster resources (cores), and the specific requirements of the operation to ensure optimal performance and avoid common performance pitfalls like task scheduling bottlenecks or executor memory issues.

Answer

The memory limit of the driver node.

Answer

The number of rows in the source file header.