Databricks-Spark-Assoc Troubleshooting and Tuning DataFrame Apps Practice Test 3 — 15 Questions

Question 1

Which configuration parameter should be adjusted to change the default number of partitions when reading from a shuffle-heavy operation?

Accepted Answer

spark.sql.shuffle.partitions. The `spark.sql.shuffle.partitions` configuration is the primary setting used to control the parallelism of shuffle-based operations in Spark SQL. Adjusting this value allows developers to match the level of parallelism to the specific data volume and cluster resources. Understanding how this setting impacts performance is essential for fine-tuning Spark applications to prevent task contention and ensure that the cluster is fully utilized during data-intensive stages of a job.

Answer

spark.executor.memory

Answer

spark.default.parallelism

Answer

spark.driver.memory

Question 2

Refer to the exhibit. What is the most likely cause of the repeated ExecutorLostFailure messages in the logs?

Accepted Answer

The executor process reached its memory limit and was killed.. ExecutorLostFailure indicates that an executor has died, often due to exceeding memory limits or heartbeats failing. When processing large data, the executor's heap memory can be exhausted, leading to a crash. This signifies that the cluster resources are insufficient for the current workload, requiring either more memory per executor or more efficient data processing strategies to prevent the executor from being terminated by the operating system or the cluster manager.

Answer

The driver node is experiencing a network latency issue.

Answer

The Spark job is attempting to use too many partitions.

Answer

The shuffle partition count is set too low for the data.

Question 3

Which file format is best suited for performance-critical Spark applications that require efficient schema enforcement and column pruning?

Accepted Answer

Parquet. Parquet is a columnar storage format that natively supports predicate pushdown and column pruning, allowing Spark to read only the necessary data from disk. This drastically reduces I/O throughput and improves query performance significantly. Understanding why columnar formats are superior to row-based formats for analytics is a foundational concept for Databricks developers designing high-performance data lakes and efficient ETL pipelines at scale.

Answer

CSV

Answer

JSON

Answer

XML

Question 4

When a Spark job is stuck in a shuffle phase, what is the most effective first step to identify the root cause of the performance bottleneck?

Accepted Answer

Check the Spark UI 'Stages' tab for uneven task durations.. The Spark UI provides a detailed breakdown of stages, tasks, and memory usage. Examining the 'Stages' and 'SQL' tabs reveals which tasks are taking the longest, which is indicative of skew or resource contention. Learning to interpret the Spark UI is the single most important skill for a Databricks developer, as it turns opaque 'stuck' jobs into actionable data regarding partition distribution and executor behavior.

Answer

Restart the cluster to clear any cached data.

Answer

Increase the spark.executor.memory value.

Answer

Enable speculative execution immediately.

Question 5

A job is reading a huge amount of data from a table, but only uses three columns. Which optimization technique will provide the most significant I/O performance benefit?

Accepted Answer

Column pruning.. Column pruning is the practice of reading only the required columns from a data source. In formats like Parquet, this reduces the total amount of data read from disk and transferred across the network. This is a crucial optimization for Databricks developers to reduce I/O bottlenecks and improve overall pipeline speed, especially when dealing with wide tables containing hundreds of unused columns in analytical query workloads.

Answer

Caching the DataFrame.

Answer

Increasing the number of partitions.

Answer

Broadcasting the table.

Question 6

Refer to the exhibit. What is the best way to resolve this error?

Accepted Answer

Write the results to a data lake instead of collecting them.. This error occurs when the result set returned to the driver from the executors exceeds the limit defined by `spark.driver.maxResultSize`. The best practice is to stop trying to bring huge datasets back to the driver, and instead write the results to a distributed storage system like S3 or ADLS. This prevents the driver from becoming a memory bottleneck and ensures the application remains scalable for large-scale data processing tasks.

Answer

Increase the spark.driver.maxResultSize setting.

Answer

Disable the Spark broadcast join threshold.

Answer

Increase the number of executor instances.

Question 7

When analyzing a Spark job's execution plan, what does a 'BroadcastHashJoin' indicate compared to a 'SortMergeJoin'?

Accepted Answer

BroadcastHashJoin avoids a shuffle operation.. A 'BroadcastHashJoin' indicates that Spark has decided to send the smaller table to all executors to perform the join in memory, avoiding a shuffle. A 'SortMergeJoin' involves shuffling both tables, which is significantly more expensive. Understanding when Spark chooses one over the other helps developers identify if they need to explicitly broadcast small tables or adjust configuration thresholds to optimize join performance and minimize costly cluster-wide shuffle operations.

Answer

BroadcastHashJoin requires both tables to be pre-sorted.

Answer

SortMergeJoin is always faster than BroadcastHashJoin.

Answer

BroadcastHashJoin only works for inner joins.

Question 8

Your Spark application is experiencing severe data skew while performing a join between a large fact table and a small dimension table. Which technique should you apply to optimize performance?

Accepted Answer

Force a broadcast join for the small dimension table.. Broadcast hash joins are the most effective solution for skew caused by a large table join when one side is small enough to fit in memory. By broadcasting the small table to all executor nodes, Spark avoids the expensive shuffle operation that causes data skew. This prevents specific partitions from becoming hotspots, which is critical for maintaining stable and performant ETL pipelines in production Databricks environments.

Answer

Increase the number of partitions using repartition() on the large table.

Answer

Enable AQE and increase the spark.sql.shuffle.partitions configuration.

Answer

Apply a salting technique to the join keys of the small table.

Question 9

A developer notices a Spark job is failing with an OutOfMemoryError during a join operation on two large tables. The join key is highly skewed, causing one task to process significantly more data than others. Which technique should be applied to resolve this skew?

Accepted Answer

Implement salting on the join key to distribute the skewed keys.. Salting involves adding a random prefix or suffix to the join key to redistribute the data across multiple partitions. This prevents a single executor from handling a disproportionate amount of data. This approach is a standard industry pattern for resolving skew-related OOM errors, as it breaks the hot key into smaller, manageable chunks that can be processed in parallel across the cluster without bottlenecking at a single node.

Answer

Increase the spark.driver.memory configuration.

Answer

Enable AQE and increase the spark.sql.shuffle.partitions value.

Answer

Convert the larger table into a broadcast variable.

Question 10

Which action should be taken to optimize a Spark application that performs multiple operations on the same DataFrame and shows evidence of redundant re-computations in the Spark UI DAG visualization?

Accepted Answer

Use the cache() or persist() method on the DataFrame.. Persisting (or caching) a DataFrame instructs Spark to store the computed results of the DataFrame in memory or on disk. This is vital when the same DataFrame is accessed multiple times across different stages, as it prevents Spark from re-executing the entire lineage graph from the source, significantly reducing processing time and resource consumption in complex, multi-step analytical pipelines.

Answer

Increase the number of shuffle partitions.

Answer

Implement a custom partitioner for all joins.

Answer

Convert the DataFrame to a RDD.

Question 11

A Spark job is running slower than expected due to excessive shuffling. Which TWO of the following techniques would directly reduce the volume of data transferred over the network?

Accepted Answer

Broadcast join for small tables.. Reducing network traffic is critical in Spark performance tuning. By utilizing features like broadcast joins, you avoid the shuffle phase entirely when one table is small. Additionally, using filter and select operations early in the transformation pipeline ensures that only the necessary rows and columns are transmitted, significantly decreasing the total data volume shuffled during wide transformations.

Answer

Increase spark.sql.shuffle.partitions.

Answer

Enable dynamic resource allocation.

Answer

Increase spark.driver.memory.

Question 12

Refer to the exhibit. You are reviewing the logs for a Spark application and notice the warning regarding broadcasting a large task binary. What is the most likely cause and mitigation?

Accepted Answer

A large variable is captured in a closure; use a Broadcast Variable.. The warning indicates that a large object, likely a high-dimensional collection or a large variable, is being captured in a closure and broadcast to all executors. This increases network pressure and memory usage. The mitigation is to use a broadcast variable or remove the reference to the large object from the closure to prevent serialization overhead and potential performance degradation.

Answer

The partition size is too large; increase spark.sql.files.maxPartitionBytes.

Answer

The executors have insufficient memory; increase spark.executor.memory.

Answer

The cluster is out of network bandwidth; enable compression.

Question 13

You are debugging a PySpark DataFrame job on Databricks that performs multiple transformations and actions on a large delta table. You notice that the execution plan shows redundant computations where the same upstream DataFrame is evaluated repeatedly. Which transformation should you apply to optimize this workflow and avoid recomputing the upstream lineage?

Accepted Answer

Call df.cache() or df.persist() before referencing the DataFrame in multiple downstream actions.. Persisting or caching a DataFrame instructs Spark to store intermediate results in memory or disk across actions, preventing expensive recomputation of the entire lineage graph. This optimization is critical for iterative algorithms or workflows where a single DataFrame is referenced multiple times downstream.

Answer

Call df.broadcast() on the DataFrame before each downstream join operation.

Answer

Call df.repartition() to distribute the data evenly across partitions before execution.

Answer

Call df.coalesce() to reduce the number of shuffle partitions prior to the final write operation.

Question 14

A developer runs a PySpark job on Databricks that reads a large Delta table, filters on a timestamp column, and writes results to another Delta table. The job takes 45 minutes, but the Spark UI shows that 90% of task time is spent reading from the source table. The developer wants to reduce the read time. Which action should the developer take?

Accepted Answer

Enable Delta Lake data skipping by ensuring the timestamp column is in the table's partitioning or Z-ORDER BY columns.. The Spark UI indicates that the bottleneck is reading from the source Delta table. Delta Lake data skipping leverages file statistics to avoid reading irrelevant files when a filter predicate is applied. Ensuring the timestamp column is a partition column or has Z-ORDER BY applied allows the engine to prune files effectively, reducing I/O and overall job time.

Answer

Increase the number of shuffle partitions by setting spark.sql.shuffle.partitions to a higher value.

Answer

Cache the source DataFrame in memory using .cache() before applying the filter.

Answer

Repartition the source DataFrame by the timestamp column before filtering.

Question 15

A Spark Structured Streaming job on Databricks reads from a Delta table and writes micro-batches to another Delta table with a 30-second trigger. After several hours, the batch duration grows from 4 seconds to over 60 seconds and the job falls behind. The source table is compacted regularly, and the cluster has enough CPU. Which tuning action is most likely to restore the original batch duration?

Accepted Answer

Check for stateful aggregation or deduplication without a watermark and add a watermark with a bounded state cleanup so state does not grow unbounded.. A steady increase in batch duration over hours with a compacted source and adequate CPU strongly indicates unbounded state growth in a stateful streaming operation. Without a watermark, Spark retains all keys indefinitely, so each micro-batch reads and writes a larger state store. Adding a watermark with a bounded delay lets Spark evict old state, keeping per-batch work roughly constant and restoring stable batch times.

Answer

Increase spark.sql.shuffle.partitions to a very high value so each micro-batch uses more tasks and finishes faster.

Answer

Reduce the number of shuffle partitions and disable adaptive query execution so the streaming plan stays deterministic across micro-batches.

Answer

Enable Delta Lake optimized writes and tune the trigger interval so the job processes larger, less frequent micro-batches.