Databricks-Spark-Assoc Troubleshooting and Tuning DataFrame Apps — All Questions With Answers

Question 1hardmultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A Spark SQL query is running slowly because of a large broadcast join that keeps failing. What is the most likely reason for this failure?

Question 2mediummultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

Which TWO actions can help prevent data skew in a join operation?

Question 3mediummultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

Which action allows Spark to perform 'predicate pushdown' when reading data from a Parquet file?

Question 4hardmultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

Your Spark job is failing with an OutOfMemoryError (OOM) during a shuffle-heavy transformation. Which TWO configurations should you investigate to troubleshoot this memory pressure?

Question 5mediummultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

Which action should be prioritized when observing that a DataFrame job is performing frequent 'spilling to disk' during a sort-merge join?

Question 6hardmultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

Which THREE factors can lead to 'Data Skew' in a Spark DataFrame application?

Question 7easymultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

What is the primary function of the Spark UI 'SQL' tab when troubleshooting a poorly performing DataFrame application?

Question 8hardmultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

You are optimizing a Spark SQL query that performs a sort on a large dataset. The job is currently failing because the sort operation exceeds the memory of the executors. Which configuration should be adjusted to allow the sort to spill to disk?

Question 9mediummulti select

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

Which THREE actions help mitigate the 'small files problem' in Spark?

Question 10mediummultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A developer runs a PySpark job on Databricks that joins a 400 GB Delta table with a 2 GB Parquet table. The job is slow and the Spark UI shows that the 2 GB table is being shuffled across the network for every join stage. The developer wants to avoid this shuffle. Which change should be made to the join?

Question 11easymultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A data engineer notices that a Spark job on Databricks is running slowly and the Spark UI shows that many tasks are spilling to disk. The job performs a groupBy on a high-cardinality key. Which configuration change is most likely to reduce spilling and improve performance?

Question 12mediummultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A PySpark job on Databricks joins a 500 GB Delta table with a 2 GB Delta table using a broadcast hash join. The job runs out of memory on the executors. You have already increased executor memory, but the issue persists. What should you do to resolve the memory pressure while keeping the join efficient?

Question 13hardmulti select

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A Databricks job processing a Delta table with 50,000 small files is slow because each task opens many files. The developer wants to reduce the number of files read per task without rewriting the table. Which two actions should be taken? (Choose two.)

Question 14mediummultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A Spark application on Databricks reads a Parquet dataset with 10,000 small files, each averaging 10 KB. The job performs a simple aggregation and takes hours to complete. The developer wants to improve performance without changing the aggregation logic. Which action is most effective?

Question 15mediummultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A PySpark job on Databricks repeatedly reads the same 200 GB Delta table with different filters across several transformations, and the driver has 64 GB of RAM. You want to avoid recomputing the full table scan for each action while keeping memory pressure manageable. Which approach should you use?

Question 16easymultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A Spark DataFrame job on Databricks reads a large Parquet dataset and performs a groupBy aggregation. The job is slow, and the Spark UI shows that most tasks are reading the same partitions repeatedly. Which configuration change would most directly improve performance?

Question 17mediummultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A PySpark job on Databricks processes a 200 GB Delta table using df.groupBy('customer_id').agg(sum('amount')). The job runs for 3 hours, but Spark UI shows that 95% of tasks finish in 2 minutes while a few tasks run for over 2 hours. The shuffle read size is evenly distributed. Which Spark configuration change is most likely to resolve the long-running tasks?

Question 18mediummultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A PySpark DataFrame job on Databricks repeatedly runs out of memory during a groupBy aggregation on a 500 GB Parquet dataset. The Spark UI shows that the task with the largest shuffle read processes 20 times more records than the median task, and the job fails with java.lang.OutOfMemoryError on the executor. You want to resolve the failure without changing the aggregation logic. Which action should you take?

Question 19hardmultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A nightly PySpark job joins a 900 GB transactions DataFrame with a 3 GB customers DataFrame. The join key is heavily skewed: one customer ID accounts for roughly 35 percent of all transactions, and a single task runs for hours while the rest of the stage finishes quickly. You cannot change the source data or the join key. Which approach most directly resolves the skew?

Question 20hardmultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A Spark Structured Streaming job on Databricks reads from a Kafka topic and writes to a Delta table using foreachBatch. The job processes 10,000 records per second, but after several hours, the batch duration steadily increases and eventually the job falls behind. The Spark UI shows that the number of active batches is growing and shuffle spill is high. Which action should you take to stabilize the streaming job?

Question 21mediummultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

You are tuning a Spark SQL query that performs a groupBy on a high-cardinality key. The job is spilling heavily to disk during the aggregation. Which configuration change is most likely to reduce spilling?

Question 22hardmultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A streaming PySpark job on Databricks uses `foreachBatch` to upsert micro-batches into a Delta table. After a few hours the Spark UI shows a steadily growing number of tasks per batch and increasing batch duration, even though the input rate is constant. The state store directory has grown to several hundred gigabytes. What is the most likely cause?

Question 23hardmulti select

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A Spark job on Databricks is experiencing out-of-memory errors on the executors during a groupBy aggregation on a large skewed dataset. The Spark UI shows that a few tasks are processing significantly more data than others. Which two actions should you take to mitigate the skew and improve performance? (Choose two.)

Question 24mediummulti select

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A PySpark DataFrame job on Databricks fails with `java.lang.OutOfMemoryError: GC overhead limit exceeded` on the executors during a wide transformation, and the Spark UI shows many tasks with very large shuffle spill and record counts. Which two changes are most likely to reduce executor memory pressure for this job? (Choose two.)

Question 25hardmulti select

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

You are troubleshooting a Spark job that is running slowly due to data skew in a join. The skewed key is known to be a few values that dominate the data. Which TWO strategies can mitigate the skew? (Choose two.)

Question 26mediummultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A Spark job is experiencing data skew during a join operation on a key column. Which strategy is most effective for mitigating this issue without changing the business logic?

Question 27hardmultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

Which TWO of the following techniques effectively reduce the shuffle volume in a Databricks Spark job?

Question 28mediummultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

Refer to the exhibit. Which action is the most likely cause of the error shown in the Spark job logs?

Exhibit

Error: java.lang.OutOfMemoryError: Java heap space
State: FAILED
Task: 452
Stage: 12
Action: collect()

Question 29hardmultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A developer needs to optimize a Spark application that performs repetitive filtering and grouping on the same large DataFrame. Which feature should they implement to improve performance?

Question 30mediummultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

Which THREE factors should a developer consider when choosing a partition count for a shuffle operation?

Question 31mediummultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

Which configuration parameter should be adjusted to change the default number of partitions when reading from a shuffle-heavy operation?

Question 32hardmultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

Refer to the exhibit. What is the most likely cause of the repeated ExecutorLostFailure messages in the logs?

Exhibit

24/05/20 10:00:00 INFO TaskSchedulerImpl: Adding task set 1.0 with 200 tasks
24/05/20 10:05:00 WARN TaskSetManager: Lost task 15.0 in stage 1.0: ExecutorLostFailure

Question 33mediummultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

Which file format is best suited for performance-critical Spark applications that require efficient schema enforcement and column pruning?

Question 34hardmultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

When a Spark job is stuck in a shuffle phase, what is the most effective first step to identify the root cause of the performance bottleneck?

Question 35mediummultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A job is reading a huge amount of data from a table, but only uses three columns. Which optimization technique will provide the most significant I/O performance benefit?

Question 36hardmultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

Refer to the exhibit. What is the best way to resolve this error?

Exhibit

Error: org.apache.spark.SparkException: Job aborted due to stage failure: Total size of serialized results of 5000 tasks (2048 MB) is bigger than spark.driver.maxResultSize (1024 MB)

Question 37mediummultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

When analyzing a Spark job's execution plan, what does a 'BroadcastHashJoin' indicate compared to a 'SortMergeJoin'?

Question 38mediummultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

Your Spark application is experiencing severe data skew while performing a join between a large fact table and a small dimension table. Which technique should you apply to optimize performance?

Question 39mediummultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A developer notices a Spark job is failing with an OutOfMemoryError during a join operation on two large tables. The join key is highly skewed, causing one task to process significantly more data than others. Which technique should be applied to resolve this skew?

Question 40easymultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

Which action should be taken to optimize a Spark application that performs multiple operations on the same DataFrame and shows evidence of redundant re-computations in the Spark UI DAG visualization?

Question 41hardmulti select

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A Spark job is running slower than expected due to excessive shuffling. Which TWO of the following techniques would directly reduce the volume of data transferred over the network?

Question 42mediummultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

Refer to the exhibit. You are reviewing the logs for a Spark application and notice the warning regarding broadcasting a large task binary. What is the most likely cause and mitigation?

Exhibit

24/05/10 10:00:00 WARN DAGScheduler: Broadcasting large task binary with size 12.5 MiB

Question 43mediummultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

You are debugging a PySpark DataFrame job on Databricks that performs multiple transformations and actions on a large delta table. You notice that the execution plan shows redundant computations where the same upstream DataFrame is evaluated repeatedly. Which transformation should you apply to optimize this workflow and avoid recomputing the upstream lineage?

Question 44mediummultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A developer runs a PySpark job on Databricks that reads a large Delta table, filters on a timestamp column, and writes results to another Delta table. The job takes 45 minutes, but the Spark UI shows that 90% of task time is spent reading from the source table. The developer wants to reduce the read time. Which action should the developer take?

Question 45hardmultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A Spark Structured Streaming job on Databricks reads from a Delta table and writes micro-batches to another Delta table with a 30-second trigger. After several hours, the batch duration grows from 4 seconds to over 60 seconds and the job falls behind. The source table is compacted regularly, and the cluster has enough CPU. Which tuning action is most likely to restore the original batch duration?

Question 46hardmultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A developer is troubleshooting a Spark job that fails with an OutOfMemoryError on the driver. The job collects a large DataFrame to the driver using .collect() and then processes it locally. The developer wants to avoid the driver OOM while still obtaining the results. Which approach is most appropriate?

Question 47mediummultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A PySpark job on Databricks repeatedly calls `df.count()` and `df.show()` inside a loop across 40 iterations, and the Spark UI shows the identical lineage being recomputed on every iteration even though the source Delta table is unchanged. You want to avoid re-executing the upstream transformations without materializing the data to disk. What should you do?

Question 48easymultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A developer notices that a PySpark DataFrame transformation chain runs a full scan of a Delta table each time a new action is invoked, even though the source data has not changed. The developer wants to persist the intermediate DataFrame in memory across actions. Which method should be used?

Question 49mediummultiple choice

Study the full Python automation breakdown →

A Databricks notebook job calls a PySpark UDF built on a Python function that performs string parsing. The job completes successfully on a small sample, but on the full production dataset it fails with a PythonException and the executor logs show high garbage collection time. Which change is the most appropriate first step to make the job reliable without changing business logic?

Question 50hardmulti select

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A developer is tuning a Spark job that performs a join between a large fact table and a medium-sized dimension table. The job suffers from data skew, with a few keys having a disproportionately large number of rows. The developer wants to mitigate the skew. Which two actions are most effective? (Choose two.)

Question 51hardmultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A Spark job performing a join between a 10 GB table and a 5 MB lookup table is running slowly, and the physical plan shows a SortMergeJoin. You want to avoid the shuffle. What should you do?

Question 52hardmultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A structured streaming DataFrame writes to a Delta table with a foreachBatch function that performs an upsert. After a cluster restart, the stream reprocesses some micro-batches and duplicate rows appear in the target table. The foreachBatch code already uses MERGE keyed on a unique id. Which change best prevents duplicates after restart?

Question 53hardmultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A Databricks job joins a 500 GB sales table with a 300 GB returns table on a customer_id key. A few customer_id values account for a large fraction of rows on both sides, and the job fails with executor OOM during the join. The developer wants to distribute the hot keys across more partitions without changing the query logic. Which technique should be used?

Question 54hardmultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A Spark job writes a large DataFrame to a Delta table partitioned by date. The job is taking much longer than expected, and the Spark UI shows that many tasks are writing very small files. You have already set spark.sql.shuffle.partitions to 200. What is the most effective way to reduce the number of small files written?

Question 55mediummultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A developer runs a PySpark job on Databricks that filters a Delta table and then calls count() and show(). The Spark UI shows two separate scans of the same table. The developer wants to avoid the second scan without changing the filter logic. Which action should be taken?

Question 56easymultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A developer runs a PySpark job that joins a 10 GB DataFrame with a 50 MB lookup DataFrame. The job takes far longer than expected, and the Spark UI shows a SortMergeJoin with a large shuffle read and write for both sides. The developer wants the smallest change that most improves performance. Which action should the developer take?

Question 57mediummulti select

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A PySpark DataFrame job on Databricks runs slowly. Inspection of the Spark UI shows that a shuffle stage writes 200 partitions but downstream stages process only a few, and the physical plan shows an Exchange before a filter. Which two changes are most likely to improve performance? (Choose two.)

Question 58mediummultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A Databricks job reads a Parquet dataset, applies a chain of `withColumn` transformations, and writes it back with `.write.mode("overwrite").parquet(path)`. The Spark UI shows 6000 small output files totaling 50 GB, and a downstream reader is slow because of per-file overhead. You want to reduce the file count while keeping the write as a Spark-native operation. What should you do?

Question 59mediummultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A Spark application reads a large CSV file and performs a series of transformations, including a filter and a join with a small lookup table. The job is running slowly, and the Spark UI shows that the CSV parsing stage is taking a long time. Which action would most improve performance?

Question 60mediummultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A streaming DataFrame job on Databricks writes to a Delta table every 10 seconds. Over several hours, the number of files in the target directory grows into the hundreds of thousands, and downstream reads slow dramatically. The job uses foreachBatch with a write that produces many small files per micro-batch. Which action should be taken to reduce the small-files problem for this streaming write?

Question 61easymultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A developer notices that a DataFrame transformation chain is executed twice: once for a count action used for logging and again for a write action. The source is a large Delta table and the repeated scan adds several minutes. Which action avoids the duplicate computation with the least risk?

Question 62hardmultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A PySpark job reads a Delta table, applies a filter on a timestamp column, and then performs a window function partitioned by customer_id ordered by event_time. The job is slow, and the physical plan shows the filter is applied after the window. The developer wants the filter to reduce data before the window shuffle. Which action should the developer take?

Question 63hardmultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A job joining a large fact table with a small dimension table runs out of memory on executors during the join. The dimension table is about 40 MB after filtering and the configured spark.sql.autoBroadcastJoinThreshold is 10 MB. The join key is highly skewed in the fact table. Which action is most appropriate?

Question 64easymultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A developer notices that a Spark DataFrame job on Databricks is running slowly and the Spark UI shows that many tasks are reading from a Delta table with a large number of small files. The job performs a filter on a date column and then aggregates results. Which optimization technique will most directly improve read performance in this scenario?

Question 65easymultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A Databricks job fails with an OutOfMemoryError on the driver. The job collects a large DataFrame to the driver for local processing. Which action should you take to resolve this?

Question 66hardmultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A Spark job reads a large Parquet dataset, performs a groupBy on a high-cardinality column, and writes the result to a Delta table. The job fails with a FetchFailedException on a particular executor. The Spark UI shows that the executor had sufficient memory but the shuffle fetch failed due to a connection reset. Which configuration change is most likely to resolve this issue?

Question 1hardmultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A Spark SQL query is running slowly because of a large broadcast join that keeps failing. What is the most likely reason for this failure?

Question 2mediummultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

Which TWO actions can help prevent data skew in a join operation?

Question 3mediummultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

Which action allows Spark to perform 'predicate pushdown' when reading data from a Parquet file?

Question 4hardmultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

Your Spark job is failing with an OutOfMemoryError (OOM) during a shuffle-heavy transformation. Which TWO configurations should you investigate to troubleshoot this memory pressure?

Question 5mediummultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

Which action should be prioritized when observing that a DataFrame job is performing frequent 'spilling to disk' during a sort-merge join?

Question 6hardmultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

Which THREE factors can lead to 'Data Skew' in a Spark DataFrame application?

Question 7easymultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

What is the primary function of the Spark UI 'SQL' tab when troubleshooting a poorly performing DataFrame application?

Question 8hardmultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

You are optimizing a Spark SQL query that performs a sort on a large dataset. The job is currently failing because the sort operation exceeds the memory of the executors. Which configuration should be adjusted to allow the sort to spill to disk?

Question 9mediummulti select

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

Which THREE actions help mitigate the 'small files problem' in Spark?

Question 10mediummultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A developer runs a PySpark job on Databricks that joins a 400 GB Delta table with a 2 GB Parquet table. The job is slow and the Spark UI shows that the 2 GB table is being shuffled across the network for every join stage. The developer wants to avoid this shuffle. Which change should be made to the join?

Question 11easymultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A data engineer notices that a Spark job on Databricks is running slowly and the Spark UI shows that many tasks are spilling to disk. The job performs a groupBy on a high-cardinality key. Which configuration change is most likely to reduce spilling and improve performance?

Question 12mediummultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A PySpark job on Databricks joins a 500 GB Delta table with a 2 GB Delta table using a broadcast hash join. The job runs out of memory on the executors. You have already increased executor memory, but the issue persists. What should you do to resolve the memory pressure while keeping the join efficient?

Question 13hardmulti select

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A Databricks job processing a Delta table with 50,000 small files is slow because each task opens many files. The developer wants to reduce the number of files read per task without rewriting the table. Which two actions should be taken? (Choose two.)

Question 14mediummultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A Spark application on Databricks reads a Parquet dataset with 10,000 small files, each averaging 10 KB. The job performs a simple aggregation and takes hours to complete. The developer wants to improve performance without changing the aggregation logic. Which action is most effective?

Question 15mediummultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A PySpark job on Databricks repeatedly reads the same 200 GB Delta table with different filters across several transformations, and the driver has 64 GB of RAM. You want to avoid recomputing the full table scan for each action while keeping memory pressure manageable. Which approach should you use?

Question 16easymultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A Spark DataFrame job on Databricks reads a large Parquet dataset and performs a groupBy aggregation. The job is slow, and the Spark UI shows that most tasks are reading the same partitions repeatedly. Which configuration change would most directly improve performance?

Question 17mediummultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A PySpark job on Databricks processes a 200 GB Delta table using df.groupBy('customer_id').agg(sum('amount')). The job runs for 3 hours, but Spark UI shows that 95% of tasks finish in 2 minutes while a few tasks run for over 2 hours. The shuffle read size is evenly distributed. Which Spark configuration change is most likely to resolve the long-running tasks?

Question 18mediummultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A PySpark DataFrame job on Databricks repeatedly runs out of memory during a groupBy aggregation on a 500 GB Parquet dataset. The Spark UI shows that the task with the largest shuffle read processes 20 times more records than the median task, and the job fails with java.lang.OutOfMemoryError on the executor. You want to resolve the failure without changing the aggregation logic. Which action should you take?

Question 19hardmultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A nightly PySpark job joins a 900 GB transactions DataFrame with a 3 GB customers DataFrame. The join key is heavily skewed: one customer ID accounts for roughly 35 percent of all transactions, and a single task runs for hours while the rest of the stage finishes quickly. You cannot change the source data or the join key. Which approach most directly resolves the skew?

Question 20hardmultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A Spark Structured Streaming job on Databricks reads from a Kafka topic and writes to a Delta table using foreachBatch. The job processes 10,000 records per second, but after several hours, the batch duration steadily increases and eventually the job falls behind. The Spark UI shows that the number of active batches is growing and shuffle spill is high. Which action should you take to stabilize the streaming job?

Question 21mediummultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

You are tuning a Spark SQL query that performs a groupBy on a high-cardinality key. The job is spilling heavily to disk during the aggregation. Which configuration change is most likely to reduce spilling?

Question 22hardmultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A streaming PySpark job on Databricks uses `foreachBatch` to upsert micro-batches into a Delta table. After a few hours the Spark UI shows a steadily growing number of tasks per batch and increasing batch duration, even though the input rate is constant. The state store directory has grown to several hundred gigabytes. What is the most likely cause?

Question 23hardmulti select

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A Spark job on Databricks is experiencing out-of-memory errors on the executors during a groupBy aggregation on a large skewed dataset. The Spark UI shows that a few tasks are processing significantly more data than others. Which two actions should you take to mitigate the skew and improve performance? (Choose two.)

Question 24mediummulti select

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A PySpark DataFrame job on Databricks fails with `java.lang.OutOfMemoryError: GC overhead limit exceeded` on the executors during a wide transformation, and the Spark UI shows many tasks with very large shuffle spill and record counts. Which two changes are most likely to reduce executor memory pressure for this job? (Choose two.)

Question 25hardmulti select

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

You are troubleshooting a Spark job that is running slowly due to data skew in a join. The skewed key is known to be a few values that dominate the data. Which TWO strategies can mitigate the skew? (Choose two.)

Question 26mediummultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A Spark job is experiencing data skew during a join operation on a key column. Which strategy is most effective for mitigating this issue without changing the business logic?

Question 27hardmultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

Which TWO of the following techniques effectively reduce the shuffle volume in a Databricks Spark job?

Question 28mediummultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

Refer to the exhibit. Which action is the most likely cause of the error shown in the Spark job logs?

Exhibit

Error: java.lang.OutOfMemoryError: Java heap space
State: FAILED
Task: 452
Stage: 12
Action: collect()

Question 29hardmultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A developer needs to optimize a Spark application that performs repetitive filtering and grouping on the same large DataFrame. Which feature should they implement to improve performance?

Question 30mediummultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

Which THREE factors should a developer consider when choosing a partition count for a shuffle operation?

Question 31mediummultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

Which configuration parameter should be adjusted to change the default number of partitions when reading from a shuffle-heavy operation?

Question 32hardmultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

Refer to the exhibit. What is the most likely cause of the repeated ExecutorLostFailure messages in the logs?

Exhibit

24/05/20 10:00:00 INFO TaskSchedulerImpl: Adding task set 1.0 with 200 tasks
24/05/20 10:05:00 WARN TaskSetManager: Lost task 15.0 in stage 1.0: ExecutorLostFailure

Question 33mediummultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

Which file format is best suited for performance-critical Spark applications that require efficient schema enforcement and column pruning?

Question 34hardmultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

When a Spark job is stuck in a shuffle phase, what is the most effective first step to identify the root cause of the performance bottleneck?

Question 35mediummultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A job is reading a huge amount of data from a table, but only uses three columns. Which optimization technique will provide the most significant I/O performance benefit?

Question 36hardmultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

Refer to the exhibit. What is the best way to resolve this error?

Exhibit

Error: org.apache.spark.SparkException: Job aborted due to stage failure: Total size of serialized results of 5000 tasks (2048 MB) is bigger than spark.driver.maxResultSize (1024 MB)

Question 37mediummultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

When analyzing a Spark job's execution plan, what does a 'BroadcastHashJoin' indicate compared to a 'SortMergeJoin'?

Question 38mediummultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

Your Spark application is experiencing severe data skew while performing a join between a large fact table and a small dimension table. Which technique should you apply to optimize performance?

Question 39mediummultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A developer notices a Spark job is failing with an OutOfMemoryError during a join operation on two large tables. The join key is highly skewed, causing one task to process significantly more data than others. Which technique should be applied to resolve this skew?

Question 40easymultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

Which action should be taken to optimize a Spark application that performs multiple operations on the same DataFrame and shows evidence of redundant re-computations in the Spark UI DAG visualization?

Question 41hardmulti select

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A Spark job is running slower than expected due to excessive shuffling. Which TWO of the following techniques would directly reduce the volume of data transferred over the network?

Question 42mediummultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

Refer to the exhibit. You are reviewing the logs for a Spark application and notice the warning regarding broadcasting a large task binary. What is the most likely cause and mitigation?

Exhibit

24/05/10 10:00:00 WARN DAGScheduler: Broadcasting large task binary with size 12.5 MiB

Question 43mediummultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

You are debugging a PySpark DataFrame job on Databricks that performs multiple transformations and actions on a large delta table. You notice that the execution plan shows redundant computations where the same upstream DataFrame is evaluated repeatedly. Which transformation should you apply to optimize this workflow and avoid recomputing the upstream lineage?

Question 44mediummultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A developer runs a PySpark job on Databricks that reads a large Delta table, filters on a timestamp column, and writes results to another Delta table. The job takes 45 minutes, but the Spark UI shows that 90% of task time is spent reading from the source table. The developer wants to reduce the read time. Which action should the developer take?

Question 45hardmultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A Spark Structured Streaming job on Databricks reads from a Delta table and writes micro-batches to another Delta table with a 30-second trigger. After several hours, the batch duration grows from 4 seconds to over 60 seconds and the job falls behind. The source table is compacted regularly, and the cluster has enough CPU. Which tuning action is most likely to restore the original batch duration?

Question 46hardmultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A developer is troubleshooting a Spark job that fails with an OutOfMemoryError on the driver. The job collects a large DataFrame to the driver using .collect() and then processes it locally. The developer wants to avoid the driver OOM while still obtaining the results. Which approach is most appropriate?

Question 47mediummultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A PySpark job on Databricks repeatedly calls `df.count()` and `df.show()` inside a loop across 40 iterations, and the Spark UI shows the identical lineage being recomputed on every iteration even though the source Delta table is unchanged. You want to avoid re-executing the upstream transformations without materializing the data to disk. What should you do?

Question 48easymultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A developer notices that a PySpark DataFrame transformation chain runs a full scan of a Delta table each time a new action is invoked, even though the source data has not changed. The developer wants to persist the intermediate DataFrame in memory across actions. Which method should be used?

Question 49mediummultiple choice

Study the full Python automation breakdown →

A Databricks notebook job calls a PySpark UDF built on a Python function that performs string parsing. The job completes successfully on a small sample, but on the full production dataset it fails with a PythonException and the executor logs show high garbage collection time. Which change is the most appropriate first step to make the job reliable without changing business logic?

Question 50hardmulti select

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A developer is tuning a Spark job that performs a join between a large fact table and a medium-sized dimension table. The job suffers from data skew, with a few keys having a disproportionately large number of rows. The developer wants to mitigate the skew. Which two actions are most effective? (Choose two.)

Question 51hardmultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A Spark job performing a join between a 10 GB table and a 5 MB lookup table is running slowly, and the physical plan shows a SortMergeJoin. You want to avoid the shuffle. What should you do?

Question 52hardmultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A structured streaming DataFrame writes to a Delta table with a foreachBatch function that performs an upsert. After a cluster restart, the stream reprocesses some micro-batches and duplicate rows appear in the target table. The foreachBatch code already uses MERGE keyed on a unique id. Which change best prevents duplicates after restart?

Question 53hardmultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A Databricks job joins a 500 GB sales table with a 300 GB returns table on a customer_id key. A few customer_id values account for a large fraction of rows on both sides, and the job fails with executor OOM during the join. The developer wants to distribute the hot keys across more partitions without changing the query logic. Which technique should be used?

Question 54hardmultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A Spark job writes a large DataFrame to a Delta table partitioned by date. The job is taking much longer than expected, and the Spark UI shows that many tasks are writing very small files. You have already set spark.sql.shuffle.partitions to 200. What is the most effective way to reduce the number of small files written?

Question 55mediummultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A developer runs a PySpark job on Databricks that filters a Delta table and then calls count() and show(). The Spark UI shows two separate scans of the same table. The developer wants to avoid the second scan without changing the filter logic. Which action should be taken?

Question 56easymultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A developer runs a PySpark job that joins a 10 GB DataFrame with a 50 MB lookup DataFrame. The job takes far longer than expected, and the Spark UI shows a SortMergeJoin with a large shuffle read and write for both sides. The developer wants the smallest change that most improves performance. Which action should the developer take?

Question 57mediummulti select

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A PySpark DataFrame job on Databricks runs slowly. Inspection of the Spark UI shows that a shuffle stage writes 200 partitions but downstream stages process only a few, and the physical plan shows an Exchange before a filter. Which two changes are most likely to improve performance? (Choose two.)

Question 58mediummultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A Databricks job reads a Parquet dataset, applies a chain of `withColumn` transformations, and writes it back with `.write.mode("overwrite").parquet(path)`. The Spark UI shows 6000 small output files totaling 50 GB, and a downstream reader is slow because of per-file overhead. You want to reduce the file count while keeping the write as a Spark-native operation. What should you do?

Question 59mediummultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A Spark application reads a large CSV file and performs a series of transformations, including a filter and a join with a small lookup table. The job is running slowly, and the Spark UI shows that the CSV parsing stage is taking a long time. Which action would most improve performance?

Question 60mediummultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A streaming DataFrame job on Databricks writes to a Delta table every 10 seconds. Over several hours, the number of files in the target directory grows into the hundreds of thousands, and downstream reads slow dramatically. The job uses foreachBatch with a write that produces many small files per micro-batch. Which action should be taken to reduce the small-files problem for this streaming write?

Question 61easymultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A developer notices that a DataFrame transformation chain is executed twice: once for a count action used for logging and again for a write action. The source is a large Delta table and the repeated scan adds several minutes. Which action avoids the duplicate computation with the least risk?

Question 62hardmultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A PySpark job reads a Delta table, applies a filter on a timestamp column, and then performs a window function partitioned by customer_id ordered by event_time. The job is slow, and the physical plan shows the filter is applied after the window. The developer wants the filter to reduce data before the window shuffle. Which action should the developer take?

Question 63hardmultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A job joining a large fact table with a small dimension table runs out of memory on executors during the join. The dimension table is about 40 MB after filtering and the configured spark.sql.autoBroadcastJoinThreshold is 10 MB. The join key is highly skewed in the fact table. Which action is most appropriate?

Question 64easymultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A developer notices that a Spark DataFrame job on Databricks is running slowly and the Spark UI shows that many tasks are reading from a Delta table with a large number of small files. The job performs a filter on a date column and then aggregates results. Which optimization technique will most directly improve read performance in this scenario?

Question 65easymultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A Databricks job fails with an OutOfMemoryError on the driver. The job collects a large DataFrame to the driver for local processing. Which action should you take to resolve this?

Question 66hardmultiple choice

Read the full Troubleshooting and Tuning DataFrame Apps explanation →

A Spark job reads a large Parquet dataset, performs a groupBy on a high-cardinality column, and writes the result to a Delta table. The job fails with a FetchFailedException on a particular executor. The Spark UI shows that the executor had sufficient memory but the shuffle fetch failed due to a connection reset. Which configuration change is most likely to resolve this issue?