Databricks-DE-Pro Debugging and Deploying Practice Question
A data engineer is troubleshooting a Databricks job that fails with a `SparkException: Job aborted due to stage failure` and the error log shows `java.lang.OutOfMemoryError: GC overhead limit exceeded` on an executor. The job processes a large dataset using a `groupByKey` operation. Which action should the engineer take to resolve the issue while minimizing changes to the existing code?
⚠ Common exam trap
The trap here is thinking that adding more memory or nodes will solve an OOM caused by an inefficient shuffle operation like `groupByKey`.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
Replace `groupByKey` with `reduceByKey` to reduce memory pressure by combining values locally before shuffling.
The `groupByKey` operation shuffles all values for each key without local aggregation, which can lead to excessive memory consumption and GC overhead. Replacing it with `reduceByKey` enables map-side combining, significantly reducing the shuffle size and memory footprint. This is the most direct and code-minimal fix. Other options either mask the symptom or do not address the root cause.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✗
Increase the executor memory by adding `spark.executor.memory=16g` to the cluster Spark configuration.
Why it's wrong here
Increasing executor memory may temporarily alleviate the symptom but does not fix the underlying inefficiency of `groupByKey`, which shuffles all values. It can lead to larger GC pauses and higher costs. The problem is algorithmic, not just resource-related. While it might delay the failure, it is not a robust solution and does not minimize changes to the logic.
- ✗
Set `spark.sql.shuffle.partitions` to a higher value to increase the number of partitions after shuffle.
Why it's wrong here
Increasing shuffle partitions can improve parallelism but does not reduce the total shuffle volume or memory per key. With `groupByKey`, all values for a key still go to one partition, so a hot key can still cause OOM. This setting may help with overall performance but does not address the specific GC overhead caused by large per-key value lists.
- ✓
Replace `groupByKey` with `reduceByKey` to reduce memory pressure by combining values locally before shuffling.
Why this is correct
`groupByKey` shuffles all values for a key without local aggregation, which can cause excessive memory usage and GC overhead. `reduceByKey` performs map-side combining, reducing the amount of data shuffled and lowering memory pressure. This change directly addresses the root cause while requiring minimal code modification, as both are transformations on key-value pairs. It is the most effective fix for the described symptom.
- ✗
Switch the job to use a larger cluster with more worker nodes to distribute the load.
Why it's wrong here
Adding more nodes increases parallelism but does not reduce the data shuffled by `groupByKey`. Each executor still receives a large amount of data for a single key, potentially causing OOM on individual executors. The issue is data skew and lack of local aggregation, not insufficient cluster size. This approach is costly and may not resolve the error.
Visual reference
About these practice questions
Courseiva writes every Databricks-DE-Pro question from scratch — 267 in total, each with an explanation and a wrong-answer breakdown. None are copied from real exams or dumps. Learn why practice questions differ from exam dumps →
JA
Written and reviewed by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
Last reviewed September 2026 · checked against the official Databricks exam blueprint
This Databricks-DE-Pro practice question is part of Courseiva's free Databricks certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the Databricks-DE-Pro exam.