Databricks-DE-Pro Debugging and Deploying Practice Question
Exhibit
Error Log: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1.0 failed 4 times, most recent failure: Lost task 0.3 in stage 1.0: ExecutorLostFailure (executor 2 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 10.2 GB of 10 GB physical memory used.
Refer to the exhibit. Which action is the most appropriate to resolve this memory-related failure during the job execution?
⚠ Common exam trap
Candidates frequently try to solve memory errors by tuning Spark shuffle partitions or changing code logic, ignoring that an outright hardware memory ceiling requires a memory-optimized instance type.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
Upgrade the cluster to an instance type with higher memory capacity.
The error indicates an Out of Memory (OOM) condition where the container exceeded its allocated memory limit. Increasing the instance type to a memory-optimized VM size provides more RAM per executor, allowing Spark to process larger partitions without triggering the YARN memory killer. This is a common bottleneck in memory-intensive operations like wide transformations, where adjusting the cluster configuration is more effective than attempting to optimize complex query logic.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✗
Decrease the number of partitions using spark.sql.shuffle.partitions.
Why it's wrong here
Reducing shuffle partitions results in larger individual data blocks per task. If the cluster is already struggling with memory limits, larger partitions will exacerbate the OOM issue, leading to more frequent executor failures rather than resolving them. This setting should be increased to handle large datasets effectively.
- ✓
Upgrade the cluster to an instance type with higher memory capacity.
Why this is correct
Upgrading to a memory-optimized instance family directly addresses the physical memory constraint identified in the logs. By increasing the memory allocated to each container, the executors can accommodate the memory footprint required for the transformation, preventing the container from being killed by the underlying resource manager.
- ✗
Enable autoscaling to add more nodes to the cluster.
Why it's wrong here
Adding more nodes increases total cluster compute capacity but does not increase the amount of memory available to a single executor container. If the issue is a single task exceeding the container limit, distributing more tasks across additional nodes will not prevent that specific container from failing.
- ✗
Change the file format from Parquet to CSV for better performance.
Why it's wrong here
Parquet is a highly optimized columnar format that is significantly more memory-efficient than CSV. Switching to CSV would increase the data footprint in memory and likely lead to even worse memory pressure and performance degradation. It is not a valid strategy for resolving memory-related OOM errors.
Visual reference
About these practice questions
One of 267 original Databricks-DE-Pro practice questions on Courseiva, each with a full explanation and wrong-answer analysis — not exam dumps or protected exam content. Learn why practice questions differ from exam dumps →
JA
Written and reviewed by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
Last reviewed September 2026 · checked against the official Databricks exam blueprint
This Databricks-DE-Pro practice question is part of Courseiva's free Databricks certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the Databricks-DE-Pro exam.