Databricks-DE-Pro Cost and Performance Optimization Practice Question
An enterprise data team runs a large nightly batch job using a standard all-purpose cluster. The job frequently fails due to cloud provider spot instance pre-emptions and takes over four hours to complete. How should the engineer refactor this architecture for maximum cost efficiency and reliability?
⚠ Common exam trap
Candidates often select 'All-Purpose Clusters' for production jobs because they are easier to manage, failing to recognize that Job clusters are cheaper and more reliable for automated tasks.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
Convert the workload to use a Databricks Job cluster configured with spot instances and automatic fallback to on-demand.
Migrating the workload from an all-purpose interactive cluster to a Databricks Job cluster running on spot instances with an automatic fallback mechanism ensures cost-effective batch execution. Job clusters consume lower DBU rates than all-purpose clusters, and spot instances drastically reduce infrastructure costs while fallback guarantees completion despite cloud provider interruptions.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✗
Provision a larger all-purpose cluster with double the worker nodes to brute-force execution speed.
Why it's wrong here
Doubling worker nodes on an all-purpose cluster increases hourly DBU and cloud costs while spot pre-emptions still terminate workers, so failures persist and spend rises. It is tempting because extra parallelism shortens runtime, and would be correct when the job is compute-bound and running on reliable on-demand capacity.
- ✓
Convert the workload to use a Databricks Job cluster configured with spot instances and automatic fallback to on-demand.
Why this is correct
Job clusters are cheaper than all-purpose clusters and terminate after the run. Configuring spot instances with automatic fallback to on-demand preserves cost savings while surviving pre-emptions, directly addressing the reliability failure and the four-hour runtime.
- ✗
Upgrade the cloud provider virtual machine family to the latest generation without changing cluster types.
Why it's wrong here
Newer VM generations change price-performance and hardware capabilities, but pre-emptions still terminate spot instances and the all-purpose cluster still lacks automatic retry, so reliability is unchanged. It is tempting because newer families often cost less per compute unit, and would be correct when the workload is stable and the constraint is raw instance price.
- ✗
Increase the Apache Spark executor memory fraction and decrease shuffle partition counts.
Why it's wrong here
Tuning executor memory fraction and shuffle partitions alters in-memory execution and disk I/O, not instance lifecycle; spot pre-emptions still kill executors and the job still fails. It is tempting because shuffle tuning often resolves spills and skew, and would be correct when the job is slow or out-of-memory rather than interrupted.
About these practice questions
This Databricks-DE-Pro question is part of Courseiva's 267-question bank — original exam-style content with full explanations and wrong-answer analysis, never real exam questions or exam dumps. Learn why practice questions differ from exam dumps →
JA
Written and reviewed by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
Last reviewed September 2026 · checked against the official Databricks exam blueprint
This Databricks-DE-Pro practice question is part of Courseiva's free Databricks certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the Databricks-DE-Pro exam.