Databricks-Spark-Assoc Spark Architecture and Components Practice Question
A data engineer runs a Spark job on a Databricks cluster using the default FIFO scheduler. They notice that a long-running job is holding all cluster resources, and short ad-hoc queries submitted later are stuck waiting. The engineer wants to allow concurrent scheduling of multiple jobs within the same Spark application so that short jobs can run while the long job is still executing. Which Spark configuration should be set to enable this behavior?
⚠ Common exam trap
Watch out — candidates often confuse dynamic allocation with job scheduling, assuming that adding more executors will automatically allow concurrent jobs, when the real issue is the FIFO scheduling policy.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
spark.scheduler.mode=FAIR
The Fair Scheduler (spark.scheduler.mode=FAIR) enables multiple jobs within the same Spark application to share resources, so shorter jobs can run concurrently with a long-running job. The other options address resource scaling, registration timeouts, or straggler mitigation, none of which change intra-application job scheduling to allow concurrent execution.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✓
spark.scheduler.mode=FAIR
Why this is correct
Setting spark.scheduler.mode to FAIR enables the Fair Scheduler, which allows multiple jobs within the same Spark application to share cluster resources more evenly. This prevents a single long-running job from monopolizing all slots, so shorter jobs can start and complete without waiting for the long job to finish. This directly addresses the scenario.
- ✗
spark.speculation=true
Why it's wrong here
Speculative execution launches duplicate tasks for slow-running tasks to mitigate stragglers. It does not change job scheduling or allow multiple jobs to share resources concurrently. Enabling speculation might even increase resource usage, but it will not prevent a long job from blocking other jobs under the FIFO scheduler.
- ✗
spark.dynamicAllocation.enabled=true
Why it's wrong here
Dynamic allocation adjusts the number of executors based on workload but does not change how jobs are scheduled within an application. It will not allow concurrent job execution when the FIFO scheduler is holding resources for one job. While it can improve resource utilization, it does not solve the head-of-line blocking described.
- ✗
spark.scheduler.maxRegisteredResourcesWaitingTime=30s
Why it's wrong here
This property controls how long the scheduler waits for resources to be registered before giving up, not how jobs are prioritized or scheduled. It does not enable fair sharing among jobs. Increasing or decreasing this timeout will not allow short jobs to run concurrently with a long job; the FIFO scheduler will still queue them.
About these practice questions
One of 295 original Databricks-Spark-Assoc practice questions on Courseiva, each with a full explanation and wrong-answer analysis — not exam dumps or protected exam content. Learn why practice questions differ from exam dumps →
JA
Written and reviewed by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
Last reviewed September 2026 · checked against the official Databricks exam blueprint
This Databricks-Spark-Assoc practice question is part of Courseiva's free Databricks certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the Databricks-Spark-Assoc exam.