Databricks-Spark-Assoc Spark Architecture and Components Practice Question
A developer runs a Spark application on a Databricks cluster in Standard access mode. The application reads a Parquet file, applies a filter, and calls `df.cache()` before an action. During execution, the driver logs show that a stage is retried because a task failed with an executor lost error. Which component is responsible for rescheduling the failed task on another executor within the same application?
⚠ Common exam trap
Watch out — candidates often confuse the DAG Scheduler's stage-level retry logic with the Task Scheduler's task-level retry logic, leading to the wrong component being selected.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
The Task Scheduler
The Task Scheduler is the component that launches individual tasks on executors and handles their retries. When an executor is lost, the Task Scheduler receives the failure notification from the SchedulerBackend and reschedules the failed tasks on other executors, up to the configured maximum number of failures. The DAG Scheduler operates at the stage level, while the Cluster Manager and Catalyst Optimizer do not manage task-level retries.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✗
The DAG Scheduler
Why it's wrong here
The DAG Scheduler builds stages from the RDD lineage and submits them as TaskSets, but it does not track individual task failures or reschedule them. Task-level retry and locality-aware scheduling are handled by the Task Scheduler, which communicates with the SchedulerBackend. The DAG Scheduler only reacts if an entire stage fails due to a shuffle map output being lost, at which point it resubmits missing stages.
- ✓
The Task Scheduler
Why this is correct
The Task Scheduler is responsible for launching tasks on executors via the SchedulerBackend and for retrying failed tasks up to `spark.task.maxFailures`. When an executor is lost, the Task Scheduler detects the failure and reschedules the affected tasks on other available executors. It also handles speculative execution and locality preferences, making it the correct component for this scenario.
- ✗
The Catalyst Optimizer
Why it's wrong here
The Catalyst Optimizer is a query planning component that transforms logical plans into optimized physical plans. It operates before execution and does not participate in runtime task scheduling or failure recovery. While it may influence the number of stages through optimizations like filter pushdown, it never reschedules tasks. Task retries are purely a runtime concern of the execution engine.
- ✗
The Cluster Manager
Why it's wrong here
The Cluster Manager (e.g., YARN, Kubernetes, or Databricks' internal manager) allocates containers or pods and starts executors, but it does not manage task-level scheduling or retries. It only informs the driver about executor loss. Once the driver knows an executor is gone, the Task Scheduler handles rescheduling tasks. The Cluster Manager operates at the resource level, not the task level.
About these practice questions
Courseiva writes every Databricks-Spark-Assoc question from scratch — 295 in total, each with an explanation and a wrong-answer breakdown. None are copied from real exams or dumps. Learn why practice questions differ from exam dumps →
JA
Written and reviewed by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
Last reviewed September 2026 · checked against the official Databricks exam blueprint
This Databricks-Spark-Assoc practice question is part of Courseiva's free Databricks certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the Databricks-Spark-Assoc exam.