Databricks-Spark-Assoc Spark Architecture and Components Practice Question
A data engineer observes that a Spark Structured Streaming job on Databricks processes micro-batches with steadily increasing latency over several hours. The Spark UI shows that the number of active tasks per batch stays constant, but each task processes a growing amount of state. Which architectural behavior explains this pattern?
⚠ Common exam trap
The trap here is attributing rising streaming latency to scheduling or resource allocation rather than to the growth of keyed operator state.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
Stateful operators maintain growing keyed state in the executors as more keys arrive, increasing per-task work.
Stateful streaming operators retain keyed state across micro-batches, and as distinct keys accumulate, each task must load, update, and write a larger state store. This raises per-task processing time while task parallelism stays fixed, which is exactly the pattern shown when active tasks are constant but per-task state keeps growing.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✗
Broadcast variables are being re-sent to executors on every micro-batch, adding network overhead.
Why it's wrong here
Broadcast variables are distributed once per executor and reused, not re-sent each micro-batch in a way that scales with accumulated state. This mechanism also would not produce per-task state growth, so it does not account for the steadily increasing latency described.
- ✗
The Driver is accumulating unbounded metadata from the DAG Scheduler across batches.
Why it's wrong here
The DAG Scheduler builds a fresh set of stages for each micro-batch and does not retain unbounded per-batch metadata that would slow tasks. The described symptom is growing per-task state, not driver-side scheduling overhead, so this explanation does not fit the observed increasing task processing time.
- ✗
The Cluster Manager is throttling executor allocation, reducing parallelism per batch.
Why it's wrong here
Throttled executor allocation would reduce the number of concurrent tasks or cause tasks to queue, which would be visible as fewer active tasks or increased scheduling delay. The scenario states the active task count stays constant, so resource throttling does not explain the growing per-task latency.
- ✓
Stateful operators maintain growing keyed state in the executors as more keys arrive, increasing per-task work.
Why this is correct
Stateful operations such as streaming aggregations or deduplication keep keyed state in executor memory and on disk. As new keys accumulate over hours, each task must read and update a larger state store, so per-task processing time rises even though the number of tasks stays constant, matching the observed latency growth.
About these practice questions
One of 295 original Databricks-Spark-Assoc practice questions on Courseiva, each with a full explanation and wrong-answer analysis — not exam dumps or protected exam content. Learn why practice questions differ from exam dumps →
JA
Written and reviewed by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
Last reviewed September 2026 · checked against the official Databricks exam blueprint
This Databricks-Spark-Assoc practice question is part of Courseiva's free Databricks certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the Databricks-Spark-Assoc exam.