Databricks-DE-Assoc Troubleshooting, Monitoring, and Optimization Practice Question
A data engineer runs a Structured Streaming job that writes to a Delta table. The job processes data from a Kafka topic and uses a 10-minute watermark. After a few hours, the engineer notices that the streaming query's input rate is steady, but the processing rate has dropped significantly, and the batch duration has increased from 5 seconds to over 2 minutes. The job is running on a cluster with autoscaling enabled. Which action should the engineer take FIRST to diagnose the performance degradation?
⚠ Common exam trap
The trap here is assuming that increased batch duration always means a need for more shuffle partitions or cluster resources, rather than first checking the built-in streaming metrics that isolate the slow stage.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
Examine the Streaming Query progress metrics in the Spark UI, focusing on the 'addBatch' and 'walCommit' durations.
The Streaming Query progress metrics in the Spark UI are the primary tool for diagnosing Structured Streaming performance. They break down batch time into components like addBatch and walCommit, showing whether the delay is in data processing or Delta Lake commit operations. This targeted diagnosis avoids premature tuning and guides the engineer to the actual bottleneck.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✗
Increase the number of shuffle partitions by setting spark.sql.shuffle.partitions to a higher value.
Why it's wrong here
Increasing shuffle partitions can help with skewed aggregations, but the scenario does not indicate shuffle as the bottleneck. Blindly increasing partitions may add overhead without addressing the root cause. The first step should be diagnosis, not tuning. This action might mask symptoms temporarily but wastes resources and does not identify why batch duration increased.
- ✗
Restart the streaming job with a larger driver node to handle the increased metadata load.
Why it's wrong here
Restarting the job disrupts processing and may cause duplicate data if the checkpoint is not managed carefully. A larger driver does not necessarily address processing bottlenecks that could be on executors. Without diagnosis, this is a costly and potentially ineffective change. The driver is rarely the bottleneck for streaming processing unless there is excessive task scheduling or metadata handling.
- ✗
Repartition the Kafka source topic to increase parallelism in the streaming query.
Why it's wrong here
Repartitioning a Kafka topic is a disruptive operation that affects all consumers. While more partitions can increase parallelism, the scenario shows steady input rate, so the source is not the constraint. The bottleneck is likely in processing or writing. This action does not address the observed slowdown and could introduce operational complexity.
- ✓
Examine the Streaming Query progress metrics in the Spark UI, focusing on the 'addBatch' and 'walCommit' durations.
Why this is correct
The Spark UI's Structured Streaming tab provides per-batch timing breakdowns, including addBatch (processing) and walCommit (write-ahead log commit). These metrics pinpoint whether time is spent in computation or in Delta Lake transaction commits. Examining them first is the correct diagnostic step because it directly reveals where the latency originates without guessing.
About these practice questions
One of 276 original Databricks-DE-Assoc practice questions on Courseiva, each with a full explanation and wrong-answer analysis — not exam dumps or protected exam content. Learn why practice questions differ from exam dumps →
JA
Written and reviewed by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
Last reviewed September 2026 · checked against the official Databricks exam blueprint
This Databricks-DE-Assoc practice question is part of Courseiva's free Databricks certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the Databricks-DE-Assoc exam.