Databricks-ML-Assoc ML Workflows Practice Question
An ML engineer is orchestrating an end-to-end machine learning pipeline using Databricks Jobs. The pipeline consists of data preparation, distributed hyperparameter tuning with Hyperopt, and model registration. The engineer needs to pass the best-performing model's run ID from the Hyperopt task to the subsequent model registration task dynamically. Which mechanism should the engineer use in Databricks Jobs to achieve this?
⚠ Common exam trap
Candidates often assume they need to read and write temporary state files to DBFS or external cloud storage between tasks, forgetting that native task values are specifically engineered for this exact inter-task communication pattern.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
Use dbutils.jobs.taskValues to set the run ID in the producer task and get the value in the consumer task.
Databricks Jobs task values allow tasks to pass small amounts of data, such as strings or numeric metrics, downstream to other tasks within the same workflow. Using dbutils.jobs.taskValues.set inside the Hyperopt notebook and dbutils.jobs.taskValues.get in the registration notebook enables seamless dynamic orchestration without relying on external storage solutions.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✗
Store the run ID in a shared Delta table inside the workspace and query it from the downstream task.
Why it's wrong here
Writing the run ID to a Delta table introduces unnecessary overhead, storage management complexity, and potential concurrency race conditions. Databricks Jobs provides dedicated lightweight mechanisms specifically designed for passing small state variables between dependent workflow tasks.
- ✗
Write the run ID to a text file in DBFS root and read it back in the downstream task.
Why it's wrong here
Using DBFS root for inter-task communication is an outdated anti-pattern that clutters storage and lacks structured error handling. Modern Databricks Jobs workflows provide native task values that handle parameter passing cleanly and efficiently in memory.
- ✓
Use dbutils.jobs.taskValues to set the run ID in the producer task and get the value in the consumer task.
Why this is correct
Task values provide a robust, native API to set key-value pairs in one workflow task and retrieve them in subsequent tasks. This avoids external storage overhead and integrates directly into the Databricks Jobs execution context and UI.
- ✗
Pass the run ID as a static parameter defined in the JSON job configuration before execution begins.
Why it's wrong here
Static parameters are resolved when the job is defined, so they cannot carry a run ID produced by an earlier task at execution time. It is tempting because parameters are the standard configuration mechanism, but dynamic cross-task values require task values or the Jobs API output mechanism.
About these practice questions
This Databricks-ML-Assoc question is part of Courseiva's 319-question bank — original exam-style content with full explanations and wrong-answer analysis, never real exam questions or exam dumps. Learn why practice questions differ from exam dumps →
JA
Written and reviewed by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
Last reviewed September 2026 · checked against the official Databricks exam blueprint
This Databricks-ML-Assoc practice question is part of Courseiva's free Databricks certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the Databricks-ML-Assoc exam.