A machine learning engineer is orchestrating a multi-step training workflow on Databricks using Databricks Jobs. The workflow includes data preprocessing, model training, and evaluation. The engineer needs to ensure that the evaluation step runs only if the training step completes successfully, and that the preprocessing step runs first. Which feature of Databricks Jobs should be used to define these dependencies?
Task dependencies in Databricks Jobs allow you to specify that a task runs only after its upstream tasks succeed. By setting the depends_on field, you create a directed acyclic graph (DAG) that enforces the order: preprocessing first, then training, then evaluation. This ensures that the evaluation step only runs if training succeeds, meeting the requirement.
Why this answer
Task dependencies via the depends_on field are the core mechanism in Databricks Jobs for orchestrating multi-task workflows. They allow you to specify that a task runs only after its upstream tasks complete successfully, enabling conditional execution. This directly satisfies the need to run preprocessing first, then training, and evaluation only if training succeeds.
Other options address compute scaling, failure recovery, or version control, not workflow dependencies.
Exam trap
The trap here is confusing job orchestration features like dependencies with cluster configuration or failure recovery mechanisms, which do not control task execution order.