PMLE Automating and Orchestrating ML Pipelines Practice Question

An ML engineer is building a Vertex AI pipeline that must run a custom training component for each of 12 hyperparameter combinations. The component is defined as a custom Python function (Lightweight Python component). The engineer wants each combination to run as a separate parallel task so the pipeline completes faster, and wants the pipeline to fail fast if any single trial fails. Which approach should the engineer take?

⚠ Common exam trap

The trap here is assuming that a Vertex AI Hyperparameter Tuning CustomJob is the same as a pipeline ParallelFor fan-out, when the former is a single managed job and the latter is a pipeline-level DAG construct.

Answer choices

Why each option matters

Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.

Correct answer & explanation

✓

Use a ParallelFor loop over the hyperparameter list and set the component's retry policy to 0.

Vertex AI Pipelines supports dsl.ParallelFor to fan out a list of values into parallel component tasks in one DAG, which is the idiomatic way to run 12 hyperparameter combinations concurrently. Setting retries to 0 on the component ensures a failed trial stops the pipeline rather than being retried and hidden. The other approaches either collapse trials into a single job, serialize them, or split them across pipelines, none of which meet the parallel fan-out and fail-fast requirements.

Answer analysis

Option-by-option breakdown

For each option: why learners choose it and why it is or isn't the right answer here.

✗
Wrap the training component in a single Vertex AI CustomJob with a hyperparameter tuning job specification and let the service manage trials.
Why it's wrong here
A Vertex AI CustomJob with a hyperparameter tuning spec is a single managed training job that runs trials under Vertex AI Hyperparameter Tuning, not a pipeline fan-out of 12 independent component tasks. The engineer specifically wants each combination as a separate pipeline task; using a tuning job would collapse the trials into one component and remove the per-combination pipeline visibility and failure semantics.
✗
Create 12 separate pipelines, one per hyperparameter combination, and schedule them with Cloud Scheduler at the same time.
Why it's wrong here
Creating 12 separate pipelines and triggering them via Cloud Scheduler splits the work across independent pipeline runs, which loses the single-pipeline DAG, shared artifacts, and unified failure semantics. The engineer wants one pipeline that fans out and fails fast; multiple pipelines cannot express that dependency graph or propagate a single failure cleanly.
✓
Use a ParallelFor loop over the hyperparameter list and set the component's retry policy to 0.
Why this is correct
A ParallelFor loop in Vertex AI Pipelines (using dsl.ParallelFor) unrolls the loop into independent parallel tasks, one per hyperparameter combination, which is exactly the fan-out pattern needed for 12 trials. Setting the retry policy to 0 ensures that when any trial fails, the pipeline does not silently retry and mask the failure, so it fails fast as required.
✗
Define a sequential for-loop in the pipeline function that calls the training component 12 times in order.
Why it's wrong here
A sequential for-loop in the pipeline function creates a linear chain of component tasks, so each training run waits for the previous one to finish. That would not run the 12 combinations in parallel and would make the pipeline much slower. ParallelFor is required to fan out the iterations into independent, concurrently scheduled tasks.

Go deeper

PMLE Automating and Orchestrating ML Pipelines Practice Question

Why each option matters

Option-by-option breakdown

Related to this question