Your team manages a production ML pipeline on Google Cloud that trains a fraud detection model every 6 hours using new transaction data. The pipeline steps are: (1) Cloud Function triggered by new files in Cloud Storage to validate data, (2) Dataflow job for feature engineering, (3) Vertex AI CustomJob for training, (4) Cloud Function to deploy the model to a Vertex AI endpoint after evaluation. You notice that the pipeline sometimes fails during the Dataflow job step with an error: 'Workflow failed. Causes: The job encountered a system error. Please try again later.' The error occurs sporadically, and retrying the pipeline manually usually succeeds. The team needs a reliable automated solution. What should you do?
Cloud Composer (Airflow) can manage the pipeline DAG with automatic retries and dependencies.
Why this answer
Cloud Composer (Apache Airflow) provides native retry policies on its Dataflow operators, enabling automatic retries of the Dataflow job when it fails due to transient system errors. This addresses the sporadic failure pattern without manual intervention, ensuring the pipeline runs reliably every 6 hours.
Exam trap
The trap here is that candidates confuse scaling solutions (Option C) with fault-tolerance mechanisms, or they choose a generic queuing service (Option B) instead of a dedicated orchestrator with built-in retry policies for pipeline steps.
How to eliminate wrong answers
Option A is wrong because reducing pipeline frequency does not resolve transient system errors in Dataflow; it only delays processing and may cause data staleness. Option B is wrong because Cloud Tasks is a generic task queue that lacks native integration with Dataflow job lifecycle management and retry logic for pipeline-specific errors. Option C is wrong because increasing workers and using FlexRS improves resource availability but does not handle transient system errors that are unrelated to worker count or preemptibility; FlexRS is for cost savings on preemptible VMs, not for retry logic.