mediumMultiple Choice
Cloud Composer DAG Retry for Transient Failures — Dataflow, Quota Limits
Your team manages a production ML pipeline on Google Cloud that trains a fraud detection model every 6 hours using new transaction data. The pipeline steps are: (1) Cloud Function triggered by new files in Cloud Storage to validate data, (2) Dataflow job for feature engineering, (3) Vertex AI CustomJob for training, (4) Cloud Function to deploy the model to a Vertex AI endpoint after evaluation. You notice that the pipeline sometimes fails during the Dataflow job step with an error: 'Workflow failed. Causes: The job encountered a system error. Please try again later.' The error occurs sporadically, and retrying the pipeline manually usually succeeds. The team needs a reliable automated solution. What should you do?
Quick Answer
The answer is to orchestrate the pipeline using Cloud Composer with retry policies on the Dataflow operator. This is correct because Cloud Composer, built on Apache Airflow, allows you to define a Cloud Composer DAG retry for transient failures by setting parameters like `retries` and `retry_delay` directly on the Dataflow operator, which automatically re-launches the job when it encounters sporadic system errors like the one described. On the Google Professional Machine Learning Engineer exam, this scenario tests your understanding of production ML pipeline reliability and the distinction between transient versus permanent failures—a common trap is to over-engineer a solution with custom error handling or manual triggers when a simple retry policy suffices. Remember that for quota limit or system errors in Dataflow, the fix is often a retry, not a redesign. Memory tip: think "Composer retries compose the solution for transient Dataflow sighs."
⚠ Common exam trap
Many candidates confuse scaling solutions (Option C) with fault-tolerance mechanisms, or they choose a generic queuing service (Option B) instead of a dedicated orchestrator with built-in retry policies for pipeline steps.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
Orchestrate the pipeline using Cloud Composer with retry policies on the Dataflow operator.
Cloud Composer (Apache Airflow) provides native retry policies on its Dataflow operators, enabling automatic retries of the Dataflow job when it fails due to transient system errors. This addresses the sporadic failure pattern without manual intervention, ensuring the pipeline runs reliably every 6 hours.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✗
Schedule the pipeline to run less frequently to reduce load on the Dataflow service.
Why it's wrong here
Reducing frequency leaves the sporadic Dataflow system error unhandled; the pipeline still fails whenever it recurs, just less often. Retry logic, not scheduling cadence, addresses transient service faults. Lower frequency suits cost reduction or genuinely load-bound jobs, not intermittent infrastructure errors requiring automatic recovery.
- ✗
Use Cloud Tasks to queue the Dataflow job and retry on failure.
Why it's wrong here
Cloud Tasks queues HTTP callbacks, not Dataflow job launches; it cannot detect a failed Dataflow step or resume the remaining pipeline stages. Cloud Tasks suits asynchronous task dispatch to HTTP endpoints. Here the failure occurs inside the workflow, requiring step-level retry orchestration.
- ✗
Increase the number of Dataflow workers and use flexRS to handle transient errors.
Why it's wrong here
Adding workers and FlexRS addresses cost and capacity for batch processing, not the transient system error itself; the job can still fail with no automatic retry. FlexRS suits flexible, non-urgent batch workloads. The stem needs automated retry of the failed step, which worker scaling does not provide.
- ✓
Orchestrate the pipeline using Cloud Composer with retry policies on the Dataflow operator.
Why this is correct
Cloud Composer's Dataflow operator supports configurable retry policies, so transient system errors during the Dataflow step are retried automatically without manual intervention. This directly satisfies the reliability constraint: sporadic failures that succeed on retry become self-healing, while dependency sequencing across all four pipeline steps is preserved.
Visual reference
Go deeper
Related to this question
About these practice questions
Courseiva writes every PMLE question from scratch — 775 in total, each with an explanation and a wrong-answer breakdown. None are copied from real exams or dumps. Learn why practice questions differ from exam dumps →
Same concept, more angles
1 more way this is tested on PMLE
These questions test the same concept from different angles. Work through them to make sure you can recognise it however the exam phrases it.
Variation 1. A team is using Cloud Composer to orchestrate ML workflows. They have a DAG that triggers a Vertex AI Training job, then a prediction deployment. The deployment step occasionally fails due to quota limits. What is the best way to handle this?
medium- A.Increase the quota manually
- B.Use Vertex AI Pipelines instead of Cloud Composer
- C.Create a custom sensor to wait for quota to be available
- D.Catch the exception in the DAG and send an alert
- ✓ E.Implement exponential backoff retry in the DAG task
Why E: Cloud Composer (Apache Airflow) provides built-in retry mechanisms via task parameters like `retries` and `retry_delay`. Implementing exponential backoff in the DAG task is the best practice for handling transient quota errors, as it automatically retries the deployment step with increasing delays, reducing load on the quota system and increasing the chance of success without manual intervention. This approach aligns with Airflow's native error-handling capabilities and avoids unnecessary complexity or resource waste.
JA
Written by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
This PMLE practice question is part of Courseiva's free Google Cloud certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the PMLE exam.