A data engineer wants to orchestrate a complex workflow that includes running a Vertex AI pipeline, then a BigQuery job, and finally a Dataflow pipeline. The workflow must handle dependencies, retries, and monitoring. Which Google Cloud service is most suitable for this orchestration?
Cloud Composer is managed Apache Airflow, whose DAGs natively orchestrate heterogeneous tasks across Vertex AI, BigQuery and Dataflow with dependency handling, retries and monitoring. This satisfies the stem's requirement for cross-service workflow orchestration with dependencies and retries.
Why this answer
Cloud Composer (based on Apache Airflow) is the most suitable service for orchestrating a complex workflow with dependencies, retries, and monitoring across Vertex AI, BigQuery, and Dataflow. It provides a managed Airflow environment that natively supports DAG-based orchestration, built-in retry logic, and integration with Google Cloud services via operators like VertexAIPipelineOperator, BigQueryOperator, and DataflowTemplatedJobStartOperator.
Exam trap
A common misconception is that Workflows is sufficient for complex ML orchestration, but it lacks the built-in operator integrations and retry semantics that Cloud Composer provides for multi-service pipelines.
How to eliminate wrong answers
Option A is wrong because Cloud Tasks is a distributed task queue for executing discrete, short-lived tasks with HTTP endpoints, not for orchestrating multi-step workflows with complex dependencies and retries across different services. Option C is wrong because Cloud Scheduler is a cron-based job scheduler that triggers single events at specified times, lacking the ability to manage dependencies between multiple pipeline stages or handle retries. Option D is wrong because Workflows is a low-code orchestration service for sequential or parallel steps, but it does not natively support the rich operator ecosystem, retry policies, or monitoring capabilities that Cloud Composer provides for ML pipelines involving Vertex AI, BigQuery, and Dataflow.