Data Pipeline Orchestration

You are orchestrating a pipeline with Workflows that invokes several Cloud Functions. You need to pass the output of one function as an argument to the next. What is the correct syntax?

In a Cloud Composer DAG, you need to trigger a BigQuery job that runs as a service account other than the one assigned to the GKE nodes. How should you implement this?

When configuring a Workflows execution, you want to ensure the workflow can be triggered by an HTTP POST request. Which feature allows this?

Your organization uses Workflows to orchestrate multi-step API calls. You need to handle transient failures in a specific HTTP request by retrying it up to 5 times with exponential backoff. Which block should you use?

A Cloud Composer DAG is failing because the Airflow database is locking up during high-concurrency periods. What is the most effective way to address this?

Which Cloud Composer metric should you monitor to identify if your Airflow scheduler is struggling to parse your DAGs in a timely manner?

You are deploying a Cloud Composer 2 environment and need to ensure that the DAG files are automatically synchronized to all workers. Where should you store these files?

Which component of Cloud Composer acts as the primary interface for managing and triggering DAGs manually?

When using Cloud Scheduler to trigger a Workflows execution, what is the best practice for authenticating the scheduler to the Workflows API?

You need to run a Workflows execution based on a schedule. Which Google Cloud service is designed to manage this triggering?

Your team needs to audit all modifications made to DAG files in the GCS bucket used by Cloud Composer. What is the most efficient way to achieve this?

You are troubleshooting a stuck task in Cloud Composer. You want to see the stdout/stderr logs for that specific task attempt. Where can you find them?

In Cloud Composer, what is the default behavior if a DAG execution takes longer than the 'dagrun_timeout' setting?

Which property in a Workflows YAML definition is required to define the entry point of the workflow?

Which command-line tool do you primarily use to deploy and manage Workflows?

You are using Airflow with a Cloud Composer environment. You want to ensure that a specific task runs on a specific day of the week. How do you configure the schedule?

You want to dynamically generate tasks in an Airflow DAG based on the number of files in a GCS bucket. What is the standard way to do this?

A Cloud Composer environment is experiencing high latency in the Airflow UI. What is a common cause and mitigation for this?

Which TWO of the following are valid ways to trigger a Cloud Composer DAG?

What is the primary function of the Airflow 'Executor' in Cloud Composer?

Which TWO of the following are best practices for managing Cloud Composer DAG dependencies?

Which THREE of the following are key benefits of using Cloud Composer for data orchestration?

Which THREE of the following are valid Airflow connection types in Cloud Composer?

Which TWO of the following strategies help reduce Cloud Composer costs?

Which THREE of the following are common steps in a typical data pipeline orchestration pattern?

Which TWO of the following are true about Workflows' serverless nature?

Which THREE of the following are valid methods for troubleshooting failed tasks in Cloud Composer?

Your team runs a Cloud Composer 2 environment with several DAGs that process data in Cloud Storage. You need to design a DAG that reliably handles transient failures when calling the Cloud Storage API. The DAG should retry failed tasks automatically without manual intervention, but you also want to avoid overwhelming the API during an outage. Which combination of Airflow features should you implement to achieve this?

You are using Cloud Composer to orchestrate a data pipeline. A critical task in your DAG is failing intermittently due to a known transient error in an external API. You want to automatically retry the task up to 5 times with a 2-minute delay between retries. Which Airflow feature should you configure to achieve this?

You are orchestrating a data pipeline in Cloud Composer 2. A task in your DAG downloads a large dataset from Cloud Storage and can take up to 45 minutes to complete, but it occasionally fails and needs to be retried. You want the task to automatically retry up to 3 times with a 5-minute delay between attempts. Which Airflow task parameter should you configure to achieve this?

Your team uses Cloud Composer (Airflow) to run a daily ETL DAG. A task occasionally fails due to transient network errors when calling an external API. You want the task to retry automatically up to 3 times with a 5-minute delay between attempts, and you want the DAG to continue running other tasks even if this task ultimately fails. Which combination of Airflow task parameters should you configure?

You are designing a Cloud Workflows pipeline that calls a series of Cloud Functions. The output of the first function must be passed as an argument to the second function, and the output of the second to the third. You need to define the workflow in YAML. Which Workflows syntax correctly assigns the result of a function call to a variable and then uses that variable in a subsequent call?

Your team runs a Cloud Composer 2 environment where several DAGs import a shared internal Python library. After a library update, tasks begin failing with ModuleNotFoundError across all workers. You need the library to be installed automatically on every worker and scheduler node, including after autoscaling events. What should you do?

You are using Workflows to orchestrate a series of steps that call external HTTP APIs. One API occasionally returns a 503 error, and you want the workflow to retry that specific step with exponential backoff, but not retry on 404 errors. How should you configure the step?

You are using Cloud Scheduler to trigger a Cloud Workflows execution every hour. You need to ensure that the scheduler authenticates to the Workflows API securely. Which two of the following are valid methods for authenticating Cloud Scheduler to Workflows? (Choose two.)

You are using Cloud Composer to orchestrate a daily ETL pipeline. A task that normally completes in 10 minutes occasionally runs for over 2 hours due to unexpected data volume. You want to prevent this task from indefinitely blocking the workflow and ensure it fails after 30 minutes so downstream tasks can be retried or alerted. What should you configure on the task?

Your team runs a Cloud Composer 2 environment that processes financial data. A critical DAG must run exactly once per day at 02:00 UTC, and the Airflow scheduler must not backfill missed runs from the previous week when the DAG is unpaused. You need to configure the DAG so that only future scheduled runs execute. What should you set in the DAG definition?

You are designing a Cloud Composer 2 DAG that must process files as they arrive in a Cloud Storage bucket. You want to minimize latency and avoid unnecessary worker resource consumption. Which TWO approaches should you use? (Choose two.)

You are building a Workflows definition that calls a Cloud Run service and then passes the returned JSON payload to a Cloud Function. The Cloud Run response body is stored in a variable named result. Which syntax correctly passes that value as an argument in the subsequent call step?

You have a Cloud Composer 2 DAG that uses a PythonOperator to run a script that connects to a Cloud SQL database. The script needs a password. You want to avoid hardcoding the password in the DAG or in environment variables. What should you do?

You are using Workflows to orchestrate a data pipeline that includes a step calling a Cloud Function. The Cloud Function returns a JSON payload with a field named 'result'. You need to pass this 'result' value to a subsequent step as an argument. How should you reference the output of the Cloud Function step in the Workflows YAML?

Your Cloud Composer DAG includes a task that runs a long-running data export, and you have observed that the task sometimes exceeds the default Airflow worker pod timeout, causing the task to be killed and retried. You want to increase the allowed execution time for this task without affecting other tasks. What should you do?

A data engineer needs a lightweight, serverless orchestrator on Google Cloud to run a sequence of HTTP calls to internal microservices, retry the calls a few times on failure, and expose the whole sequence as a single REST endpoint. The workflow must not require provisioning any cluster or VM. Which Google Cloud service should be used?

You are using Cloud Scheduler to trigger a Workflows execution every hour. The Workflows definition expects an argument 'projectId' to be passed at runtime. You need to ensure the argument is provided correctly. What should you include in the Cloud Scheduler job configuration?

You are designing a Cloud Composer 2 DAG that must run a task only after a daily file has arrived in a Cloud Storage bucket. You want the DAG to avoid failing prematurely and to avoid consuming a worker slot while waiting. Which two approaches should you use? (Choose two.)

Your Workflows definition calls three Cloud Run services in sequence. The second service occasionally returns HTTP 503 during cold starts, and you want Workflows to retry those specific failures with exponential backoff while treating HTTP 400 responses as permanent failures. How should you configure this?

You maintain a Cloud Composer 2 DAG that processes files as they land in Cloud Storage. Instead of polling, you want the DAG to start processing within seconds of a new object being created, and you want the trigger to be reliable even if the Composer environment is briefly restarting. Which approach should you use?

A Cloud Composer 2 task runs a Python script that performs a long-running data transformation and regularly exceeds the default worker pod timeout, causing tasks to be killed and marked as failed. The transformation cannot be shortened and must complete within a single task. What should you do to allow the task to run to completion?