GCP-ADP Data Pipeline Orchestration Practice Test 3 — 15 Questions

Question 1

You are migrating a legacy Airflow environment to Cloud Composer. Your DAGs use custom Python operators that depend on a local Python package stored in a `requirements.txt` file. You need to ensure that the package is available to all Airflow workers and the scheduler. What is the recommended approach in Cloud Composer 2?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Upload the package to a Cloud Storage bucket and reference it in the `requirements.txt` file using a `gs://` path.. Cloud Composer 2 allows you to install Python packages from a `requirements.txt` file, including packages stored in Cloud Storage. This method ensures the package is installed on all Airflow components and persists across environment updates.

Answer

SSH into each worker node and run `pip install` manually.

Answer

Use the `PyPI` repository to host the package and add it to requirements.txt by name.

Answer

Place the package in the `dags/` folder and import it directly in your DAG.

Question 2

You are authoring a Workflows definition that must call a Cloud Run service and then branch on the returned HTTP status. Some calls return 404 because the target resource has not been provisioned yet, and you want the workflow to retry only that condition while letting 500 errors fail fast. How should you implement this in the Workflows YAML?

Accepted Answer

Wrap the call in a `try`/`except` block and use `retry` with a `predicate` that inspects `e.code` for the value 404.. Workflows retry behaviour is controlled per step through a `retry` policy whose `predicate` receives the raised error, and the `try`/`except` construct surfaces the HTTP status as `e.code`. Filtering on 404 lets the workflow keep polling until the resource exists, while any other status, including 500, escapes the except block and stops execution, matching the fail-fast requirement exactly.

Answer

Configure the Cloud Run service with a `--retry` flag so the platform automatically retries 404 responses before returning to Workflows.

Answer

Set a global `retry` block at the top of the workflow with `max_retries: 5` and rely on the default backoff to cover all transient failures.

Answer

Use a `switch` step that maps the string value of the response body to a retry subworkflow invoked with `call`.

Question 3

A Cloud Composer 2 environment runs a DAG whose final task must invoke a Workflows execution and wait for its result before the DAG marks success. The Workflows execution can take up to 40 minutes. Which approach correctly waits for completion without blocking a worker slot indefinitely?

Accepted Answer

Call `WorkflowsCreateExecutionOperator` to start the execution, then use a `WorkflowsExecutionSensor` in `reschedule` mode to poll for completion.. Long-running external work should be started with an operator that returns an execution identifier immediately, then monitored with a sensor that can release its worker slot between polling intervals. The Workflows create operator plus the Workflows execution sensor in reschedule mode accomplishes both, keeping the DAG's final task accurate while respecting Composer's worker capacity constraints during the 40-minute window.

Answer

Use the `WorkflowsCreateExecutionOperator` and set the Airflow task's `execution_timeout` to 45 minutes so the task polls until the execution finishes.

Answer

Configure the DAG with `max_active_runs=1` and rely on the Workflows execution's own retry policy to signal completion back through Pub/Sub.

Answer

Use a `PythonOperator` that calls the Workflows REST API in a loop with `time.sleep(60)` until the execution state is `SUCCEEDED`.

Question 4

You are orchestrating a pipeline with Workflows that invokes several Cloud Functions. You need to pass the output of one function as an argument to the next. What is the correct syntax?

Accepted Answer

Assign the call result to a variable and reference it in the next step.. Workflows uses a YAML/JSON syntax where each step's return value can be assigned to a variable, and subsequent steps reference that variable using ${variableName} syntax. This is the idiomatic way to chain Cloud Functions and pass outputs forward. For example: 'call: http.post' with 'result: fn1Result', then the next step uses '${fn1Result.body}'.

Answer

Use global environment variables to store intermediate state.

Answer

Combine all function calls into a single synchronous block.

Answer

Write the output to a temporary GCS file and read it back.

Question 5

In a Cloud Composer DAG, you need to trigger a BigQuery job that runs as a service account other than the one assigned to the GKE nodes. How should you implement this?

Accepted Answer

Use the 'impersonation_chain' parameter in the BigQueryInsertJobOperator.. The 'impersonation_chain' parameter is the native Airflow mechanism for a task to assume a different service account identity at runtime. When set on BigQueryInsertJobOperator, the operator calls the IAM Credentials API to generate short-lived access tokens for the target service account, so the BigQuery job executes under that SA's permissions rather than the Composer worker SA. This preserves least privilege without storing any long-lived secrets.

Answer

Hardcode the credentials in a JSON file inside the DAG folder.

Answer

Change the worker node's identity in the Cloud Console.

Answer

Configure a connection with the 'Service Account Key' type in Airflow UI.

Question 6

When configuring a Workflows execution, you want to ensure the workflow can be triggered by an HTTP POST request. Which feature allows this?

Accepted Answer

Executing the workflow via the REST API.. Workflows executions can be triggered by an HTTP POST request using the Workflows REST API (executions.create endpoint). This allows external systems to start a workflow by sending an authenticated POST to the Workflows API with the workflow ID and input arguments.

Answer

Configuring a Cloud Scheduler job with an HTTP target.

Answer

Enabling the 'HTTP Gateway' on the workflow itself.

Answer

Creating a Service Account with 'Public Access'.

Question 7

Your organization uses Workflows to orchestrate multi-step API calls. You need to handle transient failures in a specific HTTP request by retrying it up to 5 times with exponential backoff. Which block should you use?

Accepted Answer

A 'try' block with a 'retry' policy.. Workflows provides a try/retry block where you can specify a retry policy with max_retries and a backoff (initial delay, multiplier, max delay), which is exactly the mechanism for retrying a transient HTTP failure up to 5 times with exponential backoff. The retry policy is attached to the try block and applies to the steps inside it.

Answer

A 'loop' block with a counter increment.

Answer

An 'if' statement checking for 500 status codes.

Answer

A global 'exception' handler at the end of the workflow.

Question 8

A Cloud Composer DAG is failing because the Airflow database is locking up during high-concurrency periods. What is the most effective way to address this?

Accepted Answer

Scale up the Cloud Composer environment size to increase database resources.. Cloud Composer's Airflow metadata database (backed by Cloud SQL) is the bottleneck during high-concurrency DAG runs because every task instance, DAG run, and log entry writes to it. Scaling up the Composer environment size increases the Cloud SQL tier (CPU, memory, IOPS), which directly relieves database lock contention and connection saturation. This is the recommended remediation when the scheduler/DB cannot keep up with task state updates.

Answer

Delete all past DAG run logs.

Answer

Increase the number of DAGs in the repository.

Answer

Set the 'parallelism' parameter to a very high value.

Question 9

Which Cloud Composer metric should you monitor to identify if your Airflow scheduler is struggling to parse your DAGs in a timely manner?

Accepted Answer

dag_processing_total_parse_time. The dag_processing_total_parse_time metric measures the cumulative time Airflow spends parsing DAG files, which directly indicates whether the scheduler is struggling to keep up with DAG parsing. A rising or consistently high value signals that DAGs are too complex, too numerous, or inefficiently written, delaying task scheduling. This is the canonical Cloud Composer metric for diagnosing scheduler parse bottlenecks.

Answer

airflow_task_execution_time

Answer

dag_run_success_count

Answer

scheduler_heartbeat

Question 10

You are deploying a Cloud Composer 2 environment and need to ensure that the DAG files are automatically synchronized to all workers. Where should you store these files?

Accepted Answer

In the 'dags/' folder of the environment's bucket.. In Cloud Composer 2, the environment's Cloud Storage bucket contains a 'dags/' folder that is automatically synchronized to all Airflow worker nodes (and the scheduler). Uploading DAG files to gs://<environment-bucket>/dags/ is the supported, automatic way to distribute DAGs, and Composer's built-in sync process pushes changes to every component.

Answer

Inside a Docker volume configured in the environment's YAML manifest.

Answer

In the local /home/airflow/gcs/dags directory of each worker node.

Answer

In a custom persistent disk attached to the Airflow scheduler.

Question 11

Which component of Cloud Composer acts as the primary interface for managing and triggering DAGs manually?

Accepted Answer

Airflow Web Interface. The Airflow Web Interface (the Airflow UI) is the primary interface in Cloud Composer for viewing, triggering, and managing DAGs manually. It provides DAG views, task logs, graph view, and manual trigger buttons.

Answer

Cloud Storage Browser

Answer

Cloud Shell

Answer

BigQuery Console

Question 12

When using Cloud Scheduler to trigger a Workflows execution, what is the best practice for authenticating the scheduler to the Workflows API?

Accepted Answer

Use an OIDC token with a service account.. The best practice for Cloud Scheduler to authenticate to the Workflows API is to configure the job with an OIDC token tied to a service account that has permission to invoke the workflow. Cloud Scheduler supports OIDC authentication for HTTP targets, and the service account's identity is verified by Workflows via IAM. This avoids static credentials and follows least-privilege.

Answer

Use an API Key generated in the console.

Answer

Hardcode credentials in the scheduler job.

Answer

Use 'Public' access for the workflow.

Question 13

You need to run a Workflows execution based on a schedule. Which Google Cloud service is designed to manage this triggering?

Accepted Answer

Cloud Scheduler. Cloud Scheduler is Google Cloud's fully managed cron service, purpose-built to invoke HTTP endpoints, Pub/Sub topics, or App Engine routes on a defined schedule. Workflows exposes an HTTP trigger endpoint, so a Cloud Scheduler job can POST to that endpoint on a cron expression to start executions. This is the canonical, documented pattern for scheduled Workflows runs.

Answer

Eventarc

Answer

Cloud Tasks

Answer

Pub/Sub

Question 14

Your team needs to audit all modifications made to DAG files in the GCS bucket used by Cloud Composer. What is the most efficient way to achieve this?

Accepted Answer

Enable Data Access audit logs on the GCS bucket.. Enabling Data Access audit logs on the GCS bucket is the most efficient way to audit all modifications made to DAG files because it automatically logs all object-level operations, including who modified what and when. This provides a comprehensive audit trail without requiring custom code or manual monitoring. Data Access audit logs capture both read and write operations, and can be exported to Cloud Logging for analysis and retention.

Answer

Enable 'Version History' in the Airflow UI.

Answer

Use Cloud Monitoring to alert on 'gcs_object_write' events.

Answer

Write a custom DAG to scan the bucket every minute.

Question 15

You are troubleshooting a stuck task in Cloud Composer. You want to see the stdout/stderr logs for that specific task attempt. Where can you find them?

Accepted Answer

In the 'logs/' folder in the environment's GCS bucket.. In Cloud Composer, task logs (stdout/stderr) for each task attempt are stored in the environment's GCS bucket under the logs/ folder, organized by DAG and task. This is the canonical location for retrieving raw task output, especially for tasks that have completed or whose pods have been deleted.

Answer

In the Airflow metadata database directly.

Answer

In the Cloud Composer environment console logs.

Answer

Via the 'kubectl' logs command on the worker pod.