GCP-ADP Data Pipeline Orchestration — 30 Questions

Question 1

You need to run a task in Cloud Composer only if the previous task failed. Which Trigger Rule should you use?

Accepted Answer

one_failed. The one_failed trigger rule fires a task when at least one upstream task has failed, which is exactly the requirement to run a task only if a previous task failed. This is commonly used for error-handling or cleanup tasks that should execute on failure. It is the correct trigger rule for failure-conditional execution.

Answer

none_failed

Answer

always

Answer

all_success

Question 2

You have a Workflow that must wait for a specific time duration before proceeding. Which step type is most appropriate?

Accepted Answer

sys.sleep. The sys.sleep step type is most appropriate for a Workflow that must wait for a specific time duration before proceeding. In Google Cloud Workflows, sys.sleep is a built-in function that pauses the execution for a specified duration, allowing the workflow to wait before continuing. It is designed exactly for this purpose and is simple to use.

Answer

pause_execution

Answer

wait_for_condition

Answer

delay_step

Question 3

You are migrating a legacy Airflow environment to Cloud Composer. You need to ensure that local Python packages used by custom operators are installed correctly. What is the recommended approach?

Accepted Answer

Upload the requirements.txt file to the environment bucket.. The recommended approach for installing local Python packages in Cloud Composer is to upload a requirements.txt file to the environment's bucket (specifically in the dags/ folder or configured path). Composer automatically installs these dependencies on the Airflow workers and schedulers during environment creation or update.

Answer

Create a custom Docker image for the Airflow environment.

Answer

Manually SSH into worker nodes and run 'pip install'.

Answer

Pre-install packages in the DAG file using 'os.system'.

Question 4

A task in your Airflow DAG involves a long-running process that might exceed the default Airflow worker pod timeout. How should you handle this?

Accepted Answer

Set the 'execution_timeout' parameter on the task instance.. Airflow's execution_timeout parameter defines the maximum runtime allowed for a task instance before the scheduler marks it as failed and terminates it. Setting this parameter on the specific long-running task overrides the default and prevents the worker from killing the task prematurely. This is the correct, task-scoped mechanism for handling processes that legitimately need more time than the default allows.

Answer

Move the task to a custom node pool in GKE.

Answer

Increase the 'default_dag_run_timeout'.

Answer

Decrease the number of parallel tasks.

Question 5

You need to pass a JSON object into a Workflows execution. How should you format the argument in the 'gcloud workflows executions' command?

Accepted Answer

Use the --data flag with a JSON string.. The gcloud workflows executions command accepts runtime arguments via the --data flag, which takes a JSON string representing the input to the workflow. This is the documented, standard way to pass structured data into a Workflows execution. The workflow then accesses this data through its input argument.

Answer

Pipe the JSON into the stdin of the command.

Answer

Use the --env-vars flag.

Answer

Upload the JSON to a GCS bucket and point to it.

Question 6

In Cloud Composer, you want to prevent a task from retrying if it fails due to a specific error (e.g., File Not Found). How can you configure this?

Accepted Answer

Raise 'AirflowSkipException' in your task code.. Airflow does not support per-error-type retry suppression natively, so you must raise an 'AirflowSkipException' or similar in custom code to avoid retries.

Answer

Set 'retries: 0' in the task operator.

Answer

Use the 'on_failure_callback' to cancel retries.

Answer

Configure 'retry_delay' to be infinite.

Question 7

Which THREE of the following are components of a Workflows definition?

Accepted Answer

main. In a Workflows definition, the main component (A) is the required entrypoint that declares the workflow's top-level logic and orchestrates execution, making it a core part of the definition. The steps component (D) defines the individual units of work that the workflow executes, so it is also a fundamental part of the definition. The variables component (C) declares workflow-level variables that can be referenced throughout the workflow, which is likewise a standard part of a Workflows definition. By contrast, dag_schedule (B) is not a Workflows definition component but rather a scheduling construct used by other orchestration systems, and worker_config (E) is not part of a Workflows definition but relates to runtime or worker configuration outside the definition itself.

Answer

dag_schedule

Answer

worker_config

Question 8

Which TWO of the following services can trigger Workflows executions?

Accepted Answer

Cloud Scheduler. Cloud Scheduler (D) is correct because it can invoke Workflows executions directly via an HTTP target, using an OAuth token or OIDC service account to authenticate against the Workflows API, making it the standard way to run workflows on a cron schedule. Eventarc (E) is correct because it delivers events from Google Cloud sources (and custom sources) to Workflows as a destination, triggering executions in response to those events. Cloud Functions (A) can call the Workflows API, but it is not a native trigger service for Workflows in the same way; Cloud Logging (B) is a logging and routing service, not an execution trigger; and Cloud Build (C) is a CI/CD service that builds and deploys but does not natively trigger Workflows executions.

Answer

Cloud Functions

Answer

Cloud Logging

Answer

Cloud Build

Question 9

Which TWO of the following are common reasons a Workflows execution might stay in a 'PENDING' or 'FAILED' state?

Accepted Answer

Invalid YAML syntax. Option A (Invalid YAML syntax) is correct because a Workflows execution cannot be parsed or started if the workflow definition contains malformed YAML, leaving the execution stuck in a PENDING state or immediately transitioning to FAILED before any steps run. Option C (Insufficient IAM permissions) is correct because the Workflows service account must have the required IAM roles (e.g., roles/workflows.invoker, roles/logging.logWriter, or permissions to call target services); missing permissions cause the execution to fail when it attempts to invoke those resources. Option B (Airflow scheduler failure) is not applicable because Airflow is a separate orchestration product and is not part of the Workflows execution path. Option D (Database locks) is not a typical cause since Workflows does not rely on a user-managed database for execution state. Option E (Worker node exhaustion) is not relevant because Workflows is a fully managed, serverless service and does not expose or depend on worker nodes.

Answer

Airflow scheduler failure

Answer

Database locks

Answer

Worker node exhaustion

Question 10

Which THREE of the following Cloud Composer 2 features provide improved performance over Composer 1?

Accepted Answer

Unified environment management. Option A (Unified environment management) is correct because Composer 2 consolidates the control plane and worker infrastructure into a single managed environment, letting you manage the whole deployment through one interface rather than juggling separate GKE and Airflow components as in Composer 1. Option D (Configurable resource allocation) is correct because Composer 2 lets you tune CPU, memory, and storage for schedulers, workers, and web servers independently, so workloads get right-sized resources for better throughput. Option E (Environment auto-scaling) is correct because Composer 2 supports autoscaling of worker replicas based on queue demand, improving performance under variable loads compared with Composer 1's fixed worker pools. Option B is incorrect because Composer 2 reduces, not requires, manual GKE cluster management. Option C is incorrect because Composer 2 uses consumption-based pricing rather than a fixed pricing model.

Answer

Requires manual GKE cluster management

Answer

Fixed pricing model

Question 11

You are building a data pipeline using Cloud Workflows that calls a series of Cloud Functions. The output of one function must be passed as an argument to the next function. How should you structure the workflow to achieve this?

Accepted Answer

Use the `result` field of the first function call to store the output, then reference it in the next step's `args`.. Cloud Workflows allows you to capture the output of a step by assigning it to a variable using the `result` field. That variable can then be referenced in the `args` of a subsequent step. This enables seamless data passing between steps without external storage.

Answer

Use the `call` keyword to invoke the next function with the previous function's output as a parameter.

Answer

Define a global variable that is updated by each function and read by the next.

Answer

Write the output to a Cloud Storage bucket and have the next function read from it.

Question 12

Your team runs a Cloud Composer 2 environment. A DAG must run only after a new object lands in a Cloud Storage bucket, and the DAG should not poll the bucket. You want the least operational overhead. What should you do?

Accepted Answer

Create a Cloud Storage trigger on the bucket that publishes to a Pub/Sub topic, then use a Cloud Composer sensor that subscribes to that topic to unblock the DAG.. Cloud Storage object change notifications can be sent to Pub/Sub, and Cloud Composer provides a Pub/Sub sensor that waits for messages. This avoids polling, requires no custom code, and uses managed services. The other approaches either poll, require custom operators, or add an extra Cloud Function to maintain, making them less efficient or higher overhead.

Answer

Write a custom Airflow operator that uses the Cloud Storage client library to poll the bucket in a loop inside the task and returns when a new object is found.

Answer

Create a Cloud Function triggered by Cloud Storage finalize events that calls the Cloud Composer Airflow REST API to trigger the DAG.

Answer

Configure a Cloud Scheduler job that runs every minute and calls the Cloud Storage JSON API to list the bucket, then triggers the DAG when a new object appears.

Question 13

Your team uses Workflows to orchestrate a daily ETL job that calls a Cloud Run service. The job occasionally takes longer than expected, and you need to ensure that the workflow automatically retries a failed HTTP call up to 3 times with a 10-second delay between attempts. What should you configure in the Workflows YAML definition?

Accepted Answer

Use the retry field at the step level with a policy that specifies max_retries: 3 and a backoff delay of 10 seconds.. Workflows provides a step-level retry mechanism that can be configured with max_retries and backoff parameters. This allows the workflow to automatically retry a failed step, such as an HTTP call, without requiring custom error handling. The retry policy is defined directly in the step definition, making it the correct way to achieve the desired behavior.

Answer

Configure the Cloud Run service to automatically retry failed requests by setting a retry policy in the service's YAML configuration.

Answer

Add a retry policy to the call step using the retry field with max_retries: 3 and initial_delay: 10s.

Answer

Wrap the HTTP call in a try/except block and define a retry policy with predicate and max_retries in the except clause.

Question 14

You are designing a data pipeline using Cloud Composer 2. You need to ensure that your DAG can be triggered automatically based on the arrival of new files in a Cloud Storage bucket. Which TWO of the following methods can be used to trigger the DAG? (Choose two.)

Accepted Answer

Use a Cloud Storage trigger in Cloud Functions to call the Airflow REST API.. The two valid methods are using a Cloud Function with a Cloud Storage trigger to call the Airflow REST API, and using Pub/Sub notifications with an Airflow sensor. Both are event-driven and integrate with Cloud Composer. The other options either rely on manual commands, inefficient polling, or unsupported direct integration.

Answer

Enable the Cloud Storage 'Object Change Notification' feature to directly invoke the Airflow web server.

Answer

Use the gcloud composer environments run command to trigger the DAG when a file is uploaded.

Answer

Set up a Cloud Scheduler job that polls the bucket every minute and triggers the DAG if a new file exists.

Question 15

You are designing a Cloud Workflows definition that must call an external HTTP API. The API requires an OAuth 2.0 access token in the Authorization header, and the token is stored in Secret Manager. You want the workflow to retrieve the token securely at runtime without hardcoding credentials. Which step should you include in the workflow?

Accepted Answer

Use the `googleapis.secretmanager.v1.projects.secrets.versions.access` connector to fetch the secret and assign it to a variable.. Using the Secret Manager connector lets the workflow retrieve the token at runtime, keeping it out of the YAML and respecting IAM controls. This is the only secure, supported method among the choices; the others either expose credentials or rely on non-existent features.

Answer

Embed the token as a literal string in the workflow YAML and reference it in the Authorization header.

Answer

Use the `sys.get_env` function to read the token from an environment variable set on the Workflows execution environment.

Answer

Configure the workflow to authenticate as the default Compute Engine service account and rely on the metadata server to supply the token.

Question 16

You are creating a Workflows definition to orchestrate a series of microservices. You need to define the starting point of the workflow. Which field in the Workflows YAML is used to specify the entry point?

Accepted Answer

main. In Workflows, the main field is required and defines the entry point of the workflow. It contains the steps that are executed when the workflow runs. Other fields like entrypoint, start, and init are not valid in Workflows YAML and would cause errors. The main field is the correct way to specify the workflow's starting point.

Answer

start

Answer

entrypoint

Answer

init

Question 17

Your Cloud Composer 2 environment runs an Airflow DAG that starts a Dataflow streaming job and then needs to wait until that job reports a terminal state before running a downstream cleanup task. You want the DAG to pause without occupying a worker slot while the Dataflow job runs, and to resume as soon as the job finishes. What should you do?

Accepted Answer

Use the DataflowStartFlexTemplateOperator and chain it to a DataflowJobStatusSensor set to the desired terminal state.. Waiting for an external service without holding an Airflow worker slot is best done with a deferrable sensor. DataflowJobStatusSensor polls the job state and defers the task until the configured terminal state is reached, so downstream cleanup starts exactly when the job completes. A PythonOperator polling loop or a fixed delay sensor either consumes worker capacity or waits the wrong amount of time.

Answer

Use the DataflowStartFlexTemplateOperator followed by a separate DataflowStopJobOperator with a fixed delay via a TimeDeltaSensor.

Answer

Use a BashOperator that calls gcloud dataflow jobs show with the --wait flag inside the task.

Answer

Use a PythonOperator with a while loop that calls the Dataflow jobs.get API every 60 seconds and sleeps between checks.

Question 18

You are writing a Workflows YAML definition that must call a Cloud Function, extract a value from its JSON response, and pass that value as an argument to a subsequent BigQuery insert step. You need to reference the output of the first step in the second step. Which syntax should you use?

Accepted Answer

Reference the first step's result using the expression ${firstStep.response.body.field} in the second step's arguments.. Workflows exposes the result of a call step through the step's name, and HTTP responses are nested under response.body. To pass a value from a Cloud Function's JSON response into a later BigQuery step, you reference ${firstStep.response.body.field} in the arguments. Other property paths or nonexistent keywords do not resolve at runtime.

Answer

Reference the first step's result using the expression ${firstStep.body.field} in the second step's arguments.

Answer

Store the output in a variable using the assign keyword and reference the variable in the next step.

Answer

Use the result of the first step by referencing ${firstStep.response.body.field} in the arguments of the second step.

Question 19

You are creating a Cloud Workflows definition to orchestrate a data pipeline. You need to define the starting point of the workflow. Which top-level property must you include in the YAML definition to specify the first step to execute?

Accepted Answer

`main`. Cloud Workflows requires a top-level `main` block that contains a `steps` list. The first step in that list is the entry point. Other properties like `entrypoint`, `start`, or `init` are not part of the Workflows YAML schema and will cause deployment failures.

Answer

`start`

Answer

`entrypoint`

Answer

`init`

Question 20

You are using Workflows to orchestrate a data processing pipeline. One step invokes a Cloud Function that returns a JSON payload containing a processed record ID. You need to pass this record ID as an argument to a subsequent Cloud Function. How should you reference the output of the first function in the Workflows YAML definition?

Accepted Answer

Assign the output of the first function to a variable in the first step, then reference that variable in the second step.. To pass data between steps in Workflows, you must capture the output of a function call by assigning it to a variable. The 'assign' step stores the result, and subsequent steps can reference that variable using dot notation. This is the standard and recommended method for data passing within a workflow.

Answer

Store the output in a Cloud Storage bucket from the first function, then read it in the second function.

Answer

Use the 'output' keyword in the first step to define the return value, then reference it in the second step.

Answer

Use the result of the first function call directly in the args field of the next step, like ${function1.result.recordId}.