GCP-ADP Data Pipeline Orchestration Practice Test 5 — 15 Questions

Question 1

Your team runs a Cloud Composer 2 environment with several DAGs that process data in Cloud Storage. You need to design a DAG that reliably handles transient failures when calling the Cloud Storage API. The DAG should retry failed tasks automatically without manual intervention, but you also want to avoid overwhelming the API during an outage. Which combination of Airflow features should you implement to achieve this?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Set retries and retry_delay on the DAG's default_args, and use exponential backoff with max_retry_delay.. Setting retries and retry_delay in default_args enables automatic retries for failed tasks. Exponential backoff with max_retry_delay spaces out retries increasingly, reducing load on the Cloud Storage API during an outage. This combination is the recommended way to handle transient failures in Airflow while avoiding API overload, and it requires no custom code.

Answer

Use a single task with a long timeout and rely on the Cloud Storage client library's internal retry mechanism.

Answer

Implement a custom operator that catches exceptions and sleeps for a fixed interval before re-executing the API call.

Answer

Configure the DAG to run on a schedule with a high number of max_active_runs, and let failed tasks be skipped.

Question 2

You are using Cloud Composer to orchestrate a data pipeline. A critical task in your DAG is failing intermittently due to a known transient error in an external API. You want to automatically retry the task up to 5 times with a 2-minute delay between retries. Which Airflow feature should you configure to achieve this?

Accepted Answer

Set `retries=5` and `retry_delay=timedelta(minutes=2)` in the task's default arguments.. Airflow's built-in retry mechanism is configured via the `retries` and `retry_delay` parameters. These can be set at the DAG level (default_args) or per task. When a task fails, Airflow will automatically retry it after the specified delay, up to the maximum number of retries. This is the standard way to handle transient failures.

Answer

Configure a `SLA` (Service Level Agreement) with a timeout to force a retry.

Answer

Implement a custom operator that loops until success, with a sleep between attempts.

Answer

Use the `on_failure_callback` to trigger a new task run after a delay.

Question 3

You are orchestrating a data pipeline in Cloud Composer 2. A task in your DAG downloads a large dataset from Cloud Storage and can take up to 45 minutes to complete, but it occasionally fails and needs to be retried. You want the task to automatically retry up to 3 times with a 5-minute delay between attempts. Which Airflow task parameter should you configure to achieve this?

Accepted Answer

retries=3, retry_delay=timedelta(minutes=5). To configure automatic retries with a specific delay, you must set both the retries and retry_delay parameters on the task. The retries parameter controls how many times the task will be retried after failure, and retry_delay specifies the time to wait between attempts. Using retries=3 and retry_delay=timedelta(minutes=5) directly satisfies the requirement.

Answer

sla=timedelta(minutes=5), retries=3

Answer

execution_timeout=timedelta(minutes=45), retries=3

Answer

retry_exponential_backoff=True, max_retry_delay=timedelta(minutes=5)

Question 4

Your team uses Cloud Composer (Airflow) to run a daily ETL DAG. A task occasionally fails due to transient network errors when calling an external API. You want the task to retry automatically up to 3 times with a 5-minute delay between attempts, and you want the DAG to continue running other tasks even if this task ultimately fails. Which combination of Airflow task parameters should you configure?

Accepted Answer

Set retries=3, retry_delay=timedelta(minutes=5), and set the task's downstream tasks to use trigger_rule='all_done'.. To handle transient failures, configure retries and retry_delay on the task. To allow the DAG to continue despite a task failure, set the downstream tasks' trigger_rule to 'all_done', which triggers them regardless of upstream success or failure. This combination ensures retries occur and subsequent tasks are not blocked, meeting both requirements.

Answer

Set retries=3, retry_delay=timedelta(minutes=5), and set the task's trigger_rule='all_failed'.

Answer

Set retries=3, retry_delay=timedelta(minutes=5), and set the task's trigger_rule='all_success'.

Answer

Set retries=3, retry_delay=timedelta(minutes=5), and trigger_rule='all_done' on the task.

Question 5

You are designing a Cloud Workflows pipeline that calls a series of Cloud Functions. The output of the first function must be passed as an argument to the second function, and the output of the second to the third. You need to define the workflow in YAML. Which Workflows syntax correctly assigns the result of a function call to a variable and then uses that variable in a subsequent call?

Accepted Answer

- call: firstFunction
  result: firstResult
- call: secondFunction
  args:
    input: ${firstResult}. In Cloud Workflows, the 'result' keyword captures a call's return value into a variable. Subsequent calls can reference that variable within their 'args' block using ${}. This allows chaining outputs to inputs. The correct syntax uses 'result' to store the output and then 'args' with a named parameter to pass it along.

Answer

- call: firstFunction
  output: firstResult
- call: secondFunction
  args:
    input: ${firstResult}

Answer

- call: firstFunction
  result: firstResult
- call: secondFunction
  args: ${firstResult}

Answer

- call: firstFunction
  result: firstResult
- call: secondFunction
  input: ${firstResult}

Question 6

Your team runs a Cloud Composer 2 environment with several DAGs that process data in Cloud Storage. You need to design a DAG that reliably handles transient failures when calling the Cloud Storage API. The DAG should retry failed tasks automatically without manual intervention, but you also want to avoid overwhelming the API during an outage. Which combination of Airflow features should you implement to achieve this?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Install the library using a PyPI package specified in the environment's Python dependencies configuration.. Declaring the shared library as a PyPI package in the environment's Python dependencies ensures it is installed consistently on the scheduler, web server, and all workers, including those created by autoscaling. Manual installs, DAG-folder copies, and Cloud Storage zip references do not integrate with the managed environment's package lifecycle, so they fail under scaling and recreation events.

Answer

SSH into each worker pod and run pip install manually, then restart the Airflow scheduler.

Answer

Upload the library as a zip file to a Cloud Storage bucket and reference it in the DAG's default_args.

Answer

Add the library to the dags/ folder and import it with a relative path from each DAG file.

Question 7

You are using Workflows to orchestrate a series of steps that call external HTTP APIs. One API occasionally returns a 503 error, and you want the workflow to retry that specific step with exponential backoff, but not retry on 404 errors. How should you configure the step?

Accepted Answer

Set the step's retry predicate to check for a 503 status code and configure a retry policy with exponential backoff, and do not include 404 in the predicate.. Workflows steps support a retry policy with a predicate that can inspect the error or response. By predicating on 503 only, the step retries with exponential backoff on server errors and fails fast on 404. Manual loops or global retries do not provide the same granular control, and Cloud Scheduler is not designed for in-workflow retries.

Answer

Configure the entire workflow to retry on any error by setting a global retry policy, and rely on the API to eventually return success for 404s.

Answer

Wrap the API call in a try/except block and use a loop with a sleep timer that doubles each iteration, manually checking for 503 before retrying.

Answer

Use a Cloud Scheduler job to invoke the API step repeatedly with increasing intervals until it succeeds, then trigger the rest of the workflow.

Question 8

You are using Cloud Scheduler to trigger a Cloud Workflows execution every hour. You need to ensure that the scheduler authenticates to the Workflows API securely. Which two of the following are valid methods for authenticating Cloud Scheduler to Workflows? (Choose two.)

Accepted Answer

Use an OAuth token generated for a service account that has the Workflows Invoker role.. Cloud Scheduler can authenticate to Workflows using either OIDC or OAuth tokens generated for a service account. The service account must have the Workflows Invoker role. These methods leverage Google Cloud IAM and avoid managing static secrets, making them secure and recommended. API keys and basic auth are not supported for the Workflows API.

Answer

Use an API key associated with the Workflows API and include it in the request header.

Answer

Use an OAuth 2.0 client ID and secret stored in Cloud Scheduler's configuration.

Answer

Use a static username and password stored in Secret Manager and referenced by Cloud Scheduler.

Question 9

You are using Cloud Composer to orchestrate a daily ETL pipeline. A task that normally completes in 10 minutes occasionally runs for over 2 hours due to unexpected data volume. You want to prevent this task from indefinitely blocking the workflow and ensure it fails after 30 minutes so downstream tasks can be retried or alerted. What should you configure on the task?

Accepted Answer

Set execution_timeout to 30 minutes in the task definition.. In Cloud Composer, Airflow tasks support an execution_timeout parameter that defines the maximum runtime for a single task. When the timeout is reached, the task is terminated and marked as failed, enabling downstream tasks to proceed or alerts to fire. Configuring this on the specific task ensures that only that task is affected, preserving the rest of the DAG's execution.

Answer

Set the DAG's dagrun_timeout to 30 minutes.

Answer

Set the task's retries parameter to 0.

Answer

Set the scheduler's max_threads to a higher value.

Question 10

Your team runs a Cloud Composer 2 environment with several DAGs that process data in Cloud Storage. You need to design a DAG that reliably handles transient failures when calling the Cloud Storage API. The DAG should retry failed tasks automatically without manual intervention, but you also want to avoid overwhelming the API during an outage. Which combination of Airflow features should you implement to achieve this?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Set catchup=False in the DAG constructor and define schedule='0 2 * * *'.. The requirement is a daily 02:00 UTC run with no historical backfill. In Cloud Composer, catchup=False suppresses the creation of runs for intervals that elapsed before the DAG was enabled, while the cron schedule controls future timing. Other parameters such as max_active_runs or start_date adjustments do not disable backfill, and schedule=None removes the automatic cadence entirely.

Answer

Set catchup=True and use a TimeDeltaSchedule with timedelta(days=1).

Answer

Set start_date to the current date and schedule_interval=None in the DAG constructor.

Answer

Set max_active_runs=1 and schedule='0 2 * * *' in the DAG constructor.

Question 11

You are designing a Cloud Composer 2 DAG that must process files as they arrive in a Cloud Storage bucket. You want to minimize latency and avoid unnecessary worker resource consumption. Which TWO approaches should you use? (Choose two.)

Accepted Answer

Configure the bucket to send object change notifications to a Pub/Sub topic, and use a Pub/Sub sensor in the DAG to wait for messages.. Both a Pub/Sub sensor and a Cloud Function triggered by Cloud Storage events provide event-driven reactions to new files, minimizing latency and avoiding polling. The Pub/Sub sensor uses native Airflow operators and no custom code, while the Cloud Function uses the Airflow REST API to trigger the DAG. Polling sensors or frequent scheduled runs add latency and waste resources.

Answer

Schedule the DAG to run every minute with a cron expression and have it list the bucket for new files.

Answer

Use a Cloud Storage sensor that checks for new files at a fixed interval.

Answer

Increase the number of workers in the Cloud Composer environment to ensure sensors never block other tasks.

Question 12

You are building a Workflows definition that calls a Cloud Run service and then passes the returned JSON payload to a Cloud Function. The Cloud Run response body is stored in a variable named result. Which syntax correctly passes that value as an argument in the subsequent call step?

Accepted Answer

args:
  body: ${result}. Workflows uses ${...} to embed variable and expression values into YAML fields. When a previous step's response is assigned to a variable such as result, referencing ${result} substitutes that parsed JSON object directly into the next step's arguments. Omitting the braces, adding quotes, or inventing a body subfield all prevent the actual payload from being passed correctly.

Answer

args:
  body: ${"result"}

Answer

args:
  body: $result

Answer

args:
  body: ${result.body}

Question 13

You have a Cloud Composer 2 DAG that uses a PythonOperator to run a script that connects to a Cloud SQL database. The script needs a password. You want to avoid hardcoding the password in the DAG or in environment variables. What should you do?

Accepted Answer

Store the password in Secret Manager and use the Secret Manager Airflow operator or hook to retrieve it at runtime, granting the Composer service account the Secret Accessor role.. Secret Manager is designed for storing, managing, and accessing secrets securely. By using the Secret Manager operator or hook, the DAG retrieves the password at runtime, and IAM controls access. Other options store secrets in less secure locations like Cloud Storage, the Airflow database, or Airflow Variables, which lack the necessary security controls.

Answer

Use the Cloud Composer environment's Airflow metadata database to store the password in a custom table, and query it from the PythonOperator.

Answer

Define the password as an Airflow Variable in the Cloud Composer environment and reference it in the PythonOperator.

Answer

Store the password in a Cloud Storage bucket and have the PythonOperator download it using the Cloud Storage client library.

Question 14

You are using Workflows to orchestrate a data pipeline that includes a step calling a Cloud Function. The Cloud Function returns a JSON payload with a field named 'result'. You need to pass this 'result' value to a subsequent step as an argument. How should you reference the output of the Cloud Function step in the Workflows YAML?

Accepted Answer

Use the syntax ${functionStep.body.result} to access the result field.. In Workflows, the output of an HTTP call step is stored in the step's result under the body property. To access a specific field from the JSON response, you use the expression ${stepName.body.fieldName}. This is the correct way to pass data between steps, ensuring the 'result' value is correctly referenced.

Answer

Use the syntax ${functionStep.response.body.result} to access the result field.

Answer

Use the syntax ${functionStep.result} to access the result field directly.

Answer

Use the syntax ${functionStep.output.result} to access the result field.

Question 15

Your Cloud Composer DAG includes a task that runs a long-running data export, and you have observed that the task sometimes exceeds the default Airflow worker pod timeout, causing the task to be killed and retried. You want to increase the allowed execution time for this task without affecting other tasks. What should you do?

Accepted Answer

Set the execution_timeout parameter on the specific task in the DAG.. The execution_timeout parameter on an individual task sets the maximum runtime for that task, overriding any default. This is the correct way to extend time for a specific long-running task without altering global settings. Retries, schedule changes, and worker pod timeouts do not provide a targeted per-task timeout adjustment.

Answer

Add a retry_delay and increase retries in the task's default_args.

Answer

Change the DAG's schedule_interval to run less frequently.

Answer

Increase the worker pod timeout in the Cloud Composer environment configuration.