GCP-ADP Data Pipeline Orchestration Practice Test 2 — 15 Questions

Question 1

You are creating a Workflows definition to orchestrate a series of microservices. You need to define the starting point of the workflow. Which field in the Workflows YAML is used to specify the entry point?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

main. In Workflows, the main field is required and defines the entry point of the workflow. It contains the steps that are executed when the workflow runs. Other fields like entrypoint, start, and init are not valid in Workflows YAML and would cause errors. The main field is the correct way to specify the workflow's starting point.

Answer

start

Answer

entrypoint

Answer

init

Question 2

Your Cloud Composer 2 environment runs an Airflow DAG that starts a Dataflow streaming job and then needs to wait until that job reports a terminal state before running a downstream cleanup task. You want the DAG to pause without occupying a worker slot while the Dataflow job runs, and to resume as soon as the job finishes. What should you do?

Accepted Answer

Use the DataflowStartFlexTemplateOperator and chain it to a DataflowJobStatusSensor set to the desired terminal state.. Waiting for an external service without holding an Airflow worker slot is best done with a deferrable sensor. DataflowJobStatusSensor polls the job state and defers the task until the configured terminal state is reached, so downstream cleanup starts exactly when the job completes. A PythonOperator polling loop or a fixed delay sensor either consumes worker capacity or waits the wrong amount of time.

Answer

Use the DataflowStartFlexTemplateOperator followed by a separate DataflowStopJobOperator with a fixed delay via a TimeDeltaSensor.

Answer

Use a BashOperator that calls gcloud dataflow jobs show with the --wait flag inside the task.

Answer

Use a PythonOperator with a while loop that calls the Dataflow jobs.get API every 60 seconds and sleeps between checks.

Question 3

You are writing a Workflows YAML definition that must call a Cloud Function, extract a value from its JSON response, and pass that value as an argument to a subsequent BigQuery insert step. You need to reference the output of the first step in the second step. Which syntax should you use?

Accepted Answer

Reference the first step's result using the expression ${firstStep.response.body.field} in the second step's arguments.. Workflows exposes the result of a call step through the step's name, and HTTP responses are nested under response.body. To pass a value from a Cloud Function's JSON response into a later BigQuery step, you reference ${firstStep.response.body.field} in the arguments. Other property paths or nonexistent keywords do not resolve at runtime.

Answer

Reference the first step's result using the expression ${firstStep.body.field} in the second step's arguments.

Answer

Store the output in a variable using the assign keyword and reference the variable in the next step.

Answer

Use the result of the first step by referencing ${firstStep.response.body.field} in the arguments of the second step.

Question 4

You are creating a Cloud Workflows definition to orchestrate a data pipeline. You need to define the starting point of the workflow. Which top-level property must you include in the YAML definition to specify the first step to execute?

Accepted Answer

`main`. Cloud Workflows requires a top-level `main` block that contains a `steps` list. The first step in that list is the entry point. Other properties like `entrypoint`, `start`, or `init` are not part of the Workflows YAML schema and will cause deployment failures.

Answer

`start`

Answer

`entrypoint`

Answer

`init`

Question 5

You are using Workflows to orchestrate a data processing pipeline. One step invokes a Cloud Function that returns a JSON payload containing a processed record ID. You need to pass this record ID as an argument to a subsequent Cloud Function. How should you reference the output of the first function in the Workflows YAML definition?

Accepted Answer

Assign the output of the first function to a variable in the first step, then reference that variable in the second step.. To pass data between steps in Workflows, you must capture the output of a function call by assigning it to a variable. The 'assign' step stores the result, and subsequent steps can reference that variable using dot notation. This is the standard and recommended method for data passing within a workflow.

Answer

Store the output in a Cloud Storage bucket from the first function, then read it in the second function.

Answer

Use the 'output' keyword in the first step to define the return value, then reference it in the second step.

Answer

Use the result of the first function call directly in the args field of the next step, like ${function1.result.recordId}.

Question 6

You are building a Workflows pipeline that calls a Cloud Function to process an image and then stores the result in Cloud Storage. The Cloud Function returns a JSON object with a 'status' field and a 'fileUrl' field. You need to pass the 'fileUrl' to a subsequent step that uploads the file to a Cloud Storage bucket. How should you reference the output of the Cloud Function in the Workflows YAML definition?

Accepted Answer

Assign the call output to a variable, e.g., - processImage: call: http.post ... result: imageResult, then use ${imageResult.body.fileUrl} in the next step.. In Workflows, step outputs must be explicitly captured using the 'result' field. For an HTTP call, the result contains the response body. By assigning the call output to a variable (e.g., 'imageResult'), you can then reference nested fields like ${imageResult.body.fileUrl}. This is the correct way to pass data between steps and ensures the 'fileUrl' is available for subsequent steps.

Answer

Use the Cloud Function's response directly in the next step by referencing ${http.response.body.fileUrl}.

Answer

Use the predefined variable ${steps.processImage.output.fileUrl} to access the output of the Cloud Function call.

Answer

Use the result of the call step directly in the next step's arguments, such as ${result.fileUrl}.

Question 7

You are deploying a Cloud Composer environment and need to install a custom Python package that your DAGs depend on. The package is not available on PyPI and is provided as a .whl file in a Cloud Storage bucket. What is the recommended way to make this package available to all Airflow workers?

Accepted Answer

Use the gcloud composer environments update command with the --update-pypi-packages flag to install from the .whl file's Cloud Storage URL.. The recommended method to install a custom Python package from a Cloud Storage .whl file in Cloud Composer is to use the gcloud composer environments update command with the --update-pypi-packages flag, specifying the gs:// URL of the .whl file. This ensures the package is installed on all Airflow workers and the scheduler, and it persists across environment updates.

Answer

Place the .whl file in the Composer environment's plugins folder and restart the environment.

Answer

Upload the .whl file to the Composer environment's DAGs folder and reference it in the DAG.

Answer

SSH into each Airflow worker and run pip install with the Cloud Storage URL.

Question 8

You are designing a Cloud Composer 2 deployment to orchestrate a data pipeline that loads files from Cloud Storage into BigQuery. Your security team requires that the Airflow tasks access Google Cloud services using the environment's own identity rather than user credentials, and that DAG authors can use pre-built operators for GCS and BigQuery. Which two actions should you take to meet these requirements? (Choose two.)

Accepted Answer

Install the apache-airflow-providers-google package into the environment's Python dependencies.. Using the environment's own identity requires granting the Cloud Composer service account the appropriate IAM roles, since tasks default to that identity. Making the GCS and BigQuery operators available requires the apache-airflow-providers-google package, which contains those operators. Together these two actions provide both the permissions and the operator code needed for the pipeline while avoiding user credentials.

Answer

Set the environment variable AIRFLOW__CORE__DEFAULT_IMPERSONATION_SERVICE_ACCOUNT to a service account with BigQuery Admin.

Answer

Enable Private Google Access on the environment's subnet so tasks can reach the BigQuery and GCS APIs without external IP addresses.

Answer

Create an Airflow connection of type Google Cloud with a JSON key file for a dedicated user account and reference it in every operator.

Question 9

You are using Cloud Composer to orchestrate a data pipeline. A task in your Airflow DAG runs a long-running process that may exceed the default Airflow worker pod timeout. You need to ensure the task can run for up to 2 hours without being killed. What should you do?

Accepted Answer

Set the `execution_timeout` parameter on the task to a timedelta of 2 hours.. The `execution_timeout` parameter on a task is the correct way to specify a maximum runtime. It is a per-task setting that overrides the default. Other options either reference non-existent parameters or control different aspects of scheduling, not task execution duration.

Answer

Modify the `airflow.cfg` configuration to increase `worker_pod_timeout` to 7200 seconds.

Answer

Increase the `scheduler_zombie_task_threshold` to 7200 seconds.

Answer

Set the `dagrun_timeout` parameter on the DAG to 2 hours.

Question 10

You are using Cloud Scheduler to trigger a Workflows execution every hour. You need to authenticate the scheduler job to the Workflows API. Which authentication method should you configure in the Cloud Scheduler job?

Accepted Answer

OIDC token. To authenticate Cloud Scheduler to the Workflows API, you must use an OIDC token. The token should be associated with a service account that has the Workflows Invoker role. This is the standard and secure method for service-to-service authentication in Google Cloud.

Answer

OAuth token

Answer

Basic authentication

Answer

API key

Question 11

You are designing a data pipeline that uses Cloud Composer to orchestrate tasks across multiple Google Cloud services. You need to ensure that the pipeline can be triggered automatically on a schedule and also manually by data analysts. Which TWO methods can be used to trigger a Cloud Composer DAG? (Choose two.)

Accepted Answer

Configuring a schedule_interval in the DAG definition to run at specified times.. Cloud Composer DAGs can be triggered automatically by defining a schedule_interval in the DAG, which uses cron or timedelta to run at specific times. They can also be triggered manually through the Airflow web interface, which provides a 'Trigger DAG' button. These two methods cover the scheduled and manual triggering requirements.

Answer

Using the gcloud composer environments run command to execute the DAG.

Answer

Creating a Cloud Scheduler job that sends an HTTP request to the Airflow API.

Answer

Publishing a message to a Pub/Sub topic that the DAG is subscribed to.

Question 12

You are migrating an on-premises Airflow deployment to Cloud Composer 2. Your DAGs use custom operators that depend on a specific Python package version not available on PyPI. You need to ensure these operators work in Cloud Composer without modifying the DAG code. What should you do?

Accepted Answer

Build a custom container image that includes the package and configure the Composer environment to use it.. Cloud Composer 2 allows you to specify a custom container image for the environment, which can include non-PyPI packages and system dependencies. Building such an image and configuring the environment to use it ensures the custom operators have the required package without altering DAG code. Other methods do not properly install packages into the Airflow runtime.

Answer

Upload the package as a zip file to the dags/ folder and add it to sys.path in the DAG.

Answer

Use the gcloud composer environments update command to install the package from a local directory.

Answer

Place the package source in the dags/ folder and import it directly in the custom operator.

Question 13

You are designing a data pipeline using Cloud Composer. You need to ensure that a specific task runs only after two upstream tasks, taskA and taskB, have both completed successfully. Which TWO of the following approaches correctly define this dependency in the DAG? (Choose two.)

Accepted Answer

Set taskC.set_upstream(taskA) and taskC.set_upstream(taskB).. In Airflow, to make a task depend on multiple upstream tasks, you can either call set_upstream on the downstream task for each upstream task, or use the bitshift operator to define the relationships. Both methods ensure the downstream task waits for all specified upstream tasks. The other options either reverse the dependency or create an incorrect linear chain.

Answer

Use a list: taskC.set_upstream([taskA, taskB]).

Answer

Set taskC.set_downstream(taskA) and taskC.set_downstream(taskB).

Answer

Use the chain function: chain(taskA, taskB, taskC).

Question 14

Your organization uses Cloud Scheduler to start a Workflows execution every hour. The Workflows definition expects no input arguments and simply runs a sequence of API calls. Which HTTP method and target should the Cloud Scheduler job use to invoke the workflow?

Accepted Answer

POST to https://workflowexecutions.googleapis.com/v1/projects/PROJECT_ID/locations/REGION/workflows/WORKFLOW_ID/executions. Starting a Workflows execution is done through the Workflows Executions API by sending a POST request to the executions collection under the specific project, region, and workflow. This endpoint creates a new execution and returns its identifier. Retrieving metadata, using the wrong host, or invoking a nonexistent method will not start the pipeline.

Answer

PUT to https://workflows.googleapis.com/v1/projects/PROJECT_ID/locations/REGION/workflows/WORKFLOW_ID/run

Answer

GET to https://workflowexecutions.googleapis.com/v1/projects/PROJECT_ID/locations/REGION/workflows/WORKFLOW_ID

Answer

POST to https://workflows.googleapis.com/v1/projects/PROJECT_ID/locations/REGION/workflows/WORKFLOW_ID:execute

Question 15

You are designing a data pipeline using Cloud Workflows. You need to pass the output of one step as an argument to a subsequent step. Which TWO of the following are valid ways to reference the output of a previous step in a Workflows YAML definition? (Choose two.)

Accepted Answer

Use the `result` variable that stores the return value of the previous step.. The output of a step is stored in the built-in `result` variable, which can be referenced directly or assigned to a custom variable using an `assign` step. These are the only valid mechanisms; other syntaxes like step name references or output keywords do not exist in Cloud Workflows.

Answer

Use the `output` keyword in the step definition to name the output variable.

Answer

Reference the step name directly, e.g., `${stepName.output}`.

Answer

Use the `return` statement inside the step to specify the output variable name.