GCP-ADP Data Pipeline Orchestration — 25 Questions

Question 1

In Cloud Composer, you want to prevent a task from retrying if it fails due to a specific error (e.g., File Not Found). How can you configure this?

Accepted Answer

Raise 'AirflowSkipException' in your task code.. Airflow does not support per-error-type retry suppression natively, so you must raise an 'AirflowSkipException' or similar in custom code to avoid retries.

Answer

Set 'retries: 0' in the task operator.

Answer

Use the 'on_failure_callback' to cancel retries.

Answer

Configure 'retry_delay' to be infinite.

Question 2

You are orchestrating a pipeline with Workflows that invokes several Cloud Functions. You need to pass the output of one function as an argument to the next. What is the correct syntax?

Accepted Answer

Assign the call result to a variable and reference it in the next step.. In Workflows, you capture the output of a call and pass it as an input variable to the next step using the assignment operator.

Answer

Use global environment variables to store intermediate state.

Answer

Combine all function calls into a single synchronous block.

Answer

Write the output to a temporary GCS file and read it back.

Question 3

In a Cloud Composer DAG, you need to trigger a BigQuery job that runs as a service account other than the one assigned to the GKE nodes. How should you implement this?

Accepted Answer

Use the 'impersonation_chain' parameter in the BigQueryInsertJobOperator.. Using the 'impersonation_chain' parameter in the BigQueryInsertJobOperator allows the service account running the task to impersonate a different service account.

Answer

Hardcode the credentials in a JSON file inside the DAG folder.

Answer

Change the worker node's identity in the Cloud Console.

Answer

Configure a connection with the 'Service Account Key' type in Airflow UI.

Question 4

You need to run a task in Cloud Composer only if the previous task failed. Which Trigger Rule should you use?

Accepted Answer

one_failed. The 'one_failed' trigger rule allows a task to execute if at least one parent task has failed.

Answer

none_failed

Answer

always

Answer

all_success

Question 5

When configuring a Workflows execution, you want to ensure the workflow can be triggered by an HTTP POST request. Which feature allows this?

Accepted Answer

Executing the workflow via the REST API.. Workflows can be triggered via HTTP POST requests to the workflow execution API endpoint.

Answer

Configuring a Cloud Scheduler job with an HTTP target.

Answer

Enabling the 'HTTP Gateway' on the workflow itself.

Answer

Creating a Service Account with 'Public Access'.

Question 6

Your organization uses Workflows to orchestrate multi-step API calls. You need to handle transient failures in a specific HTTP request by retrying it up to 5 times with exponential backoff. Which block should you use?

Accepted Answer

A 'try' block with a 'retry' policy.. The 'try/retry' block in Workflows syntax allows for configuring custom backoff and max attempts for specific steps.

Answer

A 'loop' block with a counter increment.

Answer

An 'if' statement checking for 500 status codes.

Answer

A global 'exception' handler at the end of the workflow.

Question 7

A Cloud Composer DAG is failing because the Airflow database is locking up during high-concurrency periods. What is the most effective way to address this?

Accepted Answer

Scale up the Cloud Composer environment size to increase database resources.. Scaling the Composer environment or adjusting the database performance settings (or Airflow configuration) is the recommended path to resolve lock contention.

Answer

Delete all past DAG run logs.

Answer

Increase the number of DAGs in the repository.

Answer

Set the 'parallelism' parameter to a very high value.

Question 8

Which Cloud Composer metric should you monitor to identify if your Airflow scheduler is struggling to parse your DAGs in a timely manner?

Accepted Answer

dag_processing_total_parse_time. The 'dag_processing_total_parse_time' metric specifically tracks the time taken to process and parse DAG files.

Answer

airflow_task_execution_time

Answer

dag_run_success_count

Answer

scheduler_heartbeat

Question 9

You are deploying a Cloud Composer 2 environment and need to ensure that the DAG files are automatically synchronized to all workers. Where should you store these files?

Accepted Answer

In the 'dags/' folder of the environment's bucket.. Cloud Composer 2 automatically synchronizes DAGs stored in the 'dags/' folder of the associated Cloud Storage bucket to all environment components.

Answer

Inside a Docker volume configured in the environment's YAML manifest.

Answer

In the local /home/airflow/gcs/dags directory of each worker node.

Answer

In a custom persistent disk attached to the Airflow scheduler.

Question 10

Which component of Cloud Composer acts as the primary interface for managing and triggering DAGs manually?

Accepted Answer

Airflow Web Interface. The Airflow Web Interface is the primary UI for managing DAGs and viewing execution history.

Answer

Cloud Storage Browser

Answer

Cloud Shell

Answer

BigQuery Console

Question 11

You have a Workflow that must wait for a specific time duration before proceeding. Which step type is most appropriate?

Accepted Answer

sys.sleep. The 'sys.sleep' function in Workflows allows pausing the execution for a defined duration.

Answer

pause_execution

Answer

wait_for_condition

Answer

delay_step

Question 12

When using Cloud Scheduler to trigger a Workflows execution, what is the best practice for authenticating the scheduler to the Workflows API?

Accepted Answer

Use an OIDC token with a service account.. Using an OIDC token with a service account that has the 'Workflows Invoker' role is the secure, recommended method.

Answer

Use an API Key generated in the console.

Answer

Hardcode credentials in the scheduler job.

Answer

Use 'Public' access for the workflow.

Question 13

You are migrating a legacy Airflow environment to Cloud Composer. You need to ensure that local Python packages used by custom operators are installed correctly. What is the recommended approach?

Accepted Answer

Upload the requirements.txt file to the environment bucket.. Define custom Python dependencies in the 'requirements.txt' file in the environment's bucket for Composer to install.

Answer

Create a custom Docker image for the Airflow environment.

Answer

Manually SSH into worker nodes and run 'pip install'.

Answer

Pre-install packages in the DAG file using 'os.system'.

Question 14

You need to run a Workflows execution based on a schedule. Which Google Cloud service is designed to manage this triggering?

Accepted Answer

Cloud Scheduler. Cloud Scheduler is the fully managed service for scheduling jobs, including triggers for Workflows.

Answer

Eventarc

Answer

Cloud Tasks

Answer

Pub/Sub

Question 15

Your team needs to audit all modifications made to DAG files in the GCS bucket used by Cloud Composer. What is the most efficient way to achieve this?

Accepted Answer

Enable Data Access audit logs on the GCS bucket.. Enable Cloud Audit Logs (Data Access logs) on the GCS bucket to track all API calls including object modifications.

Answer

Enable 'Version History' in the Airflow UI.

Answer

Use Cloud Monitoring to alert on 'gcs_object_write' events.

Answer

Write a custom DAG to scan the bucket every minute.

Question 16

You are troubleshooting a stuck task in Cloud Composer. You want to see the stdout/stderr logs for that specific task attempt. Where can you find them?

Accepted Answer

In the 'logs/' folder in the environment's GCS bucket.. Airflow task logs are stored in the GCS bucket associated with the environment, organized by DAG and task name.

Answer

In the Airflow metadata database directly.

Answer

In the Cloud Composer environment console logs.

Answer

Via the 'kubectl' logs command on the worker pod.

Question 17

In Cloud Composer, what is the default behavior if a DAG execution takes longer than the 'dagrun_timeout' setting?

Accepted Answer

The DAG run is marked as failed.. If a DAG run exceeds the timeout, Airflow marks the DAG run as 'failed'.

Answer

The DAG enters a 'running' state indefinitely.

Answer

The DAG run is paused.

Answer

The scheduler automatically restarts the DAG.

Question 18

Which property in a Workflows YAML definition is required to define the entry point of the workflow?

Accepted Answer

main. The 'main' step is the required starting point for any workflow execution.

Answer

entry_point

Answer

init

Answer

start

Question 19

Which command-line tool do you primarily use to deploy and manage Workflows?

Accepted Answer

gcloud. The 'gcloud' CLI is the standard tool for managing Google Cloud services, including Workflows.

Answer

gsutil

Answer

kubectl

Answer

bq

Question 20

A task in your Airflow DAG involves a long-running process that might exceed the default Airflow worker pod timeout. How should you handle this?

Accepted Answer

Set the 'execution_timeout' parameter on the task instance.. Adjust the 'execution_timeout' parameter for the specific task in the Airflow DAG definition to allow longer processing times.

Answer

Move the task to a custom node pool in GKE.

Answer

Increase the 'default_dag_run_timeout'.

Answer

Decrease the number of parallel tasks.