PDE Maintaining and Automating Data Workloads Practice Test 8 — 15 Questions

Question 1

Your company uses Cloud Composer to orchestrate a complex data pipeline. You need to ensure that the pipeline can recover from failures and that tasks are retried automatically with exponential backoff. You also want to be alerted if a task fails after all retries. Which combination of features should you implement?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Set retries and retry_exponential_backoff on each task, and use Cloud Monitoring alerts based on Airflow metrics.. Tasks in Airflow can be configured with retries and retry_exponential_backoff to automatically retry failed tasks with increasing delays. Cloud Composer exports metrics to Cloud Monitoring, where you can set up alerts for task failures. This combination provides both automatic recovery and alerting, fulfilling the requirements.

Answer

Use Airflow's SLA feature to trigger retries and send alerts.

Answer

Set retries and retry_delay on each task, and configure email alerts on task failure.

Answer

Configure a custom operator that implements retry logic and sends alerts via Pub/Sub.

Question 2

A data engineer must give a Dataproc Serverless for Spark batch workload permission to read objects from a specific Cloud Storage bucket and write to a BigQuery dataset, following least privilege. The workload runs as a custom service account. Which approach should be used?

Accepted Answer

Grant the workload's service account `roles/storage.objectViewer` on the bucket and `roles/bigquery.dataEditor` on the dataset.. Least privilege in Google Cloud means binding predefined or custom roles to the narrowest resource scope that still satisfies the workload. Bucket-level object viewer and dataset-level data editor grant precisely the read and write operations the Spark job requires. Broad project roles, default service accounts, and perimeter controls either overshoot the needed permissions or fail to grant them at all.

Answer

Grant the workload's service account the basic `roles/editor` role on the project so both Cloud Storage and BigQuery calls succeed.

Answer

Enable the Cloud Storage and BigQuery APIs and rely on the default Compute Engine service account attached to the Dataproc Serverless workload.

Answer

Create a VPC Service Controls perimeter around the bucket and dataset and add the workload's service account to the access level.

Question 3

A retail analytics team runs a Cloud Composer DAG that starts a Dataproc job, waits for it, and then runs a BigQuery load. The Dataproc job sometimes fails due to a transient YARN resource shortage, and the team wants the DAG to automatically retry the Dataproc submission a few times before alerting. Which Airflow configuration on the Dataproc task best meets this need?

Accepted Answer

Set retries to 3 and retry_delay to 5 minutes on the Dataproc job task.. Transient failures are best handled with operator-level retries, which re-execute the same task after a delay and only surface an alert once the configured attempts are exhausted. Setting retries to 3 with a short retry_delay on the Dataproc task gives the cluster room to recover from resource pressure while keeping the DAG's dependency chain and alerting behavior unchanged.

Answer

Set the Dataproc operator's cluster_name parameter to a different cluster and rely on Dataproc to resubmit the job.

Answer

Add a downstream EmailOperator that notifies the team on failure and manually reruns the DAG.

Answer

Set max_active_runs to 3 on the DAG so multiple DAG runs can absorb the failure.

Question 4

A data engineer has a BigQuery SQL script that must run every day at 06:00, load its results into a reporting table, and retry automatically if the query fails due to transient errors. The team has no existing orchestration tooling, wants the lowest operational overhead, and needs the schedule and the SQL to be managed entirely inside Google Cloud. Which approach should the engineer use?

Accepted Answer

Create a scheduled query in BigQuery with the desired SQL and a daily schedule, targeting the reporting table as the destination.. BigQuery scheduled queries are the native, serverless way to run SQL on a recurring schedule and write output to a destination table, with automatic retry of failed runs. Because the schedule, SQL, and destination all live inside BigQuery, no additional infrastructure or orchestration tooling is needed, matching the low-overhead requirement precisely.

Answer

Use Dataflow with a batch pipeline that reads from the source tables, applies the transformation, and writes to the reporting table on a schedule.

Answer

Create a Cloud Composer environment and author a single-task DAG that runs the SQL daily with Airflow retries configured.

Answer

Deploy the SQL as a Cloud Run service and create a Cloud Scheduler HTTP job that invokes it daily at 06:00.

Question 5

A media company runs a Cloud Composer environment whose DAGs trigger Dataflow batch jobs and BigQuery loads. The operations team reports that Composer costs are rising and that DAGs occasionally stall because workers are saturated. You review the environment and find that several tasks are long-running sensors that hold worker slots while waiting on external conditions. Which TWO changes should you make to reduce worker saturation and cost? (Choose two.)

Accepted Answer

Move the long-running Dataflow job monitoring out of the DAG by having the DAG submit the job and exit while a separate process tracks completion.. Worker saturation caused by long-blocking sensors and job-waiting tasks is best relieved by changing how those tasks consume resources rather than by adding concurrency limits or rescheduling. Deferrable sensors release slots during waits, and decoupling long Dataflow job monitoring from the worker keeps the pool available for other tasks, which lowers both stalls and cost.

Answer

Change the DAG schedule from a cron expression to a timedelta interval so runs are spaced further apart.

Answer

Set the Composer environment's worker count to the minimum and rely on autoscaling to add workers only when the queue is deep.

Answer

Increase the DAG's max_active_tasks parameter so more tasks can run in parallel on the same workers.

Question 6

You are using Cloud Composer to orchestrate a data pipeline that runs a Dataproc job to process data, followed by a BigQuery load. You notice that the Dataproc job sometimes takes longer than expected, causing the BigQuery load to start before the Dataproc job finishes, resulting in incomplete data. Which Airflow feature should you use to ensure the BigQuery load only runs after the Dataproc job completes successfully?

Accepted Answer

Set a dependency between the Dataproc job task and the BigQuery load task using the >> operator.. The correct way to enforce that the BigQuery load runs only after the Dataproc job completes successfully is to define a task dependency using the >> operator. This creates a directed edge in the DAG, ensuring the downstream task waits for the upstream task to succeed. Other options either do not control ordering or do not enforce success. Task dependencies are the core mechanism in Airflow for sequencing tasks in a data pipeline.

Answer

Use a TimeSensor to wait for the Dataproc job to finish.

Answer

Use the trigger_rule parameter to set the BigQuery load task to 'all_done'.

Answer

Set the Dataproc job task's retries to a high number.

Question 7

A data engineer must ensure that a Cloud Composer DAG which loads a BigQuery table runs every day at 02:00 UTC and that a dependent downstream report DAG runs only after the load succeeds. The report DAG lives in the same Composer environment but is a separate DAG file. Which feature should be used to coordinate the two DAGs?

Accepted Answer

An ExternalTaskSensor in the report DAG referencing the load DAG and task, with a matching execution date.. ExternalTaskSensor is the purpose-built mechanism for waiting on a task in another DAG. Configuring it with the correct execution date alignment makes the report DAG block until the load task succeeds, which is exactly the cross-DAG dependency described. Time-based and intra-DAG mechanisms cannot observe another DAG's success state.

Answer

A shared XCom written by the load DAG and read by the report DAG using xcom_pull across DAGs.

Answer

A single trigger_rule of all_success applied to the report DAG's root task.

Answer

A TimeSensor in the report DAG set to the expected load completion time.

Question 8

Your team runs a Cloud Composer 2 environment to orchestrate BigQuery ELT jobs. A DAG that loads a critical fact table must only run after an upstream ingestion DAG completes, and you want Composer itself to trigger it automatically without any external scheduler. Which mechanism should you use?

Accepted Answer

An ExternalTaskSensor that watches the upstream DAG's task in the same Composer environment.. Airflow's ExternalTaskSensor exists specifically to express a dependency on a task in another DAG within the same Airflow deployment. It checks the external DAG run for the matching execution date and only allows downstream tasks to proceed after the upstream task succeeds. Time-based waits, external cron triggers, and message-based glue all fail to guarantee that the ingestion actually finished successfully before the load begins.

Answer

A TimeDeltaSensor with a fixed delay that matches the upstream DAG's typical runtime.

Answer

A Pub/Sub push subscription that invokes a Cloud Function to call the downstream DAG's trigger endpoint.

Answer

A Cloud Scheduler job that calls the Airflow REST API to trigger the downstream DAG on a cron schedule.

Question 9

A data engineer needs to run a recurring nightly extract-transform-load job that pulls data from a REST API, applies Python transformations, and writes the output to a Cloud Storage bucket. The team wants a fully managed, serverless scheduler that can retry failed runs and send notifications, and they do not want to maintain any cluster or VM. Which Google Cloud service should they use to define and run this job?

Accepted Answer

Cloud Scheduler triggering a Cloud Function that performs the API call, transformation, and Cloud Storage write.. Cloud Scheduler plus Cloud Functions delivers a fully managed, serverless combination for a recurring ETL task. Cloud Scheduler handles the cron cadence and can target a function over HTTP or Pub/Sub, while Cloud Functions executes the Python code with automatic retries, integrated logging, and the ability to publish notifications. No cluster or VM is required, satisfying the operational constraints.

Answer

A Compute Engine instance running a cron job that executes a Python script and writes to Cloud Storage.

Answer

Cloud Composer with a single-node environment and a KubernetesPodOperator that runs the Python transformation.

Answer

Dataproc Serverless with a PySpark batch that calls the REST API and writes results to Cloud Storage.

Question 10

A financial services firm stores customer transaction data in BigQuery. Compliance requires that a nightly Cloud Composer DAG verify that the previous day's partition is complete before downstream reporting DAGs run, and that the reporting DAG never start if the verification fails. The engineer wants the dependency expressed inside orchestration rather than by polling from the reporting DAG. What should the engineer do?

Accepted Answer

Use a Dataset (Dataset-scheduled DAG) in Cloud Composer: have the verification DAG produce the dataset and schedule the reporting DAG to trigger on it.. Airflow datasets in Cloud Composer create a producer-consumer dependency where the verification DAG's successful completion updates a dataset and the reporting DAG is scheduled on that update. This keeps the dependency inside orchestration, avoids polling, and inherently prevents the reporting DAG from running when verification fails.

Answer

Create a Cloud Scheduler job that calls the reporting DAG's trigger endpoint only after a Cloud Function confirms verification succeeded.

Answer

Combine the verification and reporting logic into a single DAG so that task order enforces the dependency.

Answer

Have the reporting DAG poll the verification table in a loop with a sensor until a success row appears, then proceed.

Question 11

Your company uses Cloud Composer to run a daily ETL workflow. The workflow consists of several tasks that must run in a specific order. You want to receive an alert if any task fails. Which Cloud Monitoring feature should you use?

Accepted Answer

Create an alerting policy based on the Cloud Composer metric for task failures.. The recommended way to alert on task failures in Cloud Composer is to use Cloud Monitoring alerting policies based on the built-in Cloud Composer metrics for task failures. These metrics are automatically exported and can be used to trigger alerts when failures occur. This approach is native, scalable, and integrates with various notification channels. Other methods, such as log-based metrics or polling, are either redundant or less efficient.

Answer

Set up a Cloud Function that checks the Airflow UI periodically.

Answer

Configure Airflow email alerts in the DAG's default_args.

Answer

Use Cloud Logging to create a log-based metric for task failures.

Question 12

Your team uses Cloud Composer to orchestrate a daily ETL workflow that extracts data from an on-premises database, transforms it in Dataproc, and loads it into BigQuery. The workflow occasionally fails due to transient network issues. You want to automatically retry failed tasks without manual intervention. What should you do?

Accepted Answer

Set the retries parameter and retry_delay on the Airflow tasks in the DAG.. Airflow tasks in Cloud Composer support retries and retry_delay parameters, which allow automatic retries of failed tasks. This is the correct way to handle transient failures. Other options either do not address task-level retries or could cause data duplication.

Answer

Configure the Dataproc cluster to automatically restart on failure.

Answer

Enable automatic retries on the Cloud Composer environment.

Answer

Use Cloud Scheduler to trigger the DAG multiple times until it succeeds.

Question 13

You manage a Cloud Composer 2 environment that runs a DAG with a task using the BigQueryInsertJobOperator. The task occasionally fails with 'rateLimitExceeded' when submitting many jobs in parallel. You want to limit the number of concurrent BigQuery jobs submitted by this DAG without affecting other DAGs in the same environment. What should you do?

Accepted Answer

Use a pool with a limited number of slots and assign the BigQuery tasks to that pool.. Airflow pools are designed to limit parallelism for a set of tasks. Creating a pool with a limited number of slots and assigning the BigQuery tasks to it ensures that only a controlled number of BigQuery jobs are submitted concurrently, preventing rateLimitExceeded errors. This approach is scoped to the specific tasks and does not affect other DAGs or tasks in the environment.

Answer

Increase the number of workers in the Cloud Composer environment to distribute the load.

Answer

Configure the BigQueryInsertJobOperator with a lower priority for the jobs.

Answer

Set max_active_tasks on the DAG to a low value.

Question 14

A data engineer is migrating a Composer 1 environment to Cloud Composer 2 and notices that a DAG relying on a legacy operator for a deprecated service no longer works. The engineer wants a durable fix that keeps the pipeline running and avoids repeating this problem in future upgrades. Which action should be taken?

Accepted Answer

Move the deprecated logic into a PythonOperator that calls the service's current client library, and add unit tests to validate the behaviour.. Replacing deprecated operators with a PythonOperator that calls the service's current client library removes reliance on unsupported code and keeps the DAG functional on Composer 2. Adding tests ensures the replacement behaves correctly and catches regressions during future upgrades, which is the durable, maintainable outcome the scenario asks for.

Answer

Downgrade the environment's Python version to match the requirements of the legacy operator.

Answer

Pin the Composer 2 environment to the exact Airflow version that still includes the legacy operator and avoid upgrades.

Answer

Copy the legacy operator's source into the DAGs folder so it is imported locally instead of from the provider package.

Question 15

You manage several Cloud Composer 2 environments that run production DAGs. You must define an alerting strategy that detects when a DAG run fails and when a task is stuck retrying for an unusually long time, using Cloud Monitoring. (Choose two.)

Accepted Answer

Create a Monitoring alerting policy on the airflow task instance duration or retry-related metric exposed through Cloud Monitoring for the environment.. Composer surfaces Airflow execution data in two complementary ways: structured log entries in Cloud Logging and Airflow metrics in Cloud Monitoring. A log-based alerting policy catches the explicit DAG run failure event, while a metric-based policy on task duration or retries catches a task that is hanging or looping. Together they cover both the hard failure and the stuck-retrying condition, whereas health, uptime, and audit signals describe infrastructure and control-plane state rather than workload outcomes.

Answer

Create an uptime check against the Airflow web server URL and alert when the HTTP response is not 200.

Answer

Enable Cloud Audit Logs for the Composer API and alert when any environment update method is called.

Answer

Create a Monitoring alerting policy on the composer.googleapis.com/environment/healthy metric and notify when it drops below the threshold.