PDE Maintaining and Automating Data Workloads Practice Test 7 — 15 Questions

Question 1

A data engineer schedules a Cloud Composer 2 environment to run a DAG that triggers a Dataflow batch job every night. The DAG sometimes fails because the Dataflow job takes longer than the default task timeout. The engineer wants the DAG to wait for the Dataflow job to finish rather than timing out. Which change should the engineer make?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Set the Dataflow operator's execution_timeout and deferrable parameters so the task waits for job completion.. Cloud Composer 2 Dataflow operators can block until the job reaches a terminal state, and the deferrable variant frees the worker while waiting. Setting execution_timeout to a value longer than the expected job duration prevents premature task failure. Schedule interval, Cloud Function wrapping, and retry policies do not make a task wait for the underlying Dataflow job.

Answer

Add a retry policy to the DAG so failed tasks are retried until the Dataflow job finishes.

Answer

Increase the DAG's schedule interval so runs start less frequently.

Answer

Move the Dataflow launch into a Cloud Function and call it from the DAG.

Question 2

You manage a Cloud Composer environment that runs a critical DAG every hour. The DAG includes a task that calls a Cloud Function to process data. Recently, the Cloud Function started taking longer than expected, causing the DAG to exceed its SLA. You need to detect this delay and automatically retry the task if it fails due to timeout, while minimizing changes to the DAG. What should you do?

Accepted Answer

Set the task's execution_timeout parameter to a value slightly above the expected runtime and configure retries with a delay.. Using the task's execution_timeout parameter allows Airflow to enforce a time limit on the task. If the task exceeds this limit, it fails and can be automatically retried based on the retries and retry_delay settings. This requires only a small change to the DAG's task definition and leverages native Airflow features, effectively addressing both detection of delays and automatic retries.

Answer

Use a Cloud Monitoring alert on the Cloud Function's execution time and manually trigger the DAG upon alert.

Answer

Increase the Cloud Function's timeout setting to the maximum allowed and rely on the DAG's default retry behavior.

Answer

Add a Python operator that polls the Cloud Function's status and raises an exception if it exceeds a threshold, then set retries on that operator.

Question 3

Your organization runs a Cloud Composer 2 environment that executes dozens of DAGs. Several DAGs share a connection to an external REST API that enforces a rate limit of 100 requests per minute. During peak hours, DAGs fail with HTTP 429 errors. You want to prevent these failures without changing the external API's limits. What should you do?

Accepted Answer

Create an Airflow pool with a limited number of slots and assign the affected tasks to that pool.. Airflow pools provide a shared concurrency limit that spans DAGs. Sizing a pool to keep total in-flight API calls under the provider's rate limit prevents 429 responses. Increasing workers, limiting per-DAG active runs, or relying on retries does not coordinate request volume across the many DAGs that share the external API.

Answer

Increase the number of Celery workers in the Composer environment.

Answer

Set the DAG's max_active_runs to 1 for each affected DAG.

Answer

Add exponential backoff retries to the API-calling tasks.

Question 4

A data engineering team manages BigQuery datasets across multiple projects. They want to automatically detect and respond when a scheduled query fails, and they want the response to create an incident in their existing ticketing system. The team prefers minimal custom infrastructure and wants to use native Google Cloud tooling. Which approach should they use?

Accepted Answer

Enable BigQuery audit logs, create a log-based alerting policy in Cloud Monitoring, and route the alert to a Pub/Sub topic that triggers a Cloud Function to open the ticket.. Cloud Audit Logs capture BigQuery job failures, and log-based alerting policies in Cloud Monitoring can trigger Pub/Sub notifications. A Cloud Function subscribed to the topic can call the ticketing API, creating incidents automatically. This event-driven chain uses managed services, avoids polling, and requires little custom code compared with the other options.

Answer

Schedule a Cloud Scheduler job that polls the BigQuery INFORMATION_SCHEMA.JOBS view and emails the team when errors are found.

Answer

Configure BigQuery to send failure notifications to a Cloud Storage bucket and use a Dataproc job to parse them.

Answer

Use the BigQuery REST API from a long-running Compute Engine VM to poll for failed jobs and call the ticketing API.

Question 5

Your team runs a Cloud Composer 2 environment (composer-2.1.0-airflow-2.6.3) that executes a daily BigQuery ETL workflow. The workflow must not run on weekends. You want to implement this with minimal code and without modifying the DAG's task logic. What should you do?

Accepted Answer

Set the DAG's schedule_interval to '0 2 * * 1-5' and set catchup=False.. The correct approach is to use a cron expression that runs only Monday through Friday, such as '0 2 * * 1-5', combined with catchup=False to avoid backfilling. This leverages Airflow's native scheduling, requires no changes to task logic, and keeps the DAG simple. Other methods either add external dependencies or still trigger the DAG on weekends.

Answer

Add a BranchPythonOperator that checks the day of the week and skips downstream tasks on weekends.

Answer

Set the DAG's schedule_interval to '@daily' and add a ShortCircuitOperator that stops execution on weekends.

Answer

Create a Cloud Scheduler job that triggers the Composer DAG only on weekdays via the Airflow REST API.

Question 6

You are designing the deployment process for a Dataflow streaming pipeline that processes financial transactions. The pipeline must be updated without losing in-flight state, such as open windows and timers, and without downtime. Your team uses the Apache Beam Java SDK and deploys from a CI/CD pipeline. Which update strategy should you use?

Accepted Answer

Submit an update with the --update option and a compatible transform change to replace the pipeline in place.. Dataflow update replaces a running streaming pipeline in place while preserving state, provided the transform changes are compatible with the update compatibility rules. This avoids downtime and keeps windowed state intact. Draining, snapshot-based restart, and cancel-redeploy all interrupt processing or discard state, making them unsuitable for stateful, zero-downtime updates.

Answer

Cancel the pipeline and redeploy with a new job name.

Answer

Use a snapshot and then start a new job from the snapshot to preserve state.

Answer

Stop the pipeline with a drain, then start a new pipeline from the same template.

Question 7

You manage a Cloud Dataflow streaming pipeline that reads from Pub/Sub and writes to BigQuery. The pipeline uses the BigQueryIO write transform with STREAMING_INSERTS. You need to ensure exactly-once processing semantics for the BigQuery writes. What should you do?

Accepted Answer

Use BigQueryIO.Write with withMethod(STORAGE_WRITE_API) and set withNumStorageWriteApiStreams to a value greater than zero.. To achieve exactly-once processing for BigQuery writes in a Dataflow streaming pipeline, use the BigQuery Storage Write API with withMethod(STORAGE_WRITE_API) and configure at least one stream. This method provides transactional writes and deduplication, ensuring each record is written exactly once. Other methods like STREAMING_INSERTS with insertId offer only best-effort deduplication and cannot guarantee exactly-once semantics.

Answer

Add a BigQuery insertId to each record and rely on BigQuery's built-in deduplication for streaming inserts.

Answer

Switch to BigQueryIO.Write with withMethod(FILE_LOADS) and set withTriggeringFrequency to a low value.

Answer

Enable exactly-once by setting the pipeline option --experiments=use_runner_v2 and using BigQueryIO.Write with withMethod(STREAMING_INSERTS).

Question 8

Your organization runs a Cloud Composer (Apache Airflow) environment that executes a DAG every night to move data from Cloud Storage into BigQuery. The DAG has been succeeding for months, but last week the nightly load silently produced a BigQuery table with zero rows while the Airflow task still reported success. You need to add a safeguard that fails the DAG task whenever the loaded row count is zero before downstream tasks run. What should you do?

Accepted Answer

Add a BigQueryCheckOperator task immediately after the load task that asserts the target table row count is greater than zero.. The failure mode is a task that reports success while producing an empty result set, so the safeguard must inspect the data itself and raise a task-level failure. A BigQueryCheckOperator performs exactly that assertion and integrates with Airflow's dependency graph, which prevents downstream tasks from consuming invalid output and triggers configured alerting.

Answer

Increase the retries parameter on the load task so transient issues that produce zero rows are automatically retried.

Answer

Enable Cloud Logging export for the Composer environment and create a log-based alert that matches error text from the load operator.

Answer

Configure an Airflow SLA on the load task so that a missed deadline raises an alert about the empty table.

Question 9

Your team runs a Cloud Composer (Airflow) environment that executes a nightly BigQuery ELT DAG. A downstream task must only run after an upstream task that loads a partitioned table completes, and you want the downstream task to wait for the load's completion signal without polling BigQuery repeatedly. Which Airflow mechanism should you implement to coordinate these tasks within the DAG?

Accepted Answer

Define a direct downstream dependency (loader >> downstream) so Airflow's scheduler marks the downstream task runnable only after the loader task succeeds.. A direct dependency edge is the canonical Airflow way to sequence tasks inside one DAG: the scheduler keeps the downstream task in a waiting state until the upstream load succeeds. Sensors and XComs serve other purposes and would add complexity or incorrect semantics here. The dependency edge provides the required completion gating with no polling of BigQuery.

Answer

Use a task dependency with an ExternalTaskSensor pointed at the loader task's DAG and task ID, configured with the correct execution_delta.

Answer

Set the loader task's trigger_rule to all_done and add a dependency edge so the downstream task runs immediately after the loader reaches any terminal state.

Answer

Use an XCom to push a completion flag from the loader task and have the downstream task pull it via xcom_pull, combined with a trigger rule of all_success on the dependency edge.

Question 10

You need to schedule a recurring BigQuery query that aggregates data from a partitioned table and writes the results to a new table every day at 03:00 UTC. You want a fully managed solution with minimal operational overhead. What should you use?

Accepted Answer

BigQuery scheduled queries.. BigQuery scheduled queries are a fully managed feature that lets you schedule SQL queries to run at specified intervals. They require no additional infrastructure, support cron scheduling, and can write results to destination tables. This makes them the ideal choice for a recurring aggregation query with minimal operational overhead, unlike Cloud Functions, Composer, or VM-based cron jobs.

Answer

Cloud Scheduler with a cron job that invokes a Cloud Function to run the query.

Answer

Cloud Composer with a DAG that runs a BigQueryOperator.

Answer

A cron job on a Compute Engine instance that runs the bq query command.

Question 11

A data engineering team maintains several Cloud Composer DAGs that load files from Cloud Storage into BigQuery. They want the DAG to start only after a new file has landed in the source bucket, rather than polling on a fixed schedule and often finding nothing. Which Airflow construct should they use to trigger the DAG based on the arrival of the object?

Accepted Answer

A GCSObjectExistenceSensor with a poke interval, followed by the load task.. The requirement is to begin processing only when a specific object appears, which is precisely what a Cloud Storage sensor provides. Sensors defer downstream tasks until their condition is met, so pairing a GCSObjectExistenceSensor with the load task converts the DAG from a blind schedule into a data-arrival trigger while keeping the existing load logic intact.

Answer

A TimeDeltaSensor set to the expected upload window before the load task.

Answer

A BigQueryInsertJobOperator configured to load the file directly from the bucket path.

Answer

An ExternalTaskSensor pointed at a separate file-upload DAG.

Question 12

Your team runs a Cloud Composer 2 environment in project `analytics-prod`. A nightly DAG loads Cloud Storage files into BigQuery, but a recent Cloud Storage outage caused several tasks to fail after exhausting their retries. You want failed task instances to automatically re-run without manual intervention once the upstream dependency recovers. What should you do?

Accepted Answer

Define a task-level `on_failure_callback` that calls the Airflow REST API to clear the failed task instances after a delay.. Failed task instances that have exhausted retries stay in a failed state until they are cleared, at which point the scheduler re-queues them. Triggering that clear programmatically, for example from a failure callback that waits for the dependency to recover, restores the pipeline automatically. Backfill settings, retry counts, and timeout tuning only affect future scheduling or in-flight attempts, not instances already marked failed.

Answer

Set the Airflow configuration option `[core] default_task_retries` to a higher value in the environment configuration and restart the schedulers.

Answer

Increase the DAG's `retry_delay` and `execution_timeout` so tasks wait longer for Cloud Storage to recover before failing.

Answer

Configure the DAG with `catchup=True` and set `max_active_runs=1` so missed schedule intervals are backfilled.

Question 13

A data engineer manages a Cloud Composer 2 environment. A DAG that downloads a large reference dataset each night occasionally exceeds the default task timeout because the source API is slow. The engineer wants the task to fail fast and be retried automatically rather than hanging for hours, and wants failed runs to be visible for alerting. Which configuration should be applied to the task?

Accepted Answer

Set execution_timeout on the task to a bounded value and set retries with a retry_delay so failed attempts are re-queued and surfaced in task state.. execution_timeout is the task-level setting that bounds wall-clock runtime and triggers a failure, while retries and retry_delay make the scheduler automatically re-queue the task. Together they produce fast failure, automatic retry, and a recorded state that alerting can observe, which is precisely what the scenario requires for a slow external API.

Answer

Configure the worker's celery_worker_autoscale setting to add more workers so the slow API call finishes sooner.

Answer

Set depends_on_past to true and add a retry_exponential_backoff so the task waits for the prior run before starting.

Answer

Increase the DAG's schedule interval and add a max_active_runs limit of one so overlapping runs cannot occur.

Question 14

A financial services company runs a Dataflow streaming pipeline that reads from Pub/Sub and writes enriched records to BigQuery. Compliance requires that the raw Pub/Sub messages be retained for seven years so that any record can be reprocessed if the enrichment logic is later found to be incorrect. The pipeline currently has no archival step. What should the data engineer do to satisfy the retention requirement with the least operational overhead?

Accepted Answer

Add a Cloud Storage sink to the Dataflow pipeline that writes each raw message to a Cloud Storage bucket with a lifecycle policy moving objects to Coldline or Archive storage after 30 days.. Retaining raw messages for years requires a durable, low-cost object store, and Cloud Storage with lifecycle-based class transitions is the standard fit on Google Cloud. By branching the existing Dataflow pipeline to write unmodified messages to a bucket, the company keeps a complete, reprocessable archive without building a separate ingestion path or managing long-lived Pub/Sub backlogs.

Answer

Configure a Pub/Sub dead-letter topic and set the maximum delivery attempts high enough that failed messages persist for the retention period.

Answer

Enable BigQuery table snapshots on the destination table and set a seven-year expiration on each snapshot.

Answer

Increase the Pub/Sub topic message retention duration to the maximum allowed and rely on the subscription backlog for seven years.

Question 15

You manage a Dataflow streaming pipeline that reads from Pub/Sub and writes to BigQuery. The pipeline must be updated to add a new transformation that enriches each message with data from Cloud SQL. You need to minimize downtime and ensure the pipeline continues processing without data loss. What should you do?

Accepted Answer

Use the Dataflow update feature to replace the pipeline with the new code while preserving the pipeline state.. The Dataflow update feature is designed to replace a running pipeline with a new version while preserving state, enabling seamless updates with minimal downtime and no data loss. Stopping and restarting, creating new subscriptions, or using external triggers do not maintain pipeline state and risk data duplication or loss.

Answer

Use Cloud Scheduler to trigger a Cloud Function that updates the pipeline code in place.

Answer

Create a new Pub/Sub subscription, update the pipeline to read from it, and deploy a new pipeline.

Answer

Stop the existing pipeline, update the code, and start a new pipeline from the same subscription.