PDE Maintaining and Automating Data Workloads Practice Test 3 — 15 Questions

Question 1

You have a BigQuery table with a REQUIRED column that you now need to allow NULL values. You also need to add two new nullable columns. Which TWO steps are required to achieve this schema evolution? (Choose 2)

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Add the new columns using ALTER TABLE ADD COLUMN.. To achieve the schema evolution, you need to add the new nullable columns using ALTER TABLE ADD COLUMN (A) and change the existing REQUIRED column to NULLABLE using ALTER TABLE ALTER COLUMN DROP NOT NULL (the SET OPTIONS statement in option B approximates this). Exporting and recreating the table (D and E) are not required because BigQuery supports direct schema changes without data migration.

Answer

Update the IAM permissions on the table.

Answer

Use CREATE OR REPLACE TABLE with the new schema and import data.

Answer

Export the table to Cloud Storage in Avro format.

Question 2

Your team runs a nightly Cloud Dataflow batch pipeline that reads from Cloud Storage, transforms data, and writes to BigQuery. The pipeline must execute after an upstream Cloud Composer DAG completes. You need to ensure the Dataflow job runs only after the DAG finishes and that the pipeline uses the latest template version stored in Cloud Storage. What should you do?

Accepted Answer

Use the Cloud Composer DataflowTemplateOperator with the template location in Cloud Storage, and set the upstream task dependency to ensure it runs after the DAG completes.. The DataflowTemplateOperator is the native Cloud Composer operator for launching Dataflow jobs from templates stored in Cloud Storage. Setting the upstream task dependency ensures the job starts only after the DAG completes. This approach is straightforward, requires no custom code, and uses the latest template version by referencing the Cloud Storage path. Other options either use incompatible operators or introduce unnecessary external orchestration.

Answer

Use the Cloud Composer DataflowStartFlexTemplateOperator with a trigger rule of all_success on the upstream tasks.

Answer

Create a Cloud Scheduler job that triggers the Dataflow template via the Dataflow API, and use the DAG's completion to notify Cloud Scheduler.

Answer

Configure a Cloud Function that listens for a Pub/Sub message published by the DAG upon completion, and have the function start the Dataflow job using the template.

Question 3

Your team runs a Dataflow streaming pipeline that reads from Pub/Sub and writes to BigQuery. You need to deploy a new version of the pipeline with a modified transformation. You want to minimize disruption to the running pipeline and ensure that the new version starts processing new messages immediately. Which deployment approach should you use?

Accepted Answer

Stop the pipeline with the drain option, then start a new pipeline with the updated code.. Draining the existing pipeline allows it to finish processing in-flight messages and then stop, preventing data loss. Starting a new pipeline with updated code ensures that new messages are processed with the new logic. This approach minimizes disruption and is the standard method for updating streaming pipelines in Dataflow.

Answer

Update the pipeline in place using the gcloud dataflow jobs update command.

Answer

Use the update transform in Apache Beam to replace the transformation logic dynamically.

Answer

Create a new pipeline that reads from the same Pub/Sub subscription and writes to a new BigQuery table, then switch the subscription.

Question 4

A retail company runs a Cloud Composer (Airflow) DAG that ingests files from Cloud Storage into BigQuery every night. The DAG has failed three nights in a row with a task-level error on the load task, but the engineer only learns about it after business users complain. The team wants to be automatically notified by email whenever any task in any DAG in the environment enters a failed state, with minimal ongoing effort. What should the engineer do?

Accepted Answer

Enable Cloud Composer's built-in email alerting by configuring the environment's email notification settings for task failures, so all DAGs inherit it automatically.. Cloud Composer environments expose native alerting for task failures that applies across all DAGs in the environment, so a single configuration delivers email notifications for the nightly load failure and any future DAG. Building per-task callbacks or custom polling infrastructure adds maintenance burden and gaps in coverage that the environment-level setting avoids.

Answer

Schedule a Cloud Scheduler job that queries the Airflow REST API every five minutes for task instances in the failed state and publishes a Pub/Sub message that a Cloud Function converts into email.

Answer

Configure an Airflow email alerting callback by adding on_failure_callback to every task in every DAG, referencing an SMTP connection in the Composer environment.

Answer

Create a Cloud Monitoring alerting policy on the Composer environment's airflow-worker log-based metric for task failures, with an email notification channel attached.

Question 5

You are using Cloud Dataproc to run a Spark job that processes data stored in Cloud Storage. You want to enable autoscaling for the cluster to handle variable workloads. Which two configurations are required to enable autoscaling? (Choose two.)

Accepted Answer

Specify an autoscaling policy when creating the cluster.. To enable autoscaling in Dataproc, you must create an autoscaling policy and attach it to the cluster. Additionally, the cluster must use a Dataproc image version that supports autoscaling, such as 1.4 or later. Other options like using preemptible VMs or setting a specific minimum worker count are not required for autoscaling to function.

Answer

Set the cluster's worker nodes to use preemptible VMs.

Answer

Enable the Dataproc Autoscaling API in the project.

Answer

Configure the cluster with a minimum of two worker nodes.

Question 6

You manage a Dataflow streaming pipeline that reads from Pub/Sub and writes to BigQuery. You need to update the pipeline with a new transformation that changes the output schema of the BigQuery table. The pipeline is currently running in production and must not lose data during the update. Which update strategy should you use?

Accepted Answer

Use the --update option with the new pipeline code, ensuring the job name and pipeline options are compatible.. The correct approach is to use the --update option with the new pipeline code. Dataflow's update feature allows you to modify a running streaming pipeline while preserving its state and ensuring no data loss. It is specifically designed for scenarios where you need to change transformation logic or output schema without stopping the pipeline. This maintains exactly-once processing and avoids downtime, making it the ideal choice for production streaming updates.

Answer

Stop the pipeline, update the code, and restart with the same job name.

Answer

Create a new pipeline with a different job name and run it in parallel, then delete the old pipeline after a grace period.

Answer

Drain the pipeline, then start a new pipeline with the updated code.

Question 7

A retail company runs a Dataflow streaming pipeline that reads events from Pub/Sub and writes enriched records into BigQuery. Over a weekend the on-call engineer receives an alert that the pipeline's system lag keeps climbing and never drains, even though the upstream publisher volume is unchanged. The pipeline uses a fixed 10-second window with a 5-minute allowed lateness and writes to BigQuery with no load job frequency override. What is the MOST likely cause of the sustained system lag?

Accepted Answer

A hot key in the GroupByKey used for enrichment causes one worker to process a disproportionate share of elements, so the watermark advances only as fast as that single slow shard.. System lag tracks the watermark's progress relative to the newest element in the stream. With a stable input rate, the most likely explanation for lag that never drains is uneven work distribution: a hot key in a GroupByKey forces one shard to process far more elements than its peers, and downstream windowing cannot close until that shard finishes. Increasing parallelism, pre-aggregating, or using a combiner with a key-splitting approach would address the skew.

Answer

The BigQuery Storage Write API is throttling inserts because the number of distinct streaming inserts exceeds the table's quota, so the pipeline cannot commit results fast enough.

Answer

The 5-minute allowed lateness setting is too large, so the watermark never advances and system lag grows without bound.

Answer

The Pub/Sub subscription's acknowledgement deadline is shorter than the pipeline's processing time, causing messages to be redelivered indefinitely and inflating system lag.

Question 8

A data engineer needs to automate a BigQuery ELT workflow so that a scheduled SQL transformation runs only when new data lands in a Cloud Storage bucket, and so that failures notify the on-call team. The team already uses Cloud Composer. Which two approaches satisfy these requirements? (Choose two.)

Accepted Answer

Create a Cloud Storage-triggered Cloud Function that publishes to a Pub/Sub topic, and use a Pub/Sub sensor or the Pub/Sub-triggered DAG pattern in Cloud Composer to start the transformation.. Event-driven orchestration requires a signal that data actually arrived, which a Cloud Storage-to-Pub/Sub path into a Composer DAG provides. Alerting requires an SLA or failure callback that reaches the on-call team. Combining the event trigger with SLA-based notification satisfies both stated requirements while reusing the existing Composer environment.

Answer

Enable BigQuery's scheduled queries feature to run the transformation on a cron schedule and configure Cloud Logging to email the team.

Answer

Configure the DAG with a BigQueryInsertJobOperator scheduled on a frequent interval and rely on the SQL itself to skip rows that were already processed.

Answer

Use a Cloud Composer DAG with a TimeSensor that waits until a fixed hour each day before running the transformation.

Question 9

Your team runs a Cloud Composer 2 environment (Airflow 2.6) to orchestrate BigQuery ELT. A nightly DAG ingests data from Cloud Storage, runs a BigQuery load job, then triggers a Dataflow pipeline. The DAG recently started failing intermittently at the BigQuery load task with 'Quota exceeded: Your project exceeded quota for imports per project'. You need to reduce the frequency of these failures without changing the DAG's schedule. What should you do?

Accepted Answer

Add a retry policy with exponential backoff to the BigQuery load task in the DAG.. The 'imports per project' quota error is typically transient and resolves when the quota window resets. Adding retries with exponential backoff to the BigQuery load task allows the DAG to recover automatically without changing the schedule or architecture. Increasing slots, switching to Dataflow, or using multiple projects do not directly address the transient quota and introduce unnecessary complexity or cost.

Answer

Switch the load task from a BigQuery load job to a Dataflow pipeline that writes directly to BigQuery.

Answer

Configure the DAG to use a different Google Cloud project for each load job to distribute the quota.

Answer

Increase the BigQuery slot reservations for the project to raise the import quota.

Question 10

You are designing a data pipeline that uses Cloud Dataflow to process streaming data from Pub/Sub and write to BigQuery. You need to ensure that the pipeline handles late-arriving data correctly and provides exactly-once processing semantics. Which two actions should you take? (Choose two.)

Accepted Answer

Configure the pipeline to use windowing with allowed lateness and trigger to handle late data.. To handle late-arriving data, you need windowing with allowed lateness and triggers to define when to emit results. For exactly-once processing to BigQuery, use streaming inserts with insertId for deduplication. The streaming engine and Dataflow Shuffle do not address these requirements, and WRITE_APPEND can cause duplicates.

Answer

Enable the Dataflow streaming engine to improve performance and resource utilization.

Answer

Set the BigQuery write disposition to WRITE_APPEND to ensure all data is written.

Answer

Enable Dataflow Shuffle to improve performance for batch pipelines.

Question 11

You operate a Cloud Composer 2 environment whose DAGs trigger BigQuery jobs. A new compliance rule requires that every DAG run's BigQuery queries execute under a dedicated identity with its own audit trail, separate from the Composer environment's service account. Which configuration should you apply?

Accepted Answer

Set the `impersonation_chain` parameter on the Airflow operators so they assume the dedicated identity when calling BigQuery.. Airflow's Google provider operators accept an `impersonation_chain` argument that obtains short-lived tokens for a target service account via the IAM Credentials API. Supplying the dedicated identity there causes BigQuery jobs to be attributed to that account in audit logs, while the environment's service account remains responsible only for orchestration. This separates duties without rebuilding the environment or handling static keys.

Answer

Create a BigQuery connection and reference it in the DAG's `BigQueryInsertJobOperator` using the `impersonation_chain` parameter.

Answer

Recreate the Cloud Composer environment with the dedicated identity set as the environment's service account.

Answer

Grant the dedicated identity `roles/composer.worker` and add it as an environment variable named `GOOGLE_APPLICATION_CREDENTIALS`.

Question 12

A logistics company has a Cloud Composer DAG with a task that calls a third-party routing API. The API occasionally returns HTTP 503 during peak hours, and the DAG then fails the whole run, forcing a manual restart. The engineer wants the task to automatically retry transient 503 responses with increasing delays, but not to retry on a 401 authentication error, which indicates a misconfigured credential that must be fixed by a human. What should the engineer do?

Accepted Answer

Enable retries on the task and use a custom exception with AirflowFailException for non-retryable HTTP statuses such as 401.. AirflowFailException marks a task as failed immediately without consuming configured retries, so raising it for a 401 stops the run and surfaces the credential problem to a human. Allowing ordinary failures for 503 responses lets the task's retries and retry_delay apply exponential-style backoff to transient errors, matching both halves of the requirement.

Answer

Set retries and retry_delay on the task, and raise AirflowException inside the operator only when the HTTP status is not 503.

Answer

Set max_active_tis_per_dag to a high value and rely on the scheduler to re-run the task when it fails.

Answer

Wrap the API call in a PythonOperator loop that calls the API repeatedly until it succeeds, ignoring HTTP status codes.

Question 13

You have a Dataflow batch pipeline that processes a large dataset stored in Cloud Storage and writes results to BigQuery. The pipeline uses a custom DoFn that makes API calls to an external service. You notice that the pipeline is running slowly and some workers are idle while others are overloaded. You want to improve performance and ensure even distribution of work. What should you do?

Accepted Answer

Implement a custom partitioner or use a shuffle to evenly distribute the work across workers.. Uneven work distribution in a Dataflow pipeline is often caused by data skew or hot keys. Implementing a custom partitioner or using a shuffle to evenly distribute the work ensures that all workers are utilized efficiently. Increasing resources or enabling autoscaling does not address the root cause of the imbalance.

Answer

Enable autoscaling and set a maximum number of workers.

Answer

Use a GroupByKey followed by a ParDo to redistribute the work.

Answer

Increase the number of workers and use a larger machine type.

Question 14

Your company uses Cloud Composer to orchestrate a data pipeline that loads data into BigQuery. A new compliance requirement mandates that all data access be logged and that any changes to the DAGs be auditable. You need to ensure that DAG file changes are tracked and that you can identify who modified a DAG and when. What should you do?

Accepted Answer

Store DAG files in a Cloud Storage bucket with Object Versioning enabled and enable Cloud Audit Logs for the bucket.. DAG files in Cloud Composer are stored in a Cloud Storage bucket. Enabling Object Versioning on that bucket preserves historical versions of DAG files, and enabling Cloud Audit Logs for the bucket records who accessed or modified the objects. Together, these features provide a robust audit trail for DAG changes, meeting compliance requirements without custom code.

Answer

Enable Cloud Audit Logs for the Cloud Composer environment and review the logs for DAG modifications.

Answer

Use Airflow's built-in audit log in the Cloud Composer web UI to track DAG changes.

Answer

Configure a Cloud Function that triggers on DAG file changes and writes an entry to Cloud Logging.

Question 15

You are designing automated monitoring for a Cloud Composer 2 environment that runs critical ingestion DAGs. You need to detect conditions that indicate the scheduler or workers are unhealthy before DAG runs silently stop. Which two Cloud Monitoring metrics should you alert on? (Choose two.)

Accepted Answer

`composer.googleapis.com/environment/healthy`. Silent DAG stoppage usually traces to the scheduler or the Airflow metadata database rather than to task-level resources. The environment health metric captures overall component status, and the database health metric catches the backend failure that halts scheduling. Together they detect the unhealthy states that prevent runs from starting, unlike parse time or worker memory metrics that describe performance rather than availability.

Answer

`composer.googleapis.com/environment/dag_processing/total_parse_time`

Answer

`composer.googleapis.com/environment/worker/used_memory_bytes`

Answer

`composer.googleapis.com/environment/num_celery_workers`