PDE Maintaining and Automating Data Workloads — 20 Questions

Question 1

A data engineer uses Cloud Composer to orchestrate a daily batch pipeline. A downstream task should only start after an upstream BigQuery load job finishes successfully and a specific file appears in Cloud Storage. Which combination of operators should the engineer use in the Airflow DAG?

Accepted Answer

BigQueryInsertJobOperator and GCSObjectExistenceSensor with upstream dependency. The BigQueryInsertJobOperator (or BigQueryOperator) handles the load job, and the GoogleCloudStorageObjectExistenceSensor (or GCSObjectExistenceSensor) waits for the file. Task dependencies link them.

Answer

BigQueryInsertJobOperator with wait_for_downstream=True

Answer

DataflowPythonOperator and GCSObjectExistenceSensor

Answer

BigQueryOperator and FileSensor with downstream dependency

Question 2

A company uses Dataflow streaming pipelines to process real-time events. They notice increasing system lag over time. Which two Cloud Monitoring metrics should be examined to diagnose the cause?

Accepted Answer

Dataflow job/system_lag and Dataflow job/data_freshness. System lag measures the time between event ingestion and processing. Data freshness shows the watermark. Worker CPU indicates compute resource issues.

Answer

Pub/Sub subscription/num_undelivered_messages and Dataflow job/watermark_lag

Answer

Dataproc cluster/yarn_allocated_memory_percentage and Dataflow job/worker_cpu

Answer

BigQuery query/execution_times and Dataflow job/elapsed_time

Question 3

A data team needs to share a BigQuery dataset with another business unit. They want to provide a point-in-time snapshot of the data without incurring additional storage costs for the copy. Which BigQuery feature should they use?

Accepted Answer

BigQuery table clones. Clones use the same underlying storage as the source table; snapshots also share storage but are immutable. Both are cost-effective. For regular updates, clones are more flexible.

Answer

BigQuery table snapshots

Answer

BigQuery authorized views

Answer

BigQuery export to Cloud Storage

Question 4

An engineer needs to create a reusable Dataflow pipeline that can be executed with different parameters without modifying code. Which Dataflow feature should they use?

Accepted Answer

Dataflow Flex Templates. Flex Templates allow packaging a pipeline into a Docker image with parameterization, enabling reuse across different environments.

Answer

Dataflow Shuffle

Answer

Dataflow SQL

Answer

Dataflow Classic Templates

Question 5

A company runs a Dataproc cluster for ETL jobs that process data nightly. They want to reduce costs while maintaining performance. Which strategy is MOST effective?

Accepted Answer

Use preemptible VMs for worker nodes only. Preemptible VMs are cheaper and suitable for fault-tolerant batch jobs. They can be used for worker nodes in Dataproc.

Answer

Use committed use discounts for all VMs

Answer

Enable Dataproc auto-scaling

Answer

Use preemptible VMs for all nodes including master

Question 6

A data engineer needs to alert when Pub/Sub subscription has messages older than 1 hour. Which Cloud Monitoring metric and filter should they use?

Accepted Answer

Metric: subscription/oldest_unacked_message_age; filter: subscription_id. The metric subscription/oldest_unacked_message_age gives the age of the oldest unacknowledged message. Filtering by subscription ID targets the specific subscription.

Answer

Metric: topic/send_message_operation_count; filter: topic_id

Answer

Metric: subscription/ack_message_count; filter: subscription_id

Answer

Metric: subscription/num_undelivered_messages; filter: subscription_id

Question 7

A team wants to enforce data quality rules on BigQuery tables using Dataplex. They need to run column-level checks for null values and row-level checks for value ranges on a schedule. Which Dataplex feature should they use?

Accepted Answer

Dataplex Data Quality Tasks. Dataplex Data Quality Tasks allow defining SQL-based rules (row and column) and scheduling scans.

Answer

Dataplex Data Profiling

Answer

BigQuery stored procedures with scheduled queries

Answer

Cloud DLP inspection jobs

Question 8

An organization uses BigQuery on-demand pricing. To control costs, they want to estimate the bytes processed by a query before running it. Which command or method should they use?

Accepted Answer

Use the bq query --dry_run command. BigQuery dry run estimates bytes processed without executing the query. It can be done via CLI with --dry_run flag or in the console.

Answer

Use bq ls to list table sizes

Answer

Use BigQuery reservations to get cost estimate

Answer

Use INFORMATION_SCHEMA.JOBS_BY_PROJECT to view past costs

Question 9

A company uses Cloud Composer for pipeline orchestration. They need to define task dependencies where Task B and Task C can run in parallel after Task A, and Task D must run after both B and C complete. How should they define the DAG?

Accepted Answer

A >> [B, C] >> D. Using bitshift operators: A >> [B, C] >> D sets B and C after A, and D after both B and C complete.

Answer

A >> B; B >> D; A >> C; C >> D

Answer

A >> B >> C >> D

Answer

A.set_downstream(B); B.set_upstream(C); C.set_downstream(D)

Question 10

A streaming Dataflow pipeline needs to be updated without draining the existing pipeline. Which update strategy should be used?

Accepted Answer

Replace the job with a new job using the same pipeline name. Dataflow Streaming Engine allows updates without draining by using the same pipeline name and enabling Streaming Engine.

Answer

Drain the pipeline first, then start a new one

Answer

Use a different pipeline name and cancel the old one

Answer

Stop the job, update the code, and restart

Question 11

A company wants to use Cloud DLP to inspect data in BigQuery for sensitive information and de-identify it by masking credit card numbers. They want to perform this on a schedule. Which approach should they take?

Accepted Answer

Use Cloud DLP inspect and de-identify jobs triggered by Cloud Scheduler. Cloud DLP can inspect BigQuery tables and de-identify using transforms like masking. Scheduling can be done via Cloud Scheduler.

Answer

Use Dataplex data quality rules with a custom SQL regex

Answer

Use Cloud Data Loss Prevention API with Cloud Composer

Answer

Use BigQuery column-level security with classification

Question 12

A data engineer notices that BigQuery queries are slower than expected. They want to identify the most expensive stages in the query execution. Which tool or command should they use?

Accepted Answer

Use EXPLAIN statement in BigQuery. EXPLAIN provides query plan details including costs per stage. It can be run via bq or in the console.

Answer

Use bq show to view job statistics

Answer

Use bq query --format=prettyjson and look at statistics

Answer

Use Cloud Monitoring to view query execution graphs

Question 13

A data engineer needs to migrate a schema from BigQuery where a column is currently REQUIRED and needs to become NULLABLE. Which TWO statements are correct? (Choose 2)

Accepted Answer

Drop the column and add it again as NULLABLE. BigQuery does not allow changing a column from REQUIRED to NULLABLE directly. One must either drop and recreate the column or use a query to create a new table.

Answer

Use ALTER TABLE RENAME COLUMN and then add new column

Answer

Use bq update --schema to change the mode

Answer

Use ALTER TABLE ALTER COLUMN SET DATA TYPE to change to NULLABLE

Question 14

A company runs BigQuery workloads with varying demand. They want to use flat-rate pricing with baseline slots and the ability to burst during peak times. Which TWO actions should they take? (Choose 2)

Accepted Answer

Use flex slots for short-term bursts. Committed use reservations provide baseline slots at a discount. Flex slots allow bursting for short periods without long-term commitment.

Answer

Use on-demand pricing for bursting

Answer

Set a maximum number of slots per query

Answer

Create a reservation with baseline + autoscaling slots

Question 15

A company uses Cloud Composer (Airflow) to orchestrate pipelines. They want to implement a pattern where a task polls for a file arrival in Cloud Storage and then triggers subsequent tasks. Which THREE Airflow concepts are essential? (Choose 3)

Accepted Answer

Sensors (e.g., GCSObjectExistenceSensor). A is correct because Sensors are a specialized type of operator designed to wait for a specific condition, such as file arrival in Cloud Storage. The `GCSObjectExistenceSensor` in Cloud Composer (Airflow) polls Google Cloud Storage at a configurable interval until the target file exists, making it the precise tool for this file-arrival polling pattern.

Answer

Operators (e.g., PythonOperator)

Answer

SubDAGs for grouping tasks

Question 16

A data engineer is building a batch pipeline that runs daily using Cloud Composer. The pipeline has three tasks: extract data from Cloud Storage, transform data using Dataflow, and load the transformed data into BigQuery. The engineer wants to ensure that the Dataflow job only starts after the extraction task completes successfully, and the load task only starts after the Dataflow job finishes. How should the engineer define the task dependencies in the Airflow DAG?

Accepted Answer

extract >> transform >> load. Option C is correct because Airflow uses the bitshift operator (>>) to define task dependencies in a linear sequence. The DAG must ensure that the extract task completes before the transform task starts, and the transform task completes before the load task starts. This is achieved by chaining the tasks in order: extract >> transform >> load, which enforces the required sequential execution.

Answer

extract >> [transform, load]

Answer

transform >> extract >> load

Answer

extract >> load >> transform

Question 17

You need to schedule a simple workflow that fetches data from an API every hour, transforms it using Cloud Functions, and writes the result to Cloud Storage. The workflow has no complex branching or retry logic beyond basic retries. Which orchestration service is the MOST cost-effective and simplest to implement?

Accepted Answer

Workflows. Workflows is serverless, pay-per-execution, and defined in YAML/JSON. It integrates natively with Cloud Functions and Cloud Storage. Cloud Composer is overkill for simple linear workflows and incurs cluster costs. Cloud Scheduler alone cannot orchestrate multiple steps. Dataflow is for data processing, not orchestration.

Answer

Cloud Scheduler

Answer

Cloud Composer

Answer

Dataflow

Question 18

A company runs a streaming Dataflow pipeline that reads from Pub/Sub, enriches data with a side input from BigQuery, and writes to BigQuery. After updating the pipeline code (adding a new field to the output), the engineer notices that the new pipeline version is not picking up the updated code because the job was started from a template. The engineer wants to update the streaming pipeline without draining it. What should the engineer do?

Accepted Answer

Use the gcloud dataflow jobs update command with the new Flex Template.. Option A is correct because the `gcloud dataflow jobs update` command allows you to update a running streaming Dataflow pipeline with a new Flex Template without draining or stopping the job. This command performs an in-place update, preserving the job's state and checkpointing, so the pipeline continues processing with the new code. Since the original job was started from a template, using this command with the new Flex Template ensures the updated code is picked up seamlessly.

Answer

Stop the pipeline, update the template, and restart with the same job name.

Answer

Modify the original template and redeploy it as a new job with the same pipeline name.

Answer

Use the gcloud dataflow jobs drain command, then restart with the new template.

Question 19

You are monitoring a streaming Dataflow pipeline that reads from Pub/Sub and writes to BigQuery. In Cloud Monitoring, you notice that the 'system_lag' metric is increasing over time and now exceeds 10 minutes. The 'data_watermark' metric shows a steady lag. What is the most likely cause of the increasing system lag?

Accepted Answer

The Dataflow pipeline is underprovisioned with workers, causing processing backlog.. Option D is correct because an underprovisioned pipeline lacks sufficient worker resources to process incoming messages at the rate they arrive. This causes a growing backlog in the pipeline's internal buffers, which directly increases the 'system_lag' metric (the time between data ingestion and processing). The 'data_watermark' lag remaining steady indicates that the pipeline is still making progress on event-time processing, but the overall processing capacity is insufficient to keep up with the input rate.

Answer

BigQuery write throughput is throttling the pipeline.

Answer

The Pub/Sub subscription has too many unacknowledged messages.

Answer

The pipeline is using a global window with late data handling.

Question 20

A company wants to share a large BigQuery dataset with a partner for analysis. The partner needs read-only access to a specific snapshot of the data as of a certain point in time, and the company wants to avoid additional storage costs for the partner. What is the most cost-effective approach?

Accepted Answer

Create a BigQuery table clone at the desired point in time and share it with the partner.. BigQuery table clones are zero-copy clones that share the underlying storage with the base table. They do not incur additional storage costs until the data in the clone is modified. Snapshots incur storage costs for the snapshot. Authorized views or datasets require the partner to query the base table, which may incur analysis costs but no extra storage; however, the partner may see changes to the base table. The question specifies a point-in-time snapshot, so a clone is best.

Answer

Create a BigQuery table snapshot at the desired point in time and share it.

Answer

Export the table to Cloud Storage as Avro and share a signed URL.

Answer

Grant the partner access to the original table with an authorized view.