PDE Maintaining and Automating Data Workloads — Timed 10 Questions

Question 1

A data engineer uses Cloud Composer to orchestrate a daily batch pipeline. A downstream task should only start after an upstream BigQuery load job finishes successfully and a specific file appears in Cloud Storage. Which combination of operators should the engineer use in the Airflow DAG?

Accepted Answer

BigQueryInsertJobOperator and GCSObjectExistenceSensor with upstream dependency. The BigQueryInsertJobOperator (or BigQueryOperator) handles the load job, and the GoogleCloudStorageObjectExistenceSensor (or GCSObjectExistenceSensor) waits for the file. Task dependencies link them.

Answer

BigQueryInsertJobOperator with wait_for_downstream=True

Answer

DataflowPythonOperator and GCSObjectExistenceSensor

Answer

BigQueryOperator and FileSensor with downstream dependency

Question 2

A company uses Dataflow streaming pipelines to process real-time events. They notice increasing system lag over time. Which two Cloud Monitoring metrics should be examined to diagnose the cause?

Accepted Answer

Dataflow job/system_lag and Dataflow job/data_freshness. System lag measures the time between event ingestion and processing. Data freshness shows the watermark. Worker CPU indicates compute resource issues.

Answer

Pub/Sub subscription/num_undelivered_messages and Dataflow job/watermark_lag

Answer

Dataproc cluster/yarn_allocated_memory_percentage and Dataflow job/worker_cpu

Answer

BigQuery query/execution_times and Dataflow job/elapsed_time

Question 3

A data team needs to share a BigQuery dataset with another business unit. They want to provide a point-in-time snapshot of the data without incurring additional storage costs for the copy. Which BigQuery feature should they use?

Accepted Answer

BigQuery table clones. Clones use the same underlying storage as the source table; snapshots also share storage but are immutable. Both are cost-effective. For regular updates, clones are more flexible.

Answer

BigQuery table snapshots

Answer

BigQuery authorized views

Answer

BigQuery export to Cloud Storage

Question 4

An engineer needs to create a reusable Dataflow pipeline that can be executed with different parameters without modifying code. Which Dataflow feature should they use?

Accepted Answer

Dataflow Flex Templates. Flex Templates allow packaging a pipeline into a Docker image with parameterization, enabling reuse across different environments.

Answer

Dataflow Shuffle

Answer

Dataflow SQL

Answer

Dataflow Classic Templates

Question 5

A company runs a Dataproc cluster for ETL jobs that process data nightly. They want to reduce costs while maintaining performance. Which strategy is MOST effective?

Accepted Answer

Use preemptible VMs for worker nodes only. Preemptible VMs are cheaper and suitable for fault-tolerant batch jobs. They can be used for worker nodes in Dataproc.

Answer

Use committed use discounts for all VMs

Answer

Enable Dataproc auto-scaling

Answer

Use preemptible VMs for all nodes including master

Question 6

A data engineer needs to alert when Pub/Sub subscription has messages older than 1 hour. Which Cloud Monitoring metric and filter should they use?

Accepted Answer

Metric: subscription/oldest_unacked_message_age; filter: subscription_id. The metric subscription/oldest_unacked_message_age gives the age of the oldest unacknowledged message. Filtering by subscription ID targets the specific subscription.

Answer

Metric: topic/send_message_operation_count; filter: topic_id

Answer

Metric: subscription/ack_message_count; filter: subscription_id

Answer

Metric: subscription/num_undelivered_messages; filter: subscription_id

Question 7

A team wants to enforce data quality rules on BigQuery tables using Dataplex. They need to run column-level checks for null values and row-level checks for value ranges on a schedule. Which Dataplex feature should they use?

Accepted Answer

Dataplex Data Quality Tasks. Dataplex Data Quality Tasks allow defining SQL-based rules (row and column) and scheduling scans.

Answer

Dataplex Data Profiling

Answer

BigQuery stored procedures with scheduled queries

Answer

Cloud DLP inspection jobs

Question 8

An organization uses BigQuery on-demand pricing. To control costs, they want to estimate the bytes processed by a query before running it. Which command or method should they use?

Accepted Answer

Use the bq query --dry_run command. BigQuery dry run estimates bytes processed without executing the query. It can be done via CLI with --dry_run flag or in the console.

Answer

Use bq ls to list table sizes

Answer

Use BigQuery reservations to get cost estimate

Answer

Use INFORMATION_SCHEMA.JOBS_BY_PROJECT to view past costs

Question 9

A company uses Cloud Composer for pipeline orchestration. They need to define task dependencies where Task B and Task C can run in parallel after Task A, and Task D must run after both B and C complete. How should they define the DAG?

Accepted Answer

A >> [B, C] >> D. Using bitshift operators: A >> [B, C] >> D sets B and C after A, and D after both B and C complete.

Answer

A >> B; B >> D; A >> C; C >> D

Answer

A >> B >> C >> D

Answer

A.set_downstream(B); B.set_upstream(C); C.set_downstream(D)

Question 10

A streaming Dataflow pipeline needs to be updated without draining the existing pipeline. Which update strategy should be used?

Accepted Answer

Replace the job with a new job using the same pipeline name. Dataflow Streaming Engine allows updates without draining by using the same pipeline name and enabling Streaming Engine.

Answer

Drain the pipeline first, then start a new one

Answer

Use a different pipeline name and cancel the old one

Answer

Stop the job, update the code, and restart