PDE Maintaining and Automating Data Workloads — 10 Questions

Question 1

Your team uses Cloud Dataproc for Spark ML training jobs. You want to reduce costs for non-critical, fault-tolerant training jobs. Which Dataproc feature should you use for worker nodes?

Accepted Answer

Use preemptible instances for worker nodes.. Preemptible instances are short-lived, lower-cost VMs that Cloud Dataproc can use for worker nodes. Because the training jobs are non-critical and fault-tolerant (e.g., they can handle node failures via Spark's built-in resilience), preemptible instances significantly reduce costs while still completing the workload. This directly addresses the requirement to reduce costs for fault-tolerant jobs.

Answer

Use custom machine types with more memory.

Answer

Use SSDs instead of HDDs for persistent disks.

Answer

Use committed use discounts for 1-year or 3-year terms.

Question 2

You have a BigQuery table with a REQUIRED column that you now need to allow NULL values. You also need to add two new nullable columns. Which TWO steps are required to achieve this schema evolution? (Choose 2)

Accepted Answer

Add the new columns using ALTER TABLE ADD COLUMN.. To achieve the schema evolution, you need to add the new nullable columns using ALTER TABLE ADD COLUMN (A) and change the existing REQUIRED column to NULLABLE using ALTER TABLE ALTER COLUMN DROP NOT NULL (the SET OPTIONS statement in option B approximates this). Exporting and recreating the table (D and E) are not required because BigQuery supports direct schema changes without data migration.

Answer

Update the IAM permissions on the table.

Answer

Use CREATE OR REPLACE TABLE with the new schema and import data.

Answer

Export the table to Cloud Storage in Avro format.

Question 3

A data engineer uses Cloud Composer to orchestrate a daily batch pipeline. A downstream task should only start after an upstream BigQuery load job finishes successfully and a specific file appears in Cloud Storage. Which combination of operators should the engineer use in the Airflow DAG?

Accepted Answer

BigQueryInsertJobOperator and GCSObjectExistenceSensor with upstream dependency. The BigQueryInsertJobOperator (or BigQueryOperator) handles the load job, and the GoogleCloudStorageObjectExistenceSensor (or GCSObjectExistenceSensor) waits for the file. Task dependencies link them.

Answer

BigQueryInsertJobOperator with wait_for_downstream=True

Answer

DataflowPythonOperator and GCSObjectExistenceSensor

Answer

BigQueryOperator and FileSensor with downstream dependency

Question 4

A company uses Dataflow streaming pipelines to process real-time events. They notice increasing system lag over time. Which two Cloud Monitoring metrics should be examined to diagnose the cause?

Accepted Answer

Dataflow job/system_lag and Dataflow job/data_freshness. System lag measures the time between event ingestion and processing. Data freshness shows the watermark. Worker CPU indicates compute resource issues.

Answer

Pub/Sub subscription/num_undelivered_messages and Dataflow job/watermark_lag

Answer

Dataproc cluster/yarn_allocated_memory_percentage and Dataflow job/worker_cpu

Answer

BigQuery query/execution_times and Dataflow job/elapsed_time

Question 5

A data team needs to share a BigQuery dataset with another business unit. They want to provide a point-in-time snapshot of the data without incurring additional storage costs for the copy. Which BigQuery feature should they use?

Accepted Answer

BigQuery table clones. Clones use the same underlying storage as the source table; snapshots also share storage but are immutable. Both are cost-effective. For regular updates, clones are more flexible.

Answer

BigQuery table snapshots

Answer

BigQuery authorized views

Answer

BigQuery export to Cloud Storage

Question 6

An engineer needs to create a reusable Dataflow pipeline that can be executed with different parameters without modifying code. Which Dataflow feature should they use?

Accepted Answer

Dataflow Flex Templates. Flex Templates allow packaging a pipeline into a Docker image with parameterization, enabling reuse across different environments.

Answer

Dataflow Shuffle

Answer

Dataflow SQL

Answer

Dataflow Classic Templates

Question 7

A company runs a Dataproc cluster for ETL jobs that process data nightly. They want to reduce costs while maintaining performance. Which strategy is MOST effective?

Accepted Answer

Use preemptible VMs for worker nodes only. Preemptible VMs are cheaper and suitable for fault-tolerant batch jobs. They can be used for worker nodes in Dataproc.

Answer

Use committed use discounts for all VMs

Answer

Enable Dataproc auto-scaling

Answer

Use preemptible VMs for all nodes including master

Question 8

A data engineer needs to alert when Pub/Sub subscription has messages older than 1 hour. Which Cloud Monitoring metric and filter should they use?

Accepted Answer

Metric: subscription/oldest_unacked_message_age; filter: subscription_id. The metric subscription/oldest_unacked_message_age gives the age of the oldest unacknowledged message. Filtering by subscription ID targets the specific subscription.

Answer

Metric: topic/send_message_operation_count; filter: topic_id

Answer

Metric: subscription/ack_message_count; filter: subscription_id

Answer

Metric: subscription/num_undelivered_messages; filter: subscription_id

Question 9

A team wants to enforce data quality rules on BigQuery tables using Dataplex. They need to run column-level checks for null values and row-level checks for value ranges on a schedule. Which Dataplex feature should they use?

Accepted Answer

Dataplex Data Quality Tasks. Dataplex Data Quality Tasks allow defining SQL-based rules (row and column) and scheduling scans.

Answer

Dataplex Data Profiling

Answer

BigQuery stored procedures with scheduled queries

Answer

Cloud DLP inspection jobs

Question 10

An organization uses BigQuery on-demand pricing. To control costs, they want to estimate the bytes processed by a query before running it. Which command or method should they use?

Accepted Answer

Use the bq query --dry_run command. BigQuery dry run estimates bytes processed without executing the query. It can be done via CLI with --dry_run flag or in the console.

Answer

Use bq ls to list table sizes

Answer

Use BigQuery reservations to get cost estimate

Answer

Use INFORMATION_SCHEMA.JOBS_BY_PROJECT to view past costs