PDE Maintaining and Automating Data Workloads Practice Test 4 — 15 Questions

Question 1

You are building a data pipeline that runs daily batch jobs on Dataproc, then loads results into BigQuery. You want to orchestrate the entire workflow, including dependencies between steps, retries, and monitoring. Which Google Cloud service is most appropriate?

Accepted Answer

Cloud Composer. Cloud Composer is a managed Apache Airflow service that provides DAG-based orchestration with rich operators for Dataproc, BigQuery, and other GCP services. It handles dependencies, retries, and monitoring out of the box. Workflows is simpler and serverless but lacks the extensive operator library and scheduling flexibility of Airflow.

Answer

Cloud Scheduler

Answer

Cloud Workflows

Answer

Dataflow

Question 2

Your streaming Dataflow pipeline reads from Pub/Sub, enriches data with a side input, and writes to BigQuery. You need to update the enrichment logic without draining the pipeline, to minimize data loss and maintain exactly-once semantics. What should you do?

Accepted Answer

Use the Dataflow job update mechanism to replace the pipeline with a new version.. Option C is correct because the Dataflow job update mechanism allows you to replace a running pipeline's code with a new version without draining or stopping it, preserving the existing state and minimizing data loss. This mechanism supports exactly-once semantics by ensuring that all in-flight elements are processed exactly once, even after the update, by maintaining the pipeline's checkpoint and watermark state.

Answer

Cancel the pipeline and create a new one with the updated code.

Answer

Stop the pipeline, update the code, and restart from the latest snapshot.

Answer

Drain the pipeline, update the code, and restart with the same job ID.

Question 3

You manage a BigQuery reservation with 500 baseline slots and autoscaling up to 2000 slots. Your team runs a mix of interactive queries and batch load jobs. During peak hours, you notice that interactive queries are throttled when autoscaling slots are consumed by long-running batch loads. How can you ensure interactive queries get priority access to slots?

Accepted Answer

Create a separate reservation for interactive queries with a higher priority assignment.. Option A is correct because BigQuery reservations allow you to create separate reservations for different workloads (e.g., interactive queries vs. batch loads) and assign them different priority levels. By creating a dedicated reservation for interactive queries with a higher priority, you ensure that interactive queries get preferential access to slots, even when autoscaling slots are consumed by long-running batch jobs. This directly addresses the contention issue without reducing overall capacity.

Answer

Reduce the baseline slots to 200 and rely solely on autoscaling.

Answer

Switch to on-demand pricing to eliminate slot contention.

Answer

Set the autoscaling max to 1000 slots for batch jobs.

Question 4

You want to monitor the latency of messages in a Pub/Sub subscription. Which Cloud Monitoring metric should you use to see the age of the oldest unacknowledged message?

Accepted Answer

pubsub.googleapis.com/subscription/oldest_unacked_message_age. The metric 'subscription/oldest_unacked_message_age' measures the age (in seconds) of the oldest unacknowledged message in a subscription. This helps track processing lag. The other metrics measure different aspects: num_undelivered_messages is count, not age; topic metrics are irrelevant for subscription lag; publish_latency is about publishing, not consumption.

Answer

pubsub.googleapis.com/subscription/num_undelivered_messages

Answer

pubsub.googleapis.com/topic/send_request_count

Answer

pubsub.googleapis.com/topic/publish_latency

Question 5

You need to create a reusable Dataflow pipeline for transforming CSV files in Cloud Storage into Avro files in another bucket. The pipeline should be configurable via runtime parameters (e.g., input and output paths). Which approach should you use?

Accepted Answer

Create a Dataflow Flex Template with a Docker image and parameterized metadata.. Option D is correct because Dataflow Flex Templates allow you to package your pipeline code and dependencies into a Docker image, and define parameterized metadata (e.g., input and output paths) that are exposed as runtime parameters. This approach provides full customization of the execution environment and supports reusable, configurable pipelines for transforming CSV to Avro across different Cloud Storage buckets.

Answer

Use Cloud Functions triggered by Cloud Storage events.

Answer

Create a Dataflow Classic Template with the pipeline code and parameters.

Answer

Use Cloud Run Jobs to run the transformation as a container.

Question 6

Your team uses Cloud Dataproc for Spark ML training jobs. You want to reduce costs for non-critical, fault-tolerant training jobs. Which Dataproc feature should you use for worker nodes?

Accepted Answer

Use preemptible instances for worker nodes.. Preemptible instances are short-lived, lower-cost VMs that Cloud Dataproc can use for worker nodes. Because the training jobs are non-critical and fault-tolerant (e.g., they can handle node failures via Spark's built-in resilience), preemptible instances significantly reduce costs while still completing the workload. This directly addresses the requirement to reduce costs for fault-tolerant jobs.

Answer

Use custom machine types with more memory.

Answer

Use SSDs instead of HDDs for persistent disks.

Answer

Use committed use discounts for 1-year or 3-year terms.

Question 7

A BigQuery table has a REQUIRED column 'user_id' that now needs to accept NULL values due to upstream data changes. You want to alter the schema with minimal downtime and no data loss. What should you do?

Accepted Answer

Run `ALTER TABLE dataset.table ALTER COLUMN user_id DROP NOT NULL;`. BigQuery allows changing a column from REQUIRED to NULLABLE using the ALTER TABLE ALTER COLUMN SET DATA TYPE statement. This operation is a metadata change and does not require table recreation or data copy. Dropping and recreating the table would cause downtime and data loss. Using a view is a workaround but doesn't change the underlying schema. Exporting and reloading is disruptive.

Answer

Use the bq command: `bq update --set_nullable_fields user_id dataset.table`

Answer

Create a view that casts user_id to NULLABLE and use the view instead.

Answer

Drop the table and recreate it with the column as NULLABLE.

Question 8

You need to estimate the cost of a BigQuery query before running it. Which command or feature should you use?

Accepted Answer

Use the bq command with the --dry_run flag.. Option D is correct because the `bq` command with the `--dry_run` flag allows you to estimate the amount of data a BigQuery query will process before actually executing it. This dry run does not read any data or incur charges; it simply returns the estimated bytes to be processed, which you can use to calculate the cost based on BigQuery's pricing model.

Answer

Check the BigQuery jobs list for similar queries.

Answer

Use the BigQuery cache to estimate if the query is cached.

Answer

Run EXPLAIN on the query to see the query plan.

Question 9

You are designing a data quality pipeline that must inspect PII in BigQuery tables and de-identify sensitive columns before sharing with analysts. Which GCP service should you use?

Accepted Answer

Cloud DLP. Cloud DLP (Data Loss Prevention) is the correct choice because it is purpose-built for inspecting, classifying, and de-identifying sensitive data such as PII. It integrates natively with BigQuery via inspection jobs and de-identification templates, allowing you to scan tables for over 150 built-in infoTypes (e.g., email, SSN) and apply transformations like masking, tokenization, or encryption before sharing data with analysts.

Answer

Dataplex

Answer

Cloud Data Catalog

Answer

Dataflow

Question 10

Your Dataflow streaming job is experiencing high system lag. You want to identify the root cause. Which Cloud Monitoring metrics should you examine first? (Choose the best option.)

Accepted Answer

System lag and worker CPU utilization. For streaming Dataflow jobs, system lag measures the maximum time between the event timestamp and when it is processed. High system lag typically indicates that the pipeline cannot keep up with the input rate. Worker CPU utilization is a key metric to check if workers are overloaded. Data freshness is for batch pipelines. Element count alone doesn't show lag. Backlog bytes could be useful but worker CPU is more directly indicative of processing capacity issues.

Answer

Data freshness and worker CPU utilization

Answer

Element count and data freshness

Answer

Backlog bytes and system lag

Question 11

You need to schedule a Dataproc Spark job to run at 2 AM every day, and upon completion, trigger a BigQuery load job. Which Cloud Composer operator should you use to run the Spark job?

Accepted Answer

DataprocSubmitJobOperator. Option D is correct because the DataprocSubmitJobOperator is specifically designed to submit a job (e.g., a Spark job) to an existing Dataproc cluster. In this scenario, you need to run a Spark job on a scheduled basis, and Cloud Composer (Airflow) provides this operator to submit the job to Dataproc. After the Spark job completes, you can chain a BigQuery load operator to trigger the load, matching the requirement exactly.

Answer

DataflowPythonOperator

Answer

BigQueryOperator

Answer

DataprocClusterCreateOperator

Question 12

You want to create a cost-efficient snapshot of a large BigQuery table that can be used by other teams for read-only analytics without incurring additional storage costs for the base table data. What should you use?

Accepted Answer

Create a BigQuery table clone of the original table.. Option D is correct because a BigQuery table clone creates a read-only, cost-efficient copy of the table that references the underlying storage of the base table, so no additional storage costs are incurred for the base data. Clones are ideal for sharing snapshots for read-only analytics without duplicating storage, and they support time-travel queries within the clone's retention period.

Answer

Create a BigQuery table snapshot of the original table.

Answer

Export the table to Cloud Storage as Avro files and load into a new table.

Answer

Create a view over the original table.

Question 13

Your company uses Cloud Composer to orchestrate a data pipeline that includes Dataproc Spark jobs and BigQuery load operations. You need to pass the output file path from the Spark job to the next BigQuery task in the DAG. Which two mechanisms can you use to share data between tasks? (Choose TWO.)

Accepted Answer

Write the output path to a Cloud Storage object and read it in the next task.. Airflow XComs allow tasks to exchange small amounts of data (e.g., file paths) by pushing and pulling values. Cloud Storage can be used as an intermediate store: the Spark job writes output to GCS, and the BigQuery task reads from that location. BigQuery does not directly communicate with Dataproc. Cloud Composer variables are for global configuration, not task-to-task. Pub/Sub is not needed for simple file path sharing.

Answer

Store the output path as a Cloud Composer variable.

Answer

Publish the output path to a Pub/Sub topic and subscribe in the next task.

Answer

Use BigQuery as an intermediary to store the output path.

Question 14

You are designing a data pipeline that ingests streaming data from Pub/Sub, processes it with Dataflow, and writes to BigQuery. You need to ensure that schema changes in the incoming data (new fields) are handled without pipeline failure. Which THREE steps should you take? (Choose THREE.)

Accepted Answer

Set the BigQuery table schema to allow automatic addition of new fields by using 'ignore_unknown_values'.. To handle schema drift, BigQuery allows schema relaxation (adding nullable columns) via DDL. Dataflow can update the destination table schema using BigQuery APIs (e.g., set the schema to include unknown fields). Alternatively, using BigQuery's automatic schema detection with 'ignore_unknown_values' can allow new fields to be added automatically. Setting all fields to NULLABLE in advance is not practical. Setting the table to require all fields causes failures on unknown fields.

Answer

Configure the BigQuery table to require all fields, causing the pipeline to fail on unknown fields.

Answer

Set all columns in the BigQuery table as NULLABLE from the start.

Question 15

You are using Cloud Workflows to orchestrate a series of API calls. You need to handle errors and retries. Which THREE features of Cloud Workflows can you use? (Choose THREE.)

Accepted Answer

Use try/except blocks to catch and handle errors.. Cloud Workflows supports steps with retry policies, try/except blocks for error handling, and conditional (if-else) logic for branching. It does not have built-in built-in step-level logging (logging is done via Cloud Logging), and there is no built-in load balancer integration.

Answer

Integrate with Cloud Load Balancing for high availability.

Answer

Enable automatic logging for each step.