PDE Maintaining and Automating Data Workloads Practice Test 2 — 15 Questions

Question 1

A data engineer is building a batch pipeline that runs daily using Cloud Composer. The pipeline has three tasks: extract data from Cloud Storage, transform data using Dataflow, and load the transformed data into BigQuery. The engineer wants to ensure that the Dataflow job only starts after the extraction task completes successfully, and the load task only starts after the Dataflow job finishes. How should the engineer define the task dependencies in the Airflow DAG?

Accepted Answer

extract >> transform >> load. Option C is correct because Airflow uses the bitshift operator (>>) to define task dependencies in a linear sequence. The DAG must ensure that the extract task completes before the transform task starts, and the transform task completes before the load task starts. This is achieved by chaining the tasks in order: extract >> transform >> load, which enforces the required sequential execution.

Answer

extract >> [transform, load]

Answer

transform >> extract >> load

Answer

extract >> load >> transform

Question 2

You need to schedule a simple workflow that fetches data from an API every hour, transforms it using Cloud Functions, and writes the result to Cloud Storage. The workflow has no complex branching or retry logic beyond basic retries. Which orchestration service is the MOST cost-effective and simplest to implement?

Accepted Answer

Workflows. Workflows is serverless, pay-per-execution, and defined in YAML/JSON. It integrates natively with Cloud Functions and Cloud Storage. Cloud Composer is overkill for simple linear workflows and incurs cluster costs. Cloud Scheduler alone cannot orchestrate multiple steps. Dataflow is for data processing, not orchestration.

Answer

Cloud Scheduler

Answer

Cloud Composer

Answer

Dataflow

Question 3

A company runs a streaming Dataflow pipeline that reads from Pub/Sub, enriches data with a side input from BigQuery, and writes to BigQuery. After updating the pipeline code (adding a new field to the output), the engineer notices that the new pipeline version is not picking up the updated code because the job was started from a template. The engineer wants to update the streaming pipeline without draining it. What should the engineer do?

Accepted Answer

Use the gcloud dataflow jobs update command with the new Flex Template.. Option A is correct because the `gcloud dataflow jobs update` command allows you to update a running streaming Dataflow pipeline with a new Flex Template without draining or stopping the job. This command performs an in-place update, preserving the job's state and checkpointing, so the pipeline continues processing with the new code. Since the original job was started from a template, using this command with the new Flex Template ensures the updated code is picked up seamlessly.

Answer

Stop the pipeline, update the template, and restart with the same job name.

Answer

Modify the original template and redeploy it as a new job with the same pipeline name.

Answer

Use the gcloud dataflow jobs drain command, then restart with the new template.

Question 4

You are monitoring a streaming Dataflow pipeline that reads from Pub/Sub and writes to BigQuery. In Cloud Monitoring, you notice that the 'system_lag' metric is increasing over time and now exceeds 10 minutes. The 'data_watermark' metric shows a steady lag. What is the most likely cause of the increasing system lag?

Accepted Answer

The Dataflow pipeline is underprovisioned with workers, causing processing backlog.. Option D is correct because an underprovisioned pipeline lacks sufficient worker resources to process incoming messages at the rate they arrive. This causes a growing backlog in the pipeline's internal buffers, which directly increases the 'system_lag' metric (the time between data ingestion and processing). The 'data_watermark' lag remaining steady indicates that the pipeline is still making progress on event-time processing, but the overall processing capacity is insufficient to keep up with the input rate.

Answer

BigQuery write throughput is throttling the pipeline.

Answer

The Pub/Sub subscription has too many unacknowledged messages.

Answer

The pipeline is using a global window with late data handling.

Question 5

A company wants to share a large BigQuery dataset with a partner for analysis. The partner needs read-only access to a specific snapshot of the data as of a certain point in time, and the company wants to avoid additional storage costs for the partner. What is the most cost-effective approach?

Accepted Answer

Create a BigQuery table clone at the desired point in time and share it with the partner.. BigQuery table clones are zero-copy clones that share the underlying storage with the base table. They do not incur additional storage costs until the data in the clone is modified. Snapshots incur storage costs for the snapshot. Authorized views or datasets require the partner to query the base table, which may incur analysis costs but no extra storage; however, the partner may see changes to the base table. The question specifies a point-in-time snapshot, so a clone is best.

Answer

Create a BigQuery table snapshot at the desired point in time and share it.

Answer

Export the table to Cloud Storage as Avro and share a signed URL.

Answer

Grant the partner access to the original table with an authorized view.

Question 6

Your organization has a BigQuery flat-rate reservation with 2000 slots. During peak hours, query performance degrades because concurrent queries exceed the available slots. You want to handle these bursts without changing the base reservation. What should you do?

Accepted Answer

Purchase flex slots during peak hours to add capacity temporarily.. Option D is correct because flex slots allow you to temporarily add capacity to a BigQuery flat-rate reservation without committing to a permanent increase. This handles burst workloads during peak hours by adding slots on demand, and you only pay for the time they are used. The base reservation of 2000 slots remains unchanged, meeting the requirement.

Answer

Enable autoscaling on the reservation to automatically add slots up to a maximum.

Answer

Purchase committed use discounts to increase the base reservation to 3000 slots.

Answer

Change the pricing model to on-demand to allow unlimited slots.

Question 7

A data engineer is designing a Dataflow pipeline that reads from a Kafka topic (using Pub/Sub for Kafka) and writes to BigQuery. The data schema may change over time, with new fields appearing. The engineer wants to handle schema drift automatically without failing the pipeline. Which approach should the engineer use?

Accepted Answer

Store the entire record as a single JSON string column in BigQuery and parse it later.. Option C is correct because storing the entire record as a single JSON string column in BigQuery allows the pipeline to accept any schema changes without requiring schema modifications at write time. This approach decouples the ingestion from schema evolution, enabling the data to be parsed later using BigQuery's JSON functions (e.g., JSON_EXTRACT) or by loading into a separate schema-on-read layer. It avoids pipeline failures caused by mismatched fields or type changes.

Answer

Use a Dataflow side input that reads the latest schema from a file and updates the BigQuery schema accordingly.

Answer

Define a UDF in Dataflow that dynamically adjusts the output schema.

Answer

Configure the Dataflow pipeline to use BigQuery's schema autodetect option for each insert.

Question 8

You are troubleshooting a Dataproc cluster that runs nightly Spark jobs. The jobs are failing with out-of-memory errors. You want to reduce costs while fixing the issue. Which combination of actions should you take? (Select the BEST answer.)

Accepted Answer

Increase the number of preemptible worker nodes and use standard machine types for the master.. Preemptible workers are cheaper and can be used for worker nodes, but not for master nodes. Adding more preemptible workers can increase parallelism and reduce memory pressure per worker, but may cause more preemptions. Using high-memory master nodes is not necessary for worker memory issues. Using SSDs for scratch storage can improve performance but does not directly address OOM. Reducing worker count would exacerbate the problem.

Answer

Use SSD persistent disks for all nodes to improve I/O performance.

Answer

Decrease the number of worker nodes and increase the size of each worker.

Answer

Switch to a high-memory machine type for the master node only.

Question 9

A data engineer needs to inspect a BigQuery table for sensitive data such as credit card numbers and email addresses before sharing it with a third party. The engineer also wants to de-identify the data by masking the sensitive columns. Which Google Cloud service should be used?

Accepted Answer

Cloud DLP. Cloud DLP (Data Loss Prevention) is the correct service because it is specifically designed to inspect, classify, and de-identify sensitive data such as credit card numbers and email addresses. It provides built-in infoType detectors for over 150 types of sensitive data and supports masking, tokenization, and other de-identification techniques. The engineer can use Cloud DLP to scan BigQuery tables and then apply transformations to mask the sensitive columns before sharing.

Answer

Dataplex

Answer

BigQuery column-level security

Answer

Data Catalog

Question 10

A company runs a critical batch pipeline using Cloud Dataflow. The pipeline processes financial transactions and runs every hour. Recently, some runs have failed due to transient errors (e.g., network timeouts). The engineer wants to automatically retry failed runs without manual intervention. The pipeline is launched from a Cloud Composer DAG using DataflowPythonOperator. What is the BEST way to handle retries?

Accepted Answer

Set the 'retries' parameter in the DAG's default_args to a positive integer.. Option B is correct because Cloud Composer (Apache Airflow) natively supports task-level retries via the 'retries' parameter in default_args. When a DataflowPythonOperator fails due to a transient error, Airflow automatically re-executes the task up to the specified number of retries, without requiring custom sensors or external triggers. This is the simplest and most reliable mechanism for handling transient failures in a DAG-driven pipeline.

Answer

Add a DataflowJobStatusSensor in the DAG that waits for job completion and retries if failed.

Answer

Configure the Dataflow pipeline to automatically retry on failure using the --numberOfWorkerHarnessThreads option.

Answer

Use a Cloud Function triggered by Cloud Scheduler to re-launch the pipeline if the Dataflow job fails.

Question 11

You are optimizing a BigQuery query that scans 1 TB of data every day. The query joins a large fact table (partitioned by date) with a small dimension table. You notice that the query always scans the entire fact table, even though you only need the last 7 days of data. Which optimization will MOST reduce the bytes scanned?

Accepted Answer

Add a WHERE clause that filters on the date column used for partitioning.. Using a WHERE clause on the partitioning column (e.g., date) allows BigQuery to perform partition pruning, drastically reducing scanned bytes. Clustering on the join key can improve performance but does not reduce bytes scanned as much as partition pruning. Materialized views precompute aggregations but may not reduce scans if the base query still scans full table. Changing to a clustered table does not prune partitions automatically.

Answer

Create a materialized view that pre-aggregates the data by day.

Answer

Cluster the fact table on the join key used in the query.

Answer

Change the table to use time-unit column partitioning with a 1-day partition interval.

Question 12

A company uses Cloud Pub/Sub for a real-time data pipeline. The subscription has a backlog of millions of messages that are not being processed quickly enough. In Cloud Monitoring, you observe that the 'subscription/num_undelivered_messages' metric is high and growing, while 'subscription/oldest_unacked_message_age' is also increasing. Which action is MOST likely to reduce the backlog?

Accepted Answer

Increase the number of subscribers or the throughput capacity of the existing subscribers.. Option D is correct because the backlog indicates that subscribers cannot keep up with the message flow. Increasing the number of subscribers or scaling their throughput capacity directly addresses the processing bottleneck, allowing messages to be pulled and acknowledged faster. Cloud Pub/Sub scales horizontally, so adding more pull subscribers or increasing the resources of existing ones (e.g., more worker threads, higher CPU/memory) reduces the backlog.

Answer

Delete the subscription and recreate it with a larger message retention duration.

Answer

Reduce the acknowledgment deadline to force faster processing.

Answer

Change the subscription type from push to pull.

Question 13

A data engineer needs to set up a Dataplex data quality scan to run weekly on a BigQuery table. The scan should check that: (1) the 'email' column is not null, (2) the 'age' column is between 0 and 120, and (3) the 'country_code' column matches a list of valid ISO codes. Which TWO Dataplex features should the engineer use?

Accepted Answer

Schedule the data quality scan using Dataplex scheduling options. Dataplex data quality tasks use SQL-based rules. Row rules validate conditions per row (e.g., IS NOT NULL, BETWEEN). Column rules validate column-level conditions (e.g., in set). Schedule is set via the scheduling feature. The other options are not Dataplex data quality features: DLP is for sensitive data, Data Catalog is for metadata.

Answer

Use Cloud DLP to inspect the table for data quality issues

Answer

Create a Dataplex asset and add a tag template to enforce constraints

Question 14

A company is migrating their on-premises data warehouse to BigQuery. They have a mix of batch and streaming ingestion. The data team wants to optimize query costs. Which THREE practices should they adopt?

Accepted Answer

Use materialized views for frequently executed aggregations.. Partitioning by date reduces bytes scanned. Using clustered tables improves performance for filter/join queries. Using materialized views can precompute aggregations and reduce scans. Flat-rate pricing is about reservation management, not cost optimization per query. Limiting slots is not a cost optimization; it may cause throttling.

Answer

Switch to flat-rate pricing to cap slot usage.

Answer

Limit the number of concurrent queries by setting a maximum slot capacity.

Question 15

A data engineer is building a Cloud Workflows workflow that orchestrates multiple Cloud Functions and API calls. The workflow should handle transient failures with retries and send a notification to a Pub/Sub topic if the workflow ultimately fails. Which THREE steps should the engineer include in the workflow definition?

Accepted Answer

Use the 'googleapis.pubsub.v1.projects.topics.publish' connector to send a failure notification.. Workflows supports retry policies via 'retry' blocks, conditional steps using 'switch', and Pub/Sub publishing via the 'googleapis.pubsub.v1.projects.topics.publish' connector. The 'try/catch' is not a Workflows construct; instead, use 'step' with 'retry' and 'on_error' for failure handling. 'for' loops are for iteration, not error handling.

Answer

Use a 'for' loop to iterate over retries.

Answer

Use a 'try' / 'catch' block to handle exceptions and route to a failure step.