PDE Practice Test 28 — 15 Questions

Question 1

In Cloud Composer, a DAG has two tasks: task_A (runs an Apache Spark job on Dataproc) and task_B (loads data from Cloud Storage to BigQuery). task_B must start after task_A completes. The DAG is scheduled to run hourly. Sometimes task_B starts before task_A finishes because task_A's Dataproc job appears to complete in the Airflow metadata but the data is not yet available. What is the best way to ensure task_B only runs after the data is fully written?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Use DataprocJobOperator with a job_poll_interval and add a sensor to verify output. It addresses the root cause: the Dataproc job may report completion in Airflow metadata before the output data is fully written to Cloud Storage. By using DataprocJobOperator with a job_poll_interval, you ensure Airflow waits for the actual job completion on Dataproc, and adding a sensor to verify the output (e.g., checking for a success marker file or expected data in Cloud Storage) guarantees that task_B only starts after the data is fully available. This two-step approach prevents race conditions between job completion and data consistency.

Answer

Increase the number of retries for task_B

Answer

Use a sensor after task_A that checks for a specific file in Cloud Storage

Answer

Change the DAG schedule to run every 30 minutes

Question 2

Your team runs a weekly batch ETL pipeline using Cloud Dataproc. The pipeline reads raw data from Cloud Storage, transforms it with Apache Spark, and writes results to BigQuery. Recently, the pipeline has been failing with the error 'Out of Memory' during the shuffle phase. The cluster uses standard worker nodes (n1-standard-4). What is the most effective way to resolve this without increasing total cost?

Accepted Answer

Increase the number of Spark partitions by setting spark.sql.shuffle.partitions to a higher value.. The 'Out of Memory' error during the shuffle phase indicates that individual executor tasks are processing too much data per partition. Increasing `spark.sql.shuffle.partitions` reduces the amount of data each task handles, lowering memory pressure per executor without adding more nodes or upgrading hardware. This directly addresses the shuffle memory bottleneck while keeping the total cluster cost unchanged.

Answer

Increase the number of worker nodes by adding more n1-standard-4 instances.

Answer

Enable dynamic allocation and use preemptible VMs for some workers.

Answer

Switch worker nodes to n1-highmem-4 instances to provide more memory.

Question 3

A company processes real-time clickstream data from websites. They need to aggregate user sessions that may span multiple hours and handle events that arrive late due to network delays. The pipeline must avoid discarding late data. Which Dataflow feature should they configure?

Accepted Answer

Use session windows with a gap duration and allow late data with a suitable allowed_lateness. Session windows are ideal for aggregating user sessions that span multiple hours, as they group events based on a gap duration of inactivity. By configuring `allowed_lateness`, the pipeline can handle late-arriving events without discarding them, ensuring completeness. This directly addresses the requirement to avoid discarding late data while aggregating sessions.

Answer

Use fixed windows with a trigger that fires after every element

Answer

Use the GlobalWindow with a watermark

Answer

Use sliding windows with no allowed lateness

Question 4

A media company uses Cloud Dataflow to process video metadata from a Pub/Sub stream. The pipeline enriches metadata using a lookup table stored in Cloud Bigtable. Recently, they noticed increased latency and occasional 'Bigtable operation timeout' errors. The Bigtable instance has 3 nodes and the data is highly distributed. The Dataflow pipeline uses default settings. What is the most likely cause of the timeouts?

Accepted Answer

The Bigtable table uses a single column family with over 100 columns, leading to high read overhead. A single column family with over 100 columns in Bigtable forces the system to read all column qualifiers for each row, even if only a few are needed. This increases read overhead and latency, and can trigger 'operation timeout' errors when the Dataflow pipeline's default settings (which do not limit column qualifiers) request the entire row. The highly distributed data and 3-node cluster exacerbate the issue, but the root cause is the excessive column count within one family.

Answer

The Dataflow pipeline uses a large batch size for Bigtable reads, overwhelming the instance

Answer

The Bigtable cluster has too few nodes for the read throughput

Answer

The Dataflow pipeline does not cache Bigtable results, causing repeated lookups

Question 5

A company is migrating an on-premises Hadoop cluster to Google Cloud. They need to run existing Spark jobs with minimal modification. Which THREE strategies should they consider? (Choose THREE.)

Accepted Answer

Use Cloud Dataproc with Spark and Hive components.. Cloud Dataproc is a managed Spark and Hadoop service that supports the same Spark and Hive components used on-premises, allowing existing Spark jobs to run with minimal modification. It provides native integration with Cloud Storage, which can replace HDFS without changing job logic, and the Dataproc Jobs API enables programmatic job submission, preserving existing workflows.

Answer

Migrate to BigQuery for all analytics.

Answer

Rewrite Spark jobs as Dataflow pipelines.

Question 6

A data analyst frequently queries a BigQuery table that contains an array of structs representing product purchases. The query below runs slowly:

SELECT customer_id, COUNT(purchase) as total_purchases
FROM sales, UNNEST(purchases) as purchase
GROUP BY customer_id

What change would most improve query performance?

Accepted Answer

Create a materialized view that pre-aggregates by customer_id and purchase count. The query runs slowly because it must unnest the `purchases` array for every row and then aggregate. A materialized view pre-aggregates the data by `customer_id` and purchase count, avoiding repeated full scans and unnesting. This is the most impactful optimization because it eliminates the compute cost of UNNEST and GROUP BY at query time.

Answer

Partition the table by transaction date

Answer

Use a subquery to filter purchases first

Answer

Cluster the table by purchases.product_id

Question 7

A data science team has deployed a custom TensorFlow model on Vertex AI Prediction. They notice increasing prediction latency and a growing number of 503 errors during peak traffic hours. The model is served using a single regional endpoint with min replica count of 2 and max replica count of 10. Which TWO actions should the team take to address these issues?

Accepted Answer

Enable autoscaling with a higher max replica count and configure a CPU utilization target of 60%.. Increasing the max replica count and setting a CPU utilization target of 60% allows the Vertex AI Prediction service to scale out more aggressively during traffic spikes, reducing both latency and 503 errors. This directly addresses the bottleneck of insufficient compute capacity under peak load, as the current max of 10 replicas may be too low to handle demand.

Answer

Use a larger machine type (e.g., n1-highmem-8) instead of the current n1-standard-4 to improve per-replica throughput.

Answer

Reduce the min replica count to 0 to allow the service to scale down to zero when not in use.

Answer

Deploy the model as a batch prediction job and move all online predictions to batch.

Question 8

A Dataflow streaming job is processing high-volume sensor data from thousands of IoT devices. The job uses global windows with a 10-minute processing time trigger. Recently, the job's CPU utilization is nearly 100% and it is falling behind. Which action is most likely to reduce CPU load while maintaining data freshness?

Accepted Answer

Replace GroupByKey with Combine.globally and use a fanout.. Using `Combine.globally` with a fanout reduces the amount of data shuffled and merged in a single worker, lowering CPU load. In Dataflow, `GroupByKey` triggers a full shuffle and per-key aggregation, which is expensive for high-volume sensor data; `Combine.globally` with a fanout performs partial aggregation on each worker before a final merge, reducing network I/O and CPU cycles. This maintains data freshness because the 10-minute processing time trigger still fires on time, but with less per-element overhead.

Answer

Increase the number of workers to distribute the load.

Answer

Change the trigger to event time with a 10-minute allowed lateness.

Answer

Use side inputs to broadcast a static lookup table to all workers.

Question 9

A company processes CSV files that are uploaded to Cloud Storage by external partners. Each file is around 500 MB, and they need to be parsed and loaded into BigQuery. The processing must start as soon as the file arrives. What is the most efficient serverless architecture?

Accepted Answer

Cloud Storage triggers a Cloud Function that publishes events to Pub/Sub; a Dataflow streaming pipeline reads from Pub/Sub and writes to BigQuery.. It combines Cloud Storage event-driven triggers with Pub/Sub for reliable asynchronous message delivery, and uses Dataflow streaming with autoscaling to handle 500 MB files efficiently. This serverless architecture ensures processing starts immediately upon file arrival, scales to handle large files without manual intervention, and leverages BigQuery's streaming inserts for near-real-time data loading.

Answer

Use Cloud Scheduler to periodically check for new files and process them with Dataflow batch jobs.

Answer

Cloud Storage triggers a Dataproc job that reads the file and loads it into BigQuery.

Answer

Cloud Storage triggers a Cloud Function that directly loads the data into BigQuery using the BigQuery API.

Question 10

A company wants to implement a data lake on Google Cloud to store raw sensor data (unstructured binary files) and allow data scientists to run SQL queries on processed data. They expect to store terabytes of data and have different access patterns. Which combination of GCP services best meets these requirements?

Accepted Answer

Cloud Storage for raw data and BigQuery for processed data. Cloud Storage is the ideal service for storing raw, unstructured binary sensor data at petabyte scale, offering low-cost, durable object storage with multiple access tiers. BigQuery is a serverless, highly scalable data warehouse that allows data scientists to run SQL queries on processed data, with features like columnar storage and automatic optimization for analytical workloads. This combination directly addresses the need for raw storage and SQL-based analytics on processed data.

Answer

Bigtable for raw data and Cloud Spanner for processed data

Answer

Cloud Storage for both raw and processed data

Answer

Cloud SQL for raw data and Cloud Dataproc for processing

Question 11

A company runs a Cloud Dataflow streaming pipeline that reads from Cloud Pub/Sub, performs a fixed window of 10 seconds, joins with a slowly-changing dimension table stored in Cloud Bigtable, and writes results to BigQuery. The pipeline has been running for months but recently started exhibiting increasing latency and occasional data loss. The pipeline uses default settings with autoscaling enabled (min 2, max 20 workers). The Bigtable cluster has 3 nodes. The dimensions are updated infrequently. The latency has grown from seconds to minutes. Examining the Dataflow monitoring UI, you see that the 'System Lag' metric is increasing, and some windows are not being emitted. The CPU utilization on Bigtable nodes is below 50%. There are no errors in the logs. Which action is most likely to resolve the issue?

Accepted Answer

Set the pipeline option --maxNumWorkers to a value between 5 and 10.. The increasing system lag and unemitted windows in a streaming pipeline with autoscaling (2–20 workers) and a 3-node Bigtable cluster indicate that the pipeline is bottlenecked by the number of workers, not by Bigtable performance. With default autoscaling, Dataflow may not scale up aggressively enough to handle the sustained load, causing backlog and window expiration. Capping maxNumWorkers to 5–10 ensures sufficient parallelism without over-provisioning, allowing the pipeline to catch up and emit windows reliably.

Answer

Increase the window duration to 30 seconds to reduce the number of windows.

Answer

Redesign the pipeline to use a side input for the dimension table instead of a lookup.

Answer

Increase the number of Bigtable nodes to reduce lookup latency.

Question 12

You are designing a streaming Dataflow pipeline that reads from Cloud Pub/Sub. Some data may arrive late due to network delays. You need to ensure that late-arriving data is still processed, but after a certain point, it should be discarded to avoid unbounded state. What is the best practice?

Accepted Answer

Set a watermark and allowed lateness. In streaming Dataflow pipelines, setting a watermark and allowed lateness provides a mechanism to handle late-arriving data from Pub/Sub without unbounded state growth. The watermark defines the point after which data is considered late, and allowed lateness specifies how long to wait for late data before discarding it, balancing completeness and state management.

Answer

Switch to a batch pipeline

Answer

Use fixed windows without allowed lateness

Answer

Discard all late-arriving data

Question 13

A data engineering team uses Cloud Data Fusion to build ETL pipelines. They have a pipeline that reads from Cloud SQL, transforms data using Wrangler, and writes to BigQuery. The pipeline fails intermittently with a 'connection timeout' error from Cloud SQL. What is the best way to handle this?

Accepted Answer

Configure the Cloud SQL connector in Data Fusion to use retry logic and increase the connection timeout.. Cloud Data Fusion's Cloud SQL connector can be configured with retry logic and an increased connection timeout to handle transient network issues. This directly addresses the intermittent 'connection timeout' error without requiring architectural changes, as the error is likely due to brief network latency or resource contention, not a persistent connectivity problem.

Answer

Use Cloud NAT to provide a static IP for Data Fusion to whitelist.

Answer

Increase the number of Data Fusion nodes to distribute the load.

Answer

Migrate Cloud SQL to Cloud Spanner to handle higher concurrency.

Question 14

A retail company is building a recommendation engine that requires processing customer clickstream data in near real-time. The data is ingested via Pub/Sub, and must be joined with a lookup table of product details (updated daily) before being used for model inference. Which design pattern should they use?

Accepted Answer

Use a Dataflow pipeline that reads from Pub/Sub and uses a side input from a regularly refreshed PCollection loaded from Cloud Storage.. Dataflow can read streaming data from Pub/Sub and use a side input from a regularly refreshed PCollection loaded from Cloud Storage. This pattern allows the product lookup table (updated daily) to be periodically reloaded into the pipeline as a side input, enabling efficient, low-latency enrichment of each event without per-event external calls or batch delays.

Answer

Enrich the stream by querying BigQuery for each event using a Cloud Function.

Answer

Store product details in Cloud Memorystore (Redis) and have the streaming application look up each event.

Answer

Write events to BigQuery and use scheduled queries to join with the product table in batch.

Question 15

A media company ingests video files from partners via a REST API. Files are stored in Cloud Storage, and metadata is written to Firestore. A Cloud Function is triggered on object finalize to transcode video using Transcoder API. Sometimes, the function fails because the file is still being uploaded when triggered. How should this be fixed?

Accepted Answer

Use Cloud Pub/Sub notifications for Cloud Storage and trigger the function from the subscription.. Cloud Storage object finalize notifications are sent only after the entire file has been written and committed. By using Pub/Sub notifications for Cloud Storage and triggering the Cloud Function from the subscription, you decouple the trigger from the upload process, ensuring the function only runs when the file is fully available. This eliminates the race condition where the function is triggered before the upload completes.

Answer

Implement a Cloud Composer workflow to poll for file existence.

Answer

Require partners to use resumable uploads.

Answer

Increase the Cloud Functions timeout to allow time for the upload to finish.