PDE Practice Test 45 — 15 Questions

Question 1

A large retail company processes point-of-sale transactions from thousands of stores daily. The current batch pipeline runs on Cloud Dataproc using Spark and takes 3 hours to complete. The business wants to reduce processing time to under 30 minutes. The pipeline reads from Cloud Storage, joins with inventory data from BigQuery, performs aggregations, and writes to Cloud SQL for reporting. What is the most effective optimization?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Read inventory data from BigQuery and pre-join in BigQuery, then export to Cloud Storage as ORC files. It offloads the join operation to BigQuery, which is optimized for large-scale analytics and can process the join much faster than Spark. By pre-joining and exporting the result as ORC files (a columnar format optimized for Spark), the pipeline avoids the expensive shuffle and data transfer between Cloud Storage and BigQuery, significantly reducing the overall processing time to meet the 30-minute target.

Answer

Migrate the pipeline to Cloud Dataflow with Apache Beam for auto-scaling

Answer

Write intermediate results to Cloud SQL instead of BigQuery for faster access

Answer

Increase the number of worker nodes in the Dataproc cluster

Question 2

Your team uses Cloud Dataproc to run a Spark ML training job. The job is failing with an error: 'Container killed by YARN for exceeding memory limits.' What should you do to fix this?

Accepted Answer

Increase the spark.executor.memory property. The error 'Container killed by YARN for exceeding memory limits' indicates that the Spark executor process is using more memory than the YARN container allows. Increasing `spark.executor.memory` allocates a larger YARN container for each executor, providing the necessary headroom for the Spark application's memory demands, including overhead for off-heap memory and JVM internals.

Answer

Use preemptible VMs for faster execution

Answer

Increase the number of worker nodes

Answer

Enable the external shuffle service

Question 3

A company needs to store petabytes of time-series IoT sensor data and query it with single-digit millisecond latency at millions of reads per second. The data has a simple key-value structure with timestamps. Which Google Cloud database is MOST appropriate?

Accepted Answer

Cloud Bigtable. Cloud Bigtable is a fully managed, scalable NoSQL database designed for large analytical and operational workloads, handling petabytes of data with consistent sub-10ms latency at millions of reads per second. Its key-value model with timestamps directly matches the time-series IoT sensor data structure, and it supports high-throughput, low-latency access via the HBase API or Bigtable client libraries.

Answer

BigQuery

Answer

Cloud Spanner

Answer

Firestore

Question 4

A company runs a batch ETL pipeline on Cloud Dataproc. During peak hours, the job takes longer than expected. The pipeline reads from Cloud Storage, transforms data, and writes to BigQuery. What is the most cost-effective way to improve performance without redesigning the pipeline?

Accepted Answer

Add a secondary worker group using preemptible VMs and increase the number of workers.. Adding a secondary worker group with preemptible VMs is the most cost-effective way to improve performance because it allows you to scale out the cluster horizontally with compute instances that are significantly cheaper (up to 80% discount) than regular VMs. This directly addresses the bottleneck of processing capacity during peak hours without requiring any pipeline redesign, as Cloud Dataproc can automatically distribute work across additional workers.

Answer

Enable local SSDs on all worker nodes.

Answer

Increase the master node's machine type to n1-highmem-32.

Answer

Use Cloud Composer to schedule the job with a higher priority.

Question 5

A Dataflow batch job fails consistently with the error shown. The job uses a custom container image and runs in a VPC with a private IP. What should the engineer do to resolve the issue?

Accepted Answer

Verify that the VPC has Private Google Access enabled and that Cloud NAT is configured for outbound internet access if needed.. The error indicates that the Dataflow batch job cannot access required resources (e.g., container image, dependencies) because the VPC with private IPs lacks outbound internet connectivity. Option B is correct because enabling Private Google Access allows the VMs to reach Google APIs (like Container Registry) via the Google network, and Cloud NAT provides outbound internet access for non-Google APIs or external dependencies. Without these, the job fails to pull the custom container image or download necessary artifacts.

Answer

Request a CPU quota increase in the region.

Answer

Rebuild the custom container image and upload it to Container Registry.

Answer

Check that the custom image is based on the latest Dataflow SDK version.

Question 6

A data engineer needs to process a large dataset (500 TB) stored in Cloud Storage using Dataproc. The processing job requires reading the entire dataset and writing results back to Cloud Storage. The job is expected to run for 6 hours. Which configuration minimizes cost?

Accepted Answer

Use a cluster with a mix of standard and preemptible VMs.. Preemptible VMs cost about 80% less than standard VMs, and mixing them with standard VMs provides fault tolerance for the job's 6-hour duration. Since the job reads and writes to Cloud Storage (not local HDFS), local SSDs are unnecessary, and a single-node cluster would lack the parallelism needed to process 500 TB efficiently within 6 hours. Using a mix of standard (for critical master/worker nodes) and preemptible VMs (for worker nodes) minimizes cost while ensuring job completion.

Answer

Use a single-node cluster with standard VMs.

Answer

Use a cluster with local SSDs for faster I/O.

Answer

Use a cluster with n1-highmem-32 instances and 1000 cores.

Question 7

A financial services company operates a real-time fraud detection pipeline using Apache Beam running on Google Cloud Dataflow. The pipeline reads transactions from Pub/Sub, enriches them with customer data from Bigtable, runs a machine learning model with side inputs from a Redis cluster, and writes results to BigQuery for downstream reporting. The data must be processed with exactly-once semantics to avoid duplicate fraud alerts or missing transactions. The pipeline currently uses a global window with 5-minute accumulation, but the team is experiencing high latency and occasional duplicates when the model side input is updated (triggered every 15 minutes via a WatchTransform). Additionally, the pipeline has a dead letter queue that outputs failed records to a separate Pub/Sub topic, but these records are never reprocessed. The team needs to ensure high reliability and data quality. Which course of action should the team take to improve solution quality?

Accepted Answer

Implement sliding windows of 5 minutes with a 2-minute allowed lateness, use side inputs with periodic refreshes using the .withUpdateFrequency transformation, and set up a Cloud Function to automatically replay dead letter records back to the main Pub/Sub topic after fixing the issue.. Sliding windows with 2-minute allowed lateness handle late-arriving data without causing duplicates, and the .withUpdateFrequency transformation refreshes the side input every 15 minutes without triggering reprocessing, reducing latency. Replaying dead letter records via a Cloud Function ensures data completeness. Option A is incorrect because fixed windows with session gaps do not address side input latency and may lose late events. Option B is incorrect because batch processing is unsuitable for real-time fraud detection and introduces significant latency. Option D is incorrect because custom triggers with early firings can cause duplicates due to side input updates, and the Read transform does not efficiently handle periodic model refreshes.

Answer

Use fixed windows with a 10-minute duration and session gap of 2 minutes, disable side input caching, and log all dead letter records to Cloud Storage for manual inspection.

Answer

Switch to a batch processing approach that runs every minute using Cloud Composer, with data loaded from Pub/Sub into BigQuery and then processed with Dataproc to run the model.

Answer

Keep the global window but use a custom trigger with early firings every 30 seconds and a late-firing threshold of 1 minute, and configure the side input to be broadcast every 5 minutes using a Read transform.

Question 8

A streaming pipeline ingests events from Pub/Sub, enriches them via a slow REST API call, and writes the result to BigQuery. The API has a limit of 10 requests per second per client. The pipeline processes 1000 messages per second. Which approach minimizes latency while respecting API limits?

Accepted Answer

Use a global window with a trigger that fires every second, and inside the DoFn limit concurrent API calls to 10.. Using a global window with a trigger every second groups 1000 messages into a batch, and then throttling concurrent API calls to 10 within the DoFn (e.g., using a fixed-size thread pool) respects the API limit while minimizing latency by processing messages in parallel up to the limit. Option B is wrong because fanning out to multiple API instances doesn't help if the limit is per client; the total requests per second across all instances would still exceed the client limit. Option C is wrong because Dataflow Flex Templates are used to run parameterized pipelines, not to solve throttling issues. Option D is wrong because assigning a random key and using a sliding window distributes messages across workers, but without explicit throttling, the API limit could still be exceeded.

Answer

Fan out the stream to multiple REST API instances using Pub/Sub topic splitting.

Answer

Use a Dataflow Flex Template to run multiple pipelines, each processing a subset of messages.

Answer

Assign each message a random key and use a sliding window of 10 seconds; the API call will be distributed across workers.

Question 9

A Cloud SQL instance for PostgreSQL is experiencing heavy read traffic. The team wants to offload read queries while maintaining data consistency. Which solution meets their needs?

Accepted Answer

Create read replicas and direct read queries to them.. Read replicas in Cloud SQL for PostgreSQL allow you to offload read traffic from the primary instance by creating one or more asynchronous replicas that serve read queries. This maintains data consistency because replicas use PostgreSQL's native streaming replication, ensuring that all committed transactions on the primary are eventually reflected on the replica, providing a consistent snapshot for read operations.

Answer

Increase the tier of the Cloud SQL instance.

Answer

Create a Cloud SQL failover replica.

Answer

Use Cloud Memorystore as a cache in front of Cloud SQL.

Question 10

A data pipeline uses Cloud Pub/Sub to ingest events and Cloud Functions to transform and write to BigQuery. The system is experiencing data loss during Pub/Sub subscription outages. Which design change improves reliability?

Accepted Answer

Use Dataflow with at-least-once delivery and checkpointing. Dataflow with at-least-once delivery and checkpointing ensures that messages are not lost during Pub/Sub subscription outages because Dataflow tracks processing progress via checkpoints and can replay unacknowledged messages from the last checkpoint. This decouples the processing from the subscription's transient failures, providing fault-tolerant, exactly-once or at-least-once semantics depending on the sink.

Answer

Use a pull subscription with a custom app that polls frequently

Answer

Use long ack deadlines to keep messages in the subscription

Answer

Increase the timeout in Cloud Functions

Question 11

A company needs to stream data from a fleet of IoT devices to BigQuery for near-real-time analytics. The data volume is unpredictable and can spike during certain events. Which Google Cloud service should be used as the ingestion point to handle variable throughput with minimal operational overhead?

Accepted Answer

Cloud Pub/Sub. Cloud Pub/Sub is the correct choice because it is a fully managed, scalable messaging service designed to decouple data producers from consumers, handling unpredictable and spiky throughput without requiring manual scaling. It can ingest millions of messages per second and buffer them until BigQuery is ready to consume, ensuring near-real-time analytics with minimal operational overhead.

Answer

Cloud Datastore

Answer

Cloud Functions

Answer

Cloud Storage

Question 12

The push endpoint is returning 500 errors. What is the most likely cause?

Accepted Answer

The push endpoint requires authentication but none is set. The push endpoint likely requires authentication, but none is configured, causing the 500 errors.

Answer

The topic has no messages

Answer

The push endpoint is not a valid HTTPS URL

Answer

The ack deadline is too short

Question 13

A company uses Dataflow to process streaming data from Pub/Sub. They notice increased processing latency. What is the most likely cause?

Accepted Answer

Insufficient workers. In Dataflow, processing latency increases most commonly due to insufficient workers, as the streaming pipeline cannot keep up with the incoming data rate when the number of Compute Engine instances is too low. This causes backpressure from Pub/Sub, leading to growing unacknowledged messages and higher end-to-end latency. Autoscaling may be delayed or limited by max worker count settings, making manual or configuration-based worker scaling the primary corrective action.

Answer

Pub/Sub subscription issue

Answer

Too many shards

Answer

Wrong machine type

Question 14

A company uses Vertex AI Pipelines to orchestrate ML workflows. They want to automatically retrain the model when new data arrives, but only if the model's performance drops below a threshold. Which approach is best?

Accepted Answer

Use Vertex AI Model Monitor to detect skew and trigger retraining. Vertex AI Model Monitor is specifically designed to detect prediction drift and data skew in deployed models. When the monitor identifies that model performance has dropped below a defined threshold, it can automatically trigger a retraining pipeline via a Cloud Function or Pub/Sub notification, ensuring retraining occurs only when necessary rather than on a fixed schedule.

Answer

Use BigQuery scheduled queries to trigger pipeline

Answer

Trigger a pipeline on a schedule

Answer

Use Cloud Functions to evaluate performance and trigger pipeline

Question 15

A company uses Vertex AI Feature Store for serving features to both training and prediction. The team notices that predictions made shortly after training use different feature values, causing a training-serving skew. What is the most effective way to prevent this skew?

Accepted Answer

Configure the Feature Store to use point-in-time lookup using the training timestamp. Point-in-time lookup ensures that feature values used during training are exactly the same as those used during prediction by retrieving the feature value as it existed at the training timestamp. This directly addresses training-serving skew caused by time-dependent feature changes, which is a common issue in Vertex AI Feature Store when features are updated after training.

Answer

Retrain the model more frequently to adapt to the new feature distributions

Answer

Use batch prediction instead of online prediction to ensure consistent features

Answer

Ensure that the training and prediction environments use identical compute resources