PDE Practice Test 12 — 15 Questions

Question 1

Your team is running a Dataflow streaming pipeline that reads from Pub/Sub, transforms data, and writes to BigQuery. You notice that the pipeline's backlog is growing and the processing latency has increased from seconds to minutes. You need to diagnose and resolve the issue. Which TWO actions should you take? (Choose two.)

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Increase the maximum number of workers in the pipeline's autoscaling configuration to allow more compute resources.. Option E is correct because the Dataflow monitoring dashboard exposes streaming-specific metrics such as system lag, data freshness, and worker throughput, which are exactly the signals needed to determine whether the growing backlog is caused by insufficient compute, a hot key, or a slow sink. Option D is correct because a streaming pipeline with a growing backlog and rising latency is typically under-provisioned, and raising the maximum number of workers in the autoscaling configuration lets Dataflow horizontally scale the worker pool to absorb the increased load. Option A is not appropriate because stopping and restarting the pipeline interrupts streaming state and is unnecessary when autoscaling can be adjusted on a running job. Option B is not appropriate because increasing the WriteToBigQuery batch size does not address a Pub/Sub backlog and can actually increase latency and memory pressure. Option C is not appropriate because a dead-letter queue for failed messages does not relieve the reprocessing load of successfully processed but backlogged messages.

Answer

Stop the pipeline, increase the number of workers in the streaming engine configuration, and restart it.

Answer

Increase the batch size in the WriteToBigQuery transform to reduce I/O operations.

Answer

Configure a dead-letter queue in Cloud Storage for failed messages to reduce reprocessing load.

Question 2

You are a data engineer at a global e-commerce company. Your team manages a real-time recommendation system that ingests user clickstream events from a Pub/Sub topic (topic-clickstream). The pipeline uses Dataflow to read events, join with user profile data from Cloud Bigtable, compute recommendations using a machine learning model hosted on Cloud Run, and write results to a BigQuery table for analytics. The pipeline has been running smoothly for months, but recently the Dataflow job started failing with the error: "Workflow failed. Causes: S01:ReadPubSub/Read+Transform/ParDo(ExtractUserID)+ ... (5a3b2c1d) The job failed because a worker encountered an out-of-memory error." The Dataflow job uses the Streaming Engine feature with a worker type of n2-standard-8 (8 vCPU, 32 GB memory) and autoscaling from 2 to 20 workers. The clickstream event rate has increased from 500 events/second to 5000 events/second over the past week. The user profile data in Bigtable has also grown, with average row size increasing from 1 KB to 10 KB due to additional fields. You need to resolve the out-of-memory errors without completely redesigning the pipeline. What should you do?

Accepted Answer

Change the worker machine type to n2-highmem-8 (8 vCPU, 64 GB memory) in the Dataflow job configuration.. The out-of-memory error is caused by the increased per-worker memory load from larger Bigtable rows (1 KB to 10 KB) and higher event throughput (500 to 5000 events/sec). Switching to n2-highmem-8 doubles the memory from 32 GB to 64 GB, giving each worker more headroom to cache user profiles and process larger batches without OOM. This directly addresses the root cause without redesigning the pipeline.

Answer

Increase the maximum number of workers in autoscaling from 20 to 50.

Answer

Reduce the batch size in the Dataflow pipeline by setting the `max_batch_size` parameter to a lower value.

Answer

Increase the number of Bigtable nodes to improve read throughput.

Question 3

You are building a real-time fraud detection system using Dataflow. Events from Pub/Sub need to be grouped by user_id within a 5-minute window to detect suspicious patterns. Some events may be delayed by up to 2 minutes. How should you configure the window and trigger to balance accuracy and latency?

Accepted Answer

Fixed window of 5 minutes with allowed lateness of 2 minutes and early trigger every 1 minute. It directly addresses both the accuracy requirement (handling up to 2 minutes of late data) and the latency requirement (emitting early results every 1 minute). A fixed 5-minute window groups events into non-overlapping intervals, which is appropriate for per-user fraud pattern detection. Setting allowed lateness to 2 minutes ensures that events delayed by up to 2 minutes are still included in the correct window, and an early trigger (e.g., repeating every 1 minute) provides low-latency partial results before the window closes. This combination balances completeness and timeliness.

Answer

Sliding window of 5 minutes with a 1-minute period and no allowed lateness

Answer

Session window with a gap duration of 5 minutes

Answer

Fixed window of 5 minutes with no allowed lateness and default trigger

Question 4

Which THREE steps are required to set up a continuous training pipeline on Google Cloud using Vertex AI?

Accepted Answer

Create a Vertex AI Pipeline to orchestrate data preprocessing, training, and model evaluation.. Option B is correct because a Vertex AI Pipeline is the native orchestration mechanism that chains the preprocessing, training, and evaluation components into a repeatable, versioned workflow, which is the backbone of a continuous training pipeline. Option C is correct because continuous training requires an automated trigger — Cloud Scheduler for time-based runs or Cloud Build (often via Eventarc/pub-sub on new data) — to invoke the pipeline without manual intervention. Option E is correct because continuous training must include automated model evaluation and promotion logic, such as comparing accuracy against a threshold and deploying to a Vertex AI endpoint only when the gate passes, enabling CI/CD-style model rollout. Option A is not appropriate because a single Compute Engine VM with cron is a manual, non-managed approach that bypasses Vertex AI's managed pipelines, scaling, and metadata tracking. Option D is not appropriate because manually uploading models to the Vertex AI Model Registry after each run defeats automation; the pipeline should register models programmatically as a step.

Answer

Run training on a single Compute Engine VM with a cron job.

Answer

Manually upload the model to Vertex AI Model Registry after each training run.

Question 5

A data engineer needs to split time-series data for training a forecasting model. The data is sorted by timestamp. The engineer wants to avoid leakage where future data influences training. Which data splitting approach should they use?

Accepted Answer

Use a time-series aware split: first 80% of data by timestamp for training, last 20% for testing. Time-series data has a temporal order, so training must only use data that precedes the test data to prevent future information from leaking into the model. Option D holds out the last 20% of records by timestamp for testing and trains on the earlier 80%, which mirrors real forecasting conditions where the model predicts unseen future values. This preserves causality and gives a realistic estimate of out-of-sample performance.

Answer

Use k-fold cross-validation with random assignment

Answer

Use stratified splitting on the target variable

Answer

Perform a random 80/20 split on the entire dataset

Question 6

A company runs a critical real-time data pipeline using Dataflow that ingests events from Cloud Pub/Sub, performs aggregations using sliding windows, and writes results to BigQuery. The pipeline is deployed in us-central1. The pipeline's latency has increased recently, and the Dataflow monitoring shows that the 'system lag' metric is consistently above 5 minutes. The pipeline is using Streaming Engine and has 10 workers with 4 vCPUs each. The pipeline processes approximately 100,000 events per second. The team has verified that the source Pub/Sub topic has sufficient publish throughput and the BigQuery table has no quota issues. The pipeline logs show that some workers are experiencing GC overhead limit exceeded errors. The pipeline code uses stateful processing with a custom keyed state for deduplication. What is the most likely cause of the increased latency?

Accepted Answer

The stateful processing is causing large state sizes that lead to GC overhead; use a more efficient state backend or increase worker memory.. The GC overhead limit exceeded errors indicate that workers are spending too much time garbage collecting, which is a classic symptom of excessive heap memory usage. Stateful processing with custom keyed state for deduplication can cause large per-key state sizes, especially with sliding windows that maintain overlapping state for each key. This forces the JVM to constantly garbage collect, increasing system lag beyond 5 minutes. Using a more efficient state backend (e.g., reducing state size or using Dataflow's built-in deduplication) or increasing worker memory directly addresses the root cause.

Answer

The number of workers is insufficient; increasing to 20 workers will reduce latency.

Answer

The sliding window duration is too long; reducing it to 1 minute will improve performance.

Answer

The deduplication logic is causing a bottleneck; removing it will reduce latency.

Question 7

A company is deploying a large-scale streaming application on Google Kubernetes Engine. They need to ensure the application can handle sudden traffic spikes without dropping data. Which architectural pattern is most appropriate?

Accepted Answer

Use a Pub/Sub topic as a buffer and autoscale consumer pods based on Pub/Sub subscription backlog.. Pub/Sub provides a durable, scalable, and asynchronous message buffer that decouples the producer from the consumer. By autoscaling consumer pods based on the Pub/Sub subscription backlog (e.g., using the 'pubsub.googleapis.com/subscription/num_undelivered_messages' custom metric with Horizontal Pod Autoscaler), the application can elastically handle traffic spikes without data loss, as messages are persisted until acknowledged.

Answer

Implement custom retry logic with exponential backoff in the application.

Answer

Use Cloud SQL as a temporary buffer and process from there.

Answer

Pre-provision 3x the expected peak capacity to handle spikes.

Question 8

A data science team has built a model using scikit-learn. They want to operationalize it on Google Cloud without rewriting the code. Which approach should they take?

Accepted Answer

Package the model in a custom container and deploy to Vertex AI Endpoints. Vertex AI Endpoints support custom containers, allowing you to package your scikit-learn model with its dependencies (e.g., a Flask or FastAPI inference server) and deploy it without rewriting any code. This approach directly meets the requirement to operationalize the existing model on Google Cloud without modification.

Answer

Export the model as a PMML file and use BigQuery ML

Answer

Use AI Platform Training to host the model directly

Answer

Convert the scikit-learn model to TensorFlow SavedModel format

Question 9

You are using Dataproc to run a Spark job that reads data from Cloud Storage, performs aggregations, and writes results back to Cloud Storage. The job is failing with out-of-memory errors on the shuffle. Which optimization should you apply?

Accepted Answer

Increase spark.sql.shuffle.partitions. Out-of-memory errors during a shuffle in Spark are usually caused by too few shuffle partitions, which makes each partition (and its in-memory sort/aggregation buffer) too large. Increasing spark.sql.shuffle.partitions splits the shuffle into more, smaller partitions, reducing per-task memory pressure and allowing the job to complete.

Answer

Use RDDs instead of DataFrames

Answer

Increase spark.executor.memory

Answer

Decrease the number of executors

Question 10

A company has a trained model stored in Vertex AI Model Registry. They want to automate retraining when new training data arrives in Cloud Storage. Which approach is most efficient?

Accepted Answer

Use Cloud Functions triggered by Cloud Storage events to start a Vertex AI Training job. Cloud Functions can be directly triggered by Cloud Storage events (e.g., object finalize) to invoke the Vertex AI Training service via the AI Platform API. This creates an event-driven, serverless pipeline that retrains the model immediately when new data arrives, without polling or manual intervention, making it the most efficient and cost-effective approach.

Answer

Use Dataflow to continuously update the model

Answer

Use Cloud Scheduler to trigger a Cloud Build retraining step

Answer

Schedule a weekly Cloud Composer DAG to check for new data and retrain

Question 11

A team has trained a scikit-learn model and wants to deploy it to AI Platform Prediction for online predictions. What is the required format for the model artifact?

Accepted Answer

A model.joblib file (or model.pkl) along with any custom code.. AI Platform Prediction (now Vertex AI) supports scikit-learn models natively. The required artifact format is a serialized model file (model.joblib or model.pkl) optionally accompanied by any custom code dependencies. This is because scikit-learn models are pickled objects, and the platform deserializes them using the same Python environment specified in the runtime version.

Answer

A single .h5 file containing the model weights.

Answer

A SavedModel directory containing the model for TensorFlow.

Answer

A model.pt file for PyTorch models.

Question 12

You need to split a time-series dataset into training and evaluation sets for a forecasting model. The data is ordered by timestamp. Which splitting technique should you use?

Accepted Answer

Sequential split where training data precedes evaluation data in time.. For time-series forecasting, the training set must only contain data that occurs before the evaluation set to prevent data leakage from the future into the past. A sequential split preserves the temporal order, ensuring the model is trained on historical data and evaluated on subsequent, unseen future data. This mimics real-world forecasting where you predict future values based on past observations. Random or stratified splits violate this principle by allowing future data points to influence training, leading to overly optimistic performance estimates.

Answer

Use k-fold cross-validation with random folds.

Answer

Stratified split based on the target variable.

Answer

Random split with 80% training, 20% evaluation.

Question 13

A company is designing a CI/CD pipeline for their ML models using Cloud Build and Vertex AI. Which TWO practices should they adopt to ensure reliable and reproducible deployments?

Accepted Answer

Use immutable container images with version tags for each model deployment. Option C is correct because using immutable container images with version tags ensures that each model deployment is tied to a specific, unchangeable artifact, which is essential for reproducibility and rollback in a CI/CD pipeline. Option D is correct because including unit tests for data preprocessing and feature engineering code in the pipeline catches errors early and verifies that the transformation logic behaves consistently, which directly supports reliable and reproducible ML deployments. Option A is not required for reproducibility and would slow the pipeline; manual approval is a governance choice, not a technical practice for reliable, reproducible deployment. Option B is wrong because storing artifacts in a single bucket without versioning prevents tracking and reproducing specific model versions. Option E is wrong because deploying every model version directly to production bypasses validation and testing, undermining reliability and reproducibility.

Answer

Require manual approval for every model change before deployment

Answer

Store all model artifacts in a single Cloud Storage bucket without versioning

Answer

Deploy every model version directly to production for immediate use

Question 14

Your team runs a weekly batch ETL pipeline using Cloud Dataproc. The pipeline reads raw data from Cloud Storage, transforms it with Apache Spark, and writes results to BigQuery. Recently, the pipeline has been failing with the error 'Out of Memory' during the shuffle phase. The cluster uses standard worker nodes (n1-standard-4). What is the most effective way to resolve this without increasing total cost?

Accepted Answer

Increase the number of Spark partitions by setting spark.sql.shuffle.partitions to a higher value.. The 'Out of Memory' error during the shuffle phase indicates that individual executor tasks are processing too much data per partition. Increasing `spark.sql.shuffle.partitions` reduces the amount of data each task handles, lowering memory pressure per executor without adding more nodes or upgrading hardware. This directly addresses the shuffle memory bottleneck while keeping the total cluster cost unchanged.

Answer

Increase the number of worker nodes by adding more n1-standard-4 instances.

Answer

Enable dynamic allocation and use preemptible VMs for some workers.

Answer

Switch worker nodes to n1-highmem-4 instances to provide more memory.

Question 15

You are designing a streaming Dataflow pipeline that processes high-throughput data. Which two features can help minimize cost? (Choose TWO.)

Accepted Answer

Enable autoscaling based on CPU utilization. Enabling autoscaling based on CPU utilization allows the Dataflow pipeline to dynamically adjust the number of worker instances in response to the actual processing load. This prevents over-provisioning during low-throughput periods, directly reducing compute cost while maintaining performance during spikes.

Answer

Use batch loads to BigQuery for streaming inserts

Answer

Use preemptible VMs for all workers

Answer

Use a global window and batch output to BigQuery every hour