PDE Practice Exam 57 — 20 Questions

Question 1

Your organization has a data lake on Cloud Storage with millions of small files (average 10 KB). You need to build a batch processing pipeline using Cloud Dataproc that runs a Spark job to transform the data and output results to BigQuery. The pipeline currently takes 4 hours to run because Spark spends a large amount of time listing files and managing tasks. You want to reduce the run time without changing the cluster size. Which action should you take?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Enable the Spark Dynamic Resource Allocation and combine small files using a separate job before the main transformation. The primary bottleneck is the overhead of listing millions of small files and managing many Spark tasks. By combining small files into larger ones using a separate job before the main transformation, you reduce the number of files Spark must list and the number of tasks required, which directly cuts the 4-hour runtime. Enabling Spark Dynamic Resource Allocation ensures resources are used efficiently during this preprocessing step without changing the cluster size.

Answer

Convert the input files from CSV to Parquet format

Answer

Use Spark coalesce to reduce the number of output partitions

Answer

Increase the number of Spark partitions to process more files in parallel

Question 2

A Dataflow streaming pipeline reads from Pub/Sub, applies a ParDo that uses a side input from a BigQuery table (refreshed hourly), and writes to BigQuery. The side input is large and causes increased latency and worker OOM errors. Which design change solves this?

Accepted Answer

Use a stateful ParDo and store the lookup data in an external cache like Cloud Bigtable, performing lookups per element.. Moving the large lookup data to an external cache like Cloud Bigtable offloads memory pressure from workers, eliminating OOM errors. The side input broadcast approach keeps the entire dataset in each worker's memory, which causes OOM when the data is large. Using an external cache allows per-element lookups without storing the entire dataset in memory, reducing latency by avoiding broadcast overhead.

Answer

Increase the side input broadcast frequency to update more often.

Answer

Split the pipeline into two: one to load the side input, the other to process main input.

Answer

Use smaller worker machine types to distribute memory across more workers.

Question 3

A retail company uses a Vertex AI endpoint to serve product recommendations. The model is a TensorFlow model deployed with a custom container. Recently, users have reported that recommendations are stale. The model is retrained daily using Vertex AI Pipelines. The pipeline completes successfully, but the endpoint continues to serve the old model. The team checks the pipeline logs and sees that the new model is uploaded to the Vertex AI Model Registry. The endpoint has traffic split set to 100% for the old model. The team needs to update the endpoint to serve the new model version. What should they do?

Accepted Answer

Update the endpoint to deploy the new model version from the registry and adjust traffic split. The pipeline successfully uploaded the new model to the Vertex AI Model Registry, but the endpoint still has its traffic split configured to 100% for the old model. To serve the new model, the team must explicitly update the endpoint to deploy the new model version from the registry and adjust the traffic split to route 100% of traffic to it. This is a standard operational step in Vertex AI: uploading a model does not automatically update the endpoint's deployment or traffic allocation.

Answer

Check the pipeline for errors in the deployment step

Answer

Re-upload the model with a different version ID

Answer

Redeploy the same model to the endpoint

Question 4

A gaming company uses Cloud Pub/Sub to ingest player activity events. A Dataflow streaming pipeline consumes these events, performs stateful processing to compute session metrics, and writes results to Cloud Bigtable for low-latency queries. Recently, the pipeline's processing latency increased, and the Bigtable write throughput dropped. Monitoring shows that the pipeline is experiencing a high rate of 'out-of-order' messages and 'duplicate' events. The Pub/Sub subscription is configured with exactly-once delivery. The Dataflow job uses a GlobalWindow with a trigger that fires every 10 seconds. What is the most likely cause and solution?

Accepted Answer

The pipeline's trigger is too frequent; increase the trigger interval to 30 seconds and set allowed lateness to 1 minute to handle out-of-order events.. The high rate of out-of-order and duplicate events indicates that the pipeline's trigger is firing too frequently, causing the stateful processing to attempt to commit partial windows before all events arrive. Increasing the trigger interval to 30 seconds and setting allowed lateness to 1 minute allows the pipeline to buffer more events, reduce the number of speculative triggers, and handle late-arriving data within the lateness bound, which directly reduces processing latency and Bigtable write contention.

Answer

The Bigtable instance is under-provisioned; add more nodes to increase write throughput.

Answer

Change the Pub/Sub subscription from exactly-once to at-least-once delivery to avoid redelivery overhead.

Answer

The streaming engine is disabled; enable Streaming Engine to reduce worker memory pressure.

Question 5

A company runs a real-time fraud detection model using Cloud Dataflow for streaming inference. The model is updated every hour with new training data. The team wants to minimize downtime and ensure that both old and new model versions are available during the update. Which deployment strategy should they use?

Accepted Answer

Canary deployment: deploy the new model alongside the old one, gradually increase traffic to the new model while monitoring.. Canary deployment is the correct strategy because it allows the new model to be deployed alongside the old one, with traffic gradually shifted to the new version while monitoring for errors or performance degradation. This minimizes downtime and ensures both versions are available during the update, which is critical for a real-time fraud detection system where continuous availability and risk mitigation are paramount.

Answer

A/B testing: route a small percentage of traffic to the new model and compare performance.

Answer

Rolling deployment: gradually replace instances of the old model with the new model.

Answer

Blue/green deployment: deploy the new model to a separate endpoint, then switch all traffic at once.

Question 6

A team wants to use Cloud Storage to build a data lake with separate zones for raw, curated, and processed data. They need to automatically move objects older than 30 days from the raw zone to a cheaper storage class. How can they achieve this?

Accepted Answer

Configure a Cloud Storage object lifecycle rule with SetStorageClass action. Cloud Storage object lifecycle management rules can automatically transition objects from one storage class to a cheaper one (e.g., from Standard to Nearline or Coldline) based on age. By configuring a rule with the `SetStorageClass` action and a `Condition` of `age: 30`, objects in the raw zone bucket older than 30 days are moved to a lower-cost class without manual intervention or additional compute services.

Answer

Set a bucket retention policy that forces deletion after 30 days

Answer

Write a Cloud Function to delete objects older than 30 days

Answer

Use gsutil rsync to move objects between buckets

Question 7

Your team is responsible for operationalizing a series of machine learning models that are trained and deployed using Vertex AI Pipelines. The pipeline consists of several steps including data preprocessing, training with hyperparameter tuning, model evaluation, and deployment to an endpoint. Recently, the pipeline has been failing intermittently at the model evaluation step with an error indicating insufficient memory. The evaluation step uses a custom container with a memory limit of 4 GB. The training step uses 8 GB and completes successfully. You need to resolve the failure without drastically increasing costs. What should you do?

Accepted Answer

Optimize the evaluation code to use streaming or incremental processing to reduce peak memory usage.

Answer

Increase the memory limit for the evaluation custom container to 8 GB to match the training step.

Answer

Reduce the batch size used in the evaluation step to lower memory consumption.

Answer

Use a smaller machine type for the evaluation step to force lower memory usage.

Question 8

Which THREE practices are recommended when designing a Cloud Data Fusion pipeline to ensure efficient execution and monitoring? (Choose three.)

Accepted Answer

Limit the memory and disk usage per stage to avoid Dataproc node resource exhaustion.. Cloud Data Fusion pipelines run on Dataproc clusters, and limiting memory and disk usage per stage prevents resource exhaustion on worker nodes. This ensures that no single stage consumes all available resources, which could cause the pipeline to fail or degrade performance. Proper resource limits help maintain stable execution and avoid out-of-memory errors.

Answer

Manually partition input files to control parallelism.

Answer

Schedule pipeline runs using Cloud Scheduler and Pub/Sub triggers to avoid manual starts.

Question 9

A financial institution needs to deploy a TensorFlow model for fraud detection with strict latency requirements (<100ms). The model uses custom ops that are not available in standard TF Serving. What is the most appropriate serving solution?

Accepted Answer

Use NVIDIA Triton Inference Server with a custom backend. NVIDIA Triton Inference Server supports custom backends written in C++ or Python, allowing the integration of custom ops that are not available in standard TensorFlow Serving. This enables the model to meet strict latency requirements (<100ms) by leveraging GPU acceleration and optimized inference pipelines, while avoiding the limitations of TF Serving's fixed op registry.

Answer

Export the model as a SavedModel and serve on Vertex AI Prediction

Answer

Use Cloud Run with a custom container that includes the model and pre-loads the library

Answer

Package the model with Docker using TF Serving and add custom ops via TensorFlow's custom op registration

Question 10

A Dataflow pipeline reads from Cloud Pub/Sub and writes to Cloud Storage. The pipeline needs to guarantee exactly-once processing despite worker failures. Which configuration ensures exactly-once semantics?

Accepted Answer

Enable exactly-once delivery on the Pub/Sub subscription and use an idempotent sink. Pub/Sub subscriptions can be configured with exactly-once delivery (using the `enableExactlyOnceDelivery` flag), which ensures that each message is delivered to the subscriber exactly once. Combining this with an idempotent sink (e.g., Cloud Storage with unique filenames or deduplication logic) guarantees that even if a worker fails and the pipeline retries, the output will not contain duplicates. This is the only option that directly addresses both the source and sink to achieve end-to-end exactly-once semantics.

Answer

Use a side input from a deduplication dataset

Answer

Set the pipeline to use a global window with no early triggers

Answer

Insert a Reshuffle transform after reading

Question 11

A data engineer needs to design a data processing system that ingests large volumes of sensor data from IoT devices. The data should be stored in a schema-less format and allow for real-time analytics. Which Google Cloud service is most appropriate?

Accepted Answer

Cloud Bigtable. Cloud Bigtable is the most appropriate choice because it is a fully managed, scalable NoSQL database designed for large-scale analytical and operational workloads. It supports schema-less storage of time-series sensor data and integrates with real-time analytics tools like BigQuery and Dataflow via the HBase API, meeting the requirements for high-throughput ingestion and low-latency queries.

Answer

Cloud Spanner

Answer

Firestore

Answer

Cloud SQL

Question 12

A data engineer is running a Dataproc cluster for a batch ETL job that needs to process 10 TB of data. The job is memory-intensive. The cluster currently uses n1-standard-4 workers. Performance is poor. What is the most cost-effective change to improve performance?

Accepted Answer

Use high-memory machine types (n1-highmem-4). The job is memory-intensive, and n1-standard-4 workers have 15 GB of RAM, which may be insufficient for the workload, causing excessive disk spill or OOM errors. Switching to n1-highmem-4 provides 26 GB of RAM per worker (a 73% increase) without increasing vCPU count, directly addressing the memory bottleneck at a lower cost than adding more workers. This is the most cost-effective change because it improves performance without incurring the overhead of additional vCPUs or licensing costs.

Answer

Use preemptible workers to reduce cost

Answer

Switch to n2-standard-4 machine types

Answer

Add more n1-standard-4 workers

Question 13

A data engineer needs to process streaming data from thousands of IoT devices and generate real-time dashboards. The data volume is low but requires exactly-once processing semantics. Which Google Cloud service combination should they use?

Accepted Answer

Cloud Pub/Sub + Cloud Dataflow. Dataflow supports exactly-once processing via its streaming engine and checkpointing. Pub/Sub is the ingest service. Together they provide the required semantics.

Answer

Cloud Pub/Sub + Cloud Data Fusion

Answer

Cloud Pub/Sub + Cloud Dataproc

Answer

Cloud Pub/Sub + Cloud Dataprep

Question 14

Refer to the exhibit. A Dataflow pipeline writes to BigQuery table employee_records. The pipeline was working yesterday but fails today. What is the most likely cause?

Accepted Answer

The pipeline code was changed to send an integer for the last_name field.. If the pipeline code was changed to send an integer for the last_name field, BigQuery will reject the write due to a schema mismatch. BigQuery enforces strict type checking at ingestion time; an integer value cannot be written into a STRING column unless the schema explicitly allows coercion. Since the pipeline was working yesterday, the most likely change is in the data type being sent, not the schema itself.

Answer

The pipeline dropped the last_name field entirely.

Answer

The BigQuery table quota was exceeded.

Answer

The BigQuery table schema was changed from STRING to INTEGER for last_name.

Question 15

A company wants to ensure high availability for their Cloud SQL instance. Which TWO actions are most appropriate? (Choose two.)

Accepted Answer

Create a read replica in a different region.. Options A and C are correct. A read replica in a different region provides geographic redundancy and can be promoted to a primary in case of a regional failure, enhancing availability. Enabling automatic backups with a 7-day retention allows point-in-time recovery, which helps restore data after corruption or accidental deletion, contributing to overall availability. Option B (failover replica in same region) is typically used for high availability within a region but does not protect against regional outages, so it is not as appropriate for the scenario. Option D (increasing memory and storage) does not directly affect availability. Option E (horizontal scaling with read replicas) provides read scaling but not write failover.

Answer

Configure a failover replica in the same region.

Answer

Increase the instance's memory and storage size.

Answer

Set up horizontal scaling with multiple read replicas.

Question 16

Which TWO actions can help reduce the latency of a Vertex AI endpoint serving a large neural network model?

Accepted Answer

Deploy the model on a machine type with GPU accelerators. GPU accelerators are specifically designed to handle the parallel computations required by large neural networks, significantly reducing inference latency compared to CPU-only machines. Vertex AI endpoints with GPUs can process multiple predictions concurrently, which is critical for deep learning models where matrix operations dominate the workload.

Answer

Use a larger machine type with more CPU cores

Answer

Enable model compression with quantization

Answer

Increase the number of model versions deployed on the same endpoint

Question 17

A multinational e-commerce company runs a real-time recommendation system. The architecture: user click events are sent via HTTP to a Cloud Run service, which publishes them to a Cloud Pub/Sub topic. A Dataflow streaming pipeline reads from the subscription, joins with user profile data from Firestore, computes recommendations using a TensorFlow model (loaded as a side input), and writes results to a Redis cache (Memorystore) for low-latency serving. The pipeline is deployed in us-central1. Recently, the team noticed that recommendation latency has increased from 50ms to 500ms, and the pipeline's backlog is growing. The Dataflow monitoring shows high CPU utilization on workers, and the SystemLag metric is 2 minutes and increasing. The Redis cluster shows no performance issues. The Firestore queries are within normal latency. The team suspects the TensorFlow model inference is the bottleneck. The model is a large neural network (500MB) loaded in each worker's memory. The pipeline uses 10 n1-standard-4 workers. The pipeline is using Dataflow's streaming engine. The team wants to reduce latency without increasing cost significantly. What should they do?

Accepted Answer

Increase the number of workers by adding a secondary worker group with preemptible VMs.. Adding preemptible VMs as a secondary worker group allows horizontal scaling at lower cost, distributing the TensorFlow model inference load across more workers. This reduces CPU utilization per worker and decreases the SystemLag without significantly increasing cost, as preemptible VMs are much cheaper than regular instances. The bottleneck is CPU-bound model inference, not memory, so more workers directly address the high CPU utilization and growing backlog.

Answer

Switch to a batch pipeline that runs every minute to reduce frequency of inference.

Answer

Increase the machine type of workers to n1-highmem-8 to provide more memory for the model.

Answer

Remove the model side input and call Cloud Run for inference using a separate service.

Question 18

Your company runs a batch data processing pipeline using Cloud Dataproc and Cloud Composer. The pipeline processes hundreds of terabytes of data daily. Recently, the pipeline has been failing intermittently due to Dataproc cluster creation errors: 'Insufficient resources to create cluster in zone us-central1-f.' The project has a global quota of 1000 vCPUs for Compute Engine. The team usually uses n2-standard-8 (8 vCPU) worker nodes. You notice that the error occurs during peak usage times. You need to ensure the pipeline runs reliably without increasing the global quota. Which action should you take?

Accepted Answer

Configure the Dataproc cluster to use multiple zones via the --zone argument with a zonal list. Configuring the Dataproc cluster to use multiple zones via the `--zone` argument with a zonal list distributes worker node creation across several zones in the same region. This avoids the 'Insufficient resources' error by not exhausting capacity in a single zone, without requiring a global quota increase. Cloud Dataproc supports specifying a comma-separated list of zones, and the service will attempt to create the cluster in the first available zone.

Answer

Increase the global Compute Engine quota to 2000 vCPUs

Answer

Switch to using preemptible VMs only, which have higher availability

Answer

Use fewer workers with larger machine types, such as n2-standard-64

Question 19

Which Google Cloud service is a serverless, highly scalable data warehouse for analytical queries, supporting SQL and integration with BI tools?

Accepted Answer

BigQuery. BigQuery is a serverless, highly scalable data warehouse designed for analytical queries over large datasets. It supports standard SQL and integrates seamlessly with BI tools like Looker and Tableau, making it the correct choice for this use case.

Answer

Firestore

Answer

Cloud SQL

Answer

Cloud Spanner

Question 20

You are designing a row key for Cloud Bigtable to store user activity logs. Each log entry has a timestamp (millisecond precision) and a user ID. There will be millions of writes per second from many users. To avoid hotspotting, which row key design is BEST?

Accepted Answer

hash(userID)#userID#timestamp_millis. Best because it uses a hash of the user ID as the row key prefix, which distributes writes across all Bigtable nodes and avoids hotspotting. Appending the user ID and timestamp ensures uniqueness and supports efficient queries for a specific user's logs. This design prevents the sequential timestamp from creating a single hot node, which is critical for handling millions of writes per second.

Answer

timestamp_millis#hash(userID)

Answer

timestamp_millis#userID

Answer

userID#timestamp_millis