PDE Practice Test 9 — 10 Questions

Question 1

A company is building a real-time streaming pipeline using Pub/Sub and Dataflow to process clickstream data. The pipeline writes aggregated metrics to BigQuery every 10 seconds using a fixed window. During peak traffic, some windows produce duplicate rows in BigQuery. What is the most likely cause?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Dataflow is retrying BigQuery streaming inserts after a timeout, and the retries succeed even though the original insert succeeded.. Dataflow uses at-least-once semantics for streaming inserts into BigQuery. When a streaming insert times out, Dataflow retries the insert, and if the original insert actually succeeded but the acknowledgment was lost, the retry produces a duplicate row. This is a known behavior of BigQuery streaming inserts with retry logic.

Answer

The pipeline uses default triggers instead of after-watermark triggers.

Answer

The fixed window duration is too short, causing overlapping windows.

Answer

The pipeline is using too many Dataflow workers, causing load balancing issues.

Question 2

A data pipeline is built with Cloud Dataflow that reads from Pub/Sub, applies transformations, and writes to BigQuery. The pipeline is experiencing high latency and occasional data loss during worker failures. The engineer wants to improve reliability and performance. Which two actions should they take?

Accepted Answer

Enable Dataflow Streaming Engine and use BigQuery exactly-once sink. Enabling streaming engine moves the state management to the backend, reducing latency and improving reliability. Using exactly-once sinks (like BigQuery with exactly-once guarantees) prevents data loss.

Answer

Switch to Cloud Dataproc and use Spark Structured Streaming

Answer

Increase the number of workers and use at-least-once delivery to BigQuery

Answer

Use a global window and disable triggers

Question 3

A company wants to monitor the performance of a deployed model in production. Which metric indicates that the model's predictions are degrading?

Accepted Answer

Increase in prediction error rate. An increase in prediction error rate directly indicates that the model's outputs are deviating from the expected or ground-truth values, signaling degradation in predictive performance. This metric captures the core concept of model drift, where the statistical properties of the input data or the relationship between features and labels change over time, leading to less accurate predictions. In production ML monitoring, tracking error rate (e.g., classification accuracy, RMSE) is the primary method to detect when a model needs retraining or updating.

Answer

Increase in prediction latency

Answer

Decrease in throughput

Answer

Increase in number of requests

Question 4

You are a machine learning engineer at a FinTech company. Your team has developed a credit risk model using XGBoost and deployed it on Vertex AI Prediction using a custom container. The model is used for real-time credit decisions, and the endpoint is configured with a single machine type (n1-standard-4) and min_replica_count = 2, max_replica_count = 10. Recently, the team observed that during a promotional campaign, the endpoint's prediction latency increased from 200ms to over 2 seconds, and some requests resulted in 503 errors. You check the Cloud Monitoring metrics and see that CPU utilization reached 100% on the existing replicas, but the number of replicas never scaled beyond the initial 2. The deployment uses a custom container that runs a TensorFlow Serving-like model server. The container image is stored in Artifact Registry. The Vertex AI endpoint is configured with a traffic split of 100% to this model version. What is the most likely cause of the scaling failure, and what step should you take to resolve it?

Accepted Answer

Ensure the custom container exposes the correct metrics for CPU utilization so that Vertex AI autoscaling can trigger.. Vertex AI's autoscaling relies on the custom container exposing standard metrics (e.g., CPU utilization via the /metrics endpoint in a Prometheus format or through the Vertex AI custom metric adapter). If the container does not expose these metrics, the autoscaler cannot detect high CPU usage and will not trigger scaling beyond the initial replicas, leading to latency spikes and 503 errors under load.

Answer

Increase min_replica_count to 5 to handle the baseline load.

Answer

Change the endpoint configuration to use gRPC instead of HTTP to reduce latency.

Answer

Set the max_replica_count to a higher value like 20.

Question 5

You need to process a large volume of event data from Cloud Storage, apply complex transformations using Apache Spark, and then load the results into BigQuery. The data arrives in batches every hour. You want to minimize costs by using preemptible VMs. Which service should you use?

Accepted Answer

Dataproc. Dataproc supports preemptible (now called spot) VMs for cost savings. Dataflow does not support preemptible VMs for workers; it uses standard VMs. Cloud Composer is orchestration only. BigQuery is not for running Spark.

Answer

Cloud Composer

Answer

BigQuery

Answer

Dataflow

Question 6

A company is running a Cloud Dataflow streaming pipeline that aggregates events in 1-minute windows. They notice that the watermark is lagging significantly behind real-time. What is the most likely cause?

Accepted Answer

A hot key is causing data skew.. A hot key causes data skew, which means a disproportionate amount of data is assigned to a single key. In Cloud Dataflow, this leads to a single worker processing the bulk of the events, creating a processing bottleneck. The watermark, which tracks the progress of event-time processing, cannot advance until all data for a given window is processed, so the skewed key delays watermark progression significantly behind real-time.

Answer

The window duration is too short.

Answer

The pipeline was recently updated.

Answer

The allowed lateness is set too high.

Question 7

You are designing a near-real-time CDC pipeline to replicate changes from an on-premises PostgreSQL database to BigQuery for analytics. The source database has high transaction volume and you must ensure minimal impact on the source. Which Google Cloud service should you use to ingest the change data?

Accepted Answer

Datastream to stream changes to GCS, then load into BigQuery.. Datastream is purpose-built for CDC from MySQL, PostgreSQL, and Oracle to BigQuery or GCS. It reads the database logs (e.g., WAL) to capture changes with low latency and minimal impact on the source.

Answer

Pub/Sub with a custom connector that polls the database every minute.

Answer

Use BigQuery Data Transfer Service for PostgreSQL.

Answer

Use Dataflow with a JDBC IO connector to read from PostgreSQL.

Question 8

A data engineer is designing a streaming pipeline using Dataflow with Apache Beam. The pipeline reads from Pub/Sub, performs a stateful transformation (e.g., session windowing), and writes to BigQuery. The pipeline must handle late data and ensure exactly-once semantics. Which THREE configurations are required?

Accepted Answer

Set allowed lateness on the window to accommodate late data. Exactly-once sink (BigQuery) ensures no duplicates. Idempotent writes are built into Dataflow's BigQuery sink when using exactly-once mode. Setting allowed lateness handles late data. Watermark estimation is automatic; custom is not required. Triggers are optional; default trigger works. Windowing is inherent.

Answer

Use the File Loads write method for BigQuery

Answer

Use a custom watermark estimation function for Pub/Sub source

Question 9

A company runs a real-time anomaly detection system on Google Cloud. Streaming data from IoT devices is ingested via Pub/Sub, processed by Dataflow (Apache Beam), and results are written to Bigtable for low-latency serving. Recently, the system has been experiencing increased latency and occasional data loss. The Dataflow pipeline shows high system lag and backlog in Pub/Sub. The Bigtable cluster has 3 nodes and is reporting high CPU utilization (over 90%). The team suspects the issue is with the pipeline configuration. They have already verified that there are no errors in the pipeline code and no network issues. Which action should they take to resolve the issue?

Accepted Answer

Increase the number of Bigtable nodes to handle the write throughput.. The high CPU utilization on Bigtable (over 90%) indicates that the cluster is saturated and cannot keep up with the write throughput from Dataflow. This causes backpressure in the pipeline, leading to increased system lag and backlog in Pub/Sub, and eventually data loss when Pub/Sub messages expire. Increasing the number of Bigtable nodes directly addresses the bottleneck by distributing the write load and reducing CPU pressure, which allows the pipeline to drain the backlog and reduce latency.

Answer

Change the Dataflow worker machine type to n2-standard-8.

Answer

Decrease the batch size in the Dataflow pipeline to reduce latency.

Answer

Increase the number of Dataflow workers to process messages faster.

Question 10

A company wants to transfer 500 TB of data from an on-premises Hadoop cluster to Google Cloud Storage (GCS) for processing with Dataproc. The on-premises network has a 1 Gbps dedicated link to Google Cloud. The data must be transferred as quickly as possible, minimizing network usage. Which transfer method should they use?

Accepted Answer

Use Transfer Appliance to physically ship the data.. Transfer Appliance is the correct method because the dataset is 500 TB and the network link is only 1 Gbps. At 1 Gbps, the theoretical maximum transfer time is over 46 days, and real-world throughput (due to overhead, congestion, and Hadoop data characteristics) would be even longer. Transfer Appliance physically ships the data, bypassing the network bottleneck entirely and minimizing network usage, which is the stated requirement.

Answer

Use Storage Transfer Service over the 1 Gbps link.

Answer

Use gsutil cp in parallel with multiple threads.

Answer

Use BigQuery Data Transfer Service for Hadoop.