PDE Designing data processing systems Practice Test 2 — 15 Questions

Question 1

A company runs a Cloud Dataflow streaming pipeline that reads from Cloud Pub/Sub, performs a fixed window of 10 seconds, joins with a slowly-changing dimension table stored in Cloud Bigtable, and writes results to BigQuery. The pipeline has been running for months but recently started exhibiting increasing latency and occasional data loss. The pipeline uses default settings with autoscaling enabled (min 2, max 20 workers). The Bigtable cluster has 3 nodes. The dimensions are updated infrequently. The latency has grown from seconds to minutes. Examining the Dataflow monitoring UI, you see that the 'System Lag' metric is increasing, and some windows are not being emitted. The CPU utilization on Bigtable nodes is below 50%. There are no errors in the logs. Which action is most likely to resolve the issue?

Accepted Answer

Set the pipeline option --maxNumWorkers to a value between 5 and 10.. The increasing system lag and unemitted windows in a streaming pipeline with autoscaling (2–20 workers) and a 3-node Bigtable cluster indicate that the pipeline is bottlenecked by the number of workers, not by Bigtable performance. With default autoscaling, Dataflow may not scale up aggressively enough to handle the sustained load, causing backlog and window expiration. Capping maxNumWorkers to 5–10 ensures sufficient parallelism without over-provisioning, allowing the pipeline to catch up and emit windows reliably.

Answer

Increase the window duration to 30 seconds to reduce the number of windows.

Answer

Redesign the pipeline to use a side input for the dimension table instead of a lookup.

Answer

Increase the number of Bigtable nodes to reduce lookup latency.

Question 2

A company uses Cloud Dataproc to run nightly Spark ETL jobs that process about 500 GB of data each night. The jobs currently take 4 hours to complete. The company wants to reduce the runtime to under 2 hours to meet a new SLA. The cluster is configured with 10 worker nodes (n1-standard-4) and 1 master node (n1-standard-4). The jobs are CPU-bound and use only default settings. The cluster is deleted after each job and recreated. The data is stored in Cloud Storage. The company is open to increasing cost but wants the most cost-effective solution to meet the SLA. Which approach should they take?

Accepted Answer

Increase the number of worker nodes to 20 and use preemptible VMs for half of them.. Option C is correct because adding more worker nodes (from 10 to 20) directly increases parallelism for CPU-bound Spark jobs, and using preemptible VMs for half of them reduces cost while still meeting the SLA. Since the job is CPU-bound and uses default settings, scaling horizontally with a mix of standard and preemptible VMs is the most cost-effective way to halve runtime, as Spark can efficiently distribute the workload across more cores.

Answer

Use a regional Cloud Storage bucket to improve read throughput.

Answer

Replace worker nodes with n1-highmem-16 instances to increase memory.

Answer

Change machine type to n2-standard-8 for all nodes.

Question 3

A company runs a batch ETL pipeline on Cloud Dataproc. During peak hours, the job takes longer than expected. The pipeline reads from Cloud Storage, transforms data, and writes to BigQuery. What is the most cost-effective way to improve performance without redesigning the pipeline?

Accepted Answer

Add a secondary worker group using preemptible VMs and increase the number of workers.. Adding a secondary worker group with preemptible VMs is the most cost-effective way to improve performance because it allows you to scale out the cluster horizontally with compute instances that are significantly cheaper (up to 80% discount) than regular VMs. This directly addresses the bottleneck of processing capacity during peak hours without requiring any pipeline redesign, as Cloud Dataproc can automatically distribute work across additional workers.

Answer

Enable local SSDs on all worker nodes.

Answer

Increase the master node's machine type to n1-highmem-32.

Answer

Use Cloud Composer to schedule the job with a higher priority.

Question 4

A retail company processes real-time clickstream data using Cloud Pub/Sub and Dataflow. The pipeline aggregates events by user session and writes to Bigtable for low-latency queries. However, users report that session data is sometimes missing or duplicated. What is the most likely cause?

Accepted Answer

Pub/Sub provides at-least-once delivery, and Dataflow does not deduplicate by default.. D is correct because Pub/Sub offers at-least-once delivery, meaning the same message may be delivered multiple times. Dataflow does not automatically deduplicate messages unless explicitly configured (e.g., using idempotent sinks or custom deduplication logic). Without deduplication, the same session event can be processed more than once, leading to duplicate session data in Bigtable.

Answer

Session windowing is configured with too short a gap duration.

Answer

Bigtable schema design causes row key collisions.

Answer

Dataflow's default behavior discards late-arriving data.

Question 5

A financial services firm processes sensitive transactions using Cloud Dataflow. The pipeline reads from Pub/Sub, performs stateful processing (e.g., fraud detection), and writes to Cloud Spanner. Compliance requires exactly-once processing semantics. Which configuration ensures exactly-once processing?

Accepted Answer

Set Dataflow pipeline to exactly-once mode and design Spanner writes to be idempotent.. Option C is correct because exactly-once processing in a Dataflow pipeline requires the pipeline itself to be set to exactly-once mode (which uses consistent snapshots and transactional sinks) and the output writes to Spanner to be idempotent. This combination ensures that even if a record is reprocessed due to failures, the final state in Spanner remains consistent, satisfying compliance requirements.

Answer

Configure Pub/Sub to use exactly-once delivery mode.

Answer

Use Pub/Sub with at-least-once delivery and Dataflow with at-least-once processing mode.

Answer

Enable Dataflow's streaming engine and use Spanner's built-in retry logic.

Question 6

A logistics company uses Cloud Functions to process incoming tracking events from IoT devices. Events are sent via HTTP triggers. During peak hours, some events fail with 500 errors. What is the best strategy to handle this reliably?

Accepted Answer

Use Cloud Pub/Sub as an intermediary: send events to Pub/Sub and trigger Cloud Functions via Pub/Sub subscription.. Option D is correct because Cloud Pub/Sub decouples event ingestion from processing, providing at-least-once delivery and built-in retry with exponential backoff. This ensures that HTTP 500 errors from Cloud Functions are automatically retried without data loss, even during peak loads, and the Pub/Sub subscription can be configured with a dead-letter queue for persistent failures.

Answer

Implement client-side retry with exponential backoff.

Answer

Increase the Cloud Functions timeout to 9 minutes and memory to 2GB.

Answer

Switch to Cloud Tasks and configure retry parameters.

Question 7

A media company ingests video files from partners via a REST API. Files are stored in Cloud Storage, and metadata is written to Firestore. A Cloud Function is triggered on object finalize to transcode video using Transcoder API. Sometimes, the function fails because the file is still being uploaded when triggered. How should this be fixed?

Accepted Answer

Use Cloud Pub/Sub notifications for Cloud Storage and trigger the function from the subscription.. Option D is correct because Cloud Storage object finalize notifications are sent only after the entire file has been written and committed. By using Pub/Sub notifications for Cloud Storage and triggering the Cloud Function from the subscription, you decouple the trigger from the upload process, ensuring the function only runs when the file is fully available. This eliminates the race condition where the function is triggered before the upload completes.

Answer

Implement a Cloud Composer workflow to poll for file existence.

Answer

Require partners to use resumable uploads.

Answer

Increase the Cloud Functions timeout to allow time for the upload to finish.

Question 8

A healthcare company streams patient monitoring data to Cloud Pub/Sub. A Dataflow pipeline reads the stream, enriches with patient records from BigQuery, and writes to Bigtable for real-time queries. The BigQuery lookup is slow and causes pipeline lag. What is the best approach to improve performance?

Accepted Answer

Use a side input from a BigQuery query with a global window and periodic refresh.. Option D is correct because using a side input from BigQuery with a global window and periodic refresh allows the Dataflow pipeline to cache the patient records in memory across all workers, avoiding per-element slow lookups. This pattern leverages Beam's side input semantics to broadcast a relatively static lookup table, significantly reducing latency compared to synchronous BigQuery queries for each incoming event.

Answer

Increase the number of Dataflow workers and use vertical scaling.

Answer

Use BigQuery's streaming read API in the pipeline.

Answer

Pre-join the data in a batch pipeline and load into Bigtable.

Question 9

A company uses Cloud Dataproc to run Spark ML jobs. The jobs are memory-intensive and often fail with OutOfMemory errors. Which action would most effectively reduce memory pressure without changing the Spark code?

Accepted Answer

Increase the number of worker nodes and reduce the number of cores per worker.. Increasing the number of worker nodes while reducing the number of cores per worker reduces memory pressure by distributing the workload across more JVMs, each with a smaller heap. This lowers the per-executor memory requirement and reduces the risk of OutOfMemory errors without modifying Spark code. In Cloud Dataproc, this approach directly addresses memory contention by giving each executor fewer tasks to process concurrently.

Answer

Increase the master node's memory.

Answer

Increase the number of Spark partitions.

Answer

Use preemptible VMs for workers.

Question 10

Which TWO statements are correct about designing a data pipeline using Cloud Dataflow for processing unbounded data?

Accepted Answer

Watermarks are used to measure the progress of event time.. Watermarks in Cloud Dataflow measure the progress of event time, indicating when all data up to a certain timestamp is expected to have arrived. This allows the pipeline to handle late-arriving data and determine when to close windows for unbounded data streams.

Answer

Triggers can only emit results at the end of a window.

Answer

Dataflow guarantees exactly-once processing for streaming pipelines.

Answer

Fixed windows are always based on processing time.

Question 11

Which THREE considerations are important when designing a data lake on Google Cloud using Cloud Storage?

Accepted Answer

Choose the appropriate storage class based on access patterns.. Option C is correct because selecting the appropriate storage class (e.g., Standard, Nearline, Coldline, Archive) based on data access patterns directly optimizes cost and performance in Cloud Storage. For a data lake, where data may be accessed frequently initially and rarely later, matching the storage class to the access pattern avoids paying premium rates for infrequently accessed data.

Answer

Use Cloud Storage's eventual consistency model for cost savings.

Answer

Define a schema when writing data to enforce data quality.

Question 12

Which TWO approaches are recommended for handling late-arriving data in a streaming Dataflow pipeline?

Accepted Answer

Use side inputs to provide default values for late data.. Option A is correct because side inputs in Apache Beam (the programming model underlying Dataflow) allow you to provide default values or supplementary data to handle late-arriving elements gracefully. When a late element arrives after the window has been emitted, a side input can supply a fallback value, ensuring the pipeline can still process the data without discarding it. This approach is recommended for handling late data in streaming pipelines where completeness is not critical.

Answer

Use fixed windows with a duration of 1 second to minimize lateness.

Answer

Set the trigger to fire only at the end of the window.

Answer

Use a filter transform to drop late-arriving elements.

Question 13

A multinational e-commerce company runs a real-time recommendation system. The architecture: user click events are sent via HTTP to a Cloud Run service, which publishes them to a Cloud Pub/Sub topic. A Dataflow streaming pipeline reads from the subscription, joins with user profile data from Firestore, computes recommendations using a TensorFlow model (loaded as a side input), and writes results to a Redis cache (Memorystore) for low-latency serving. The pipeline is deployed in us-central1. Recently, the team noticed that recommendation latency has increased from 50ms to 500ms, and the pipeline's backlog is growing. The Dataflow monitoring shows high CPU utilization on workers, and the SystemLag metric is 2 minutes and increasing. The Redis cluster shows no performance issues. The Firestore queries are within normal latency. The team suspects the TensorFlow model inference is the bottleneck. The model is a large neural network (500MB) loaded in each worker's memory. The pipeline uses 10 n1-standard-4 workers. The pipeline is using Dataflow's streaming engine. The team wants to reduce latency without increasing cost significantly. What should they do?

Accepted Answer

Increase the number of workers by adding a secondary worker group with preemptible VMs.. Option A is correct because adding preemptible VMs as a secondary worker group allows horizontal scaling at lower cost, distributing the TensorFlow model inference load across more workers. This reduces CPU utilization per worker and decreases the SystemLag without significantly increasing cost, as preemptible VMs are much cheaper than regular instances. The bottleneck is CPU-bound model inference, not memory, so more workers directly address the high CPU utilization and growing backlog.

Answer

Switch to a batch pipeline that runs every minute to reduce frequency of inference.

Answer

Increase the machine type of workers to n1-highmem-8 to provide more memory for the model.

Answer

Remove the model side input and call Cloud Run for inference using a separate service.

Question 14

Your company is building a real-time fraud detection system using Google Cloud. Transactions are streamed into Pub/Sub, and you need to process them with low latency (under 100ms per event) and aggregate data over sliding windows. Which Google Cloud service is best suited for this processing logic?

Accepted Answer

Dataflow. Dataflow is the best choice because it provides a unified stream and batch processing model with native support for Pub/Sub, exactly-once semantics, and low-latency sliding window aggregations. Its autoscaling and millisecond-level checkpointing enable sub-100ms per event processing, which is critical for real-time fraud detection.

Answer

BigQuery streaming inserts with scheduled queries

Answer

Dataproc with Spark Streaming

Answer

Cloud Functions

Question 15

Which TWO statements about designing a data processing pipeline on Google Cloud are correct? (Choose 2.)

Accepted Answer

Cloud Data Fusion allows you to build and manage data pipelines visually without writing code.. Cloud Data Fusion provides a visual, no-code interface for building and managing data pipelines, enabling users to design ETL/ELT workflows through a drag-and-drop UI. It abstracts the underlying complexity of Apache Spark and Cloud Dataproc, making it suitable for users who prefer a graphical approach over writing code.

Answer

Pub/Sub guarantees message ordering across all subscribers globally.

Answer

Cloud Bigtable is ideal for data warehousing and SQL analytics.

Answer

Dataproc is the best choice for fully managed data warehousing and analytics.