PDE Designing data processing systems Practice Test 3 — 15 Questions

Question 1

Based on the exhibit, what is the most likely cause of the out-of-memory error?

Accepted Answer

The worker machine type has insufficient memory for the message size and throughput.. The out-of-memory error in a Dataflow pipeline is most likely caused by the worker machine type having insufficient memory for the message size and throughput. When messages are large or the throughput is high, each worker must hold data in memory for processing, windowing, and shuffling. If the worker's memory is too small, the JVM heap runs out of memory, leading to an OOM error.

Answer

The BigQuery output table schema does not match the transformed data, causing write failures.

Answer

The Pub/Sub subscription is not acknowledging messages quickly enough, causing a backlog.

Answer

The fixed window duration of 1 minute is too short, causing excessive state overhead.

Question 2

You are a data engineer at a global e-commerce company. Your team manages a real-time recommendation system that ingests user clickstream events from a Pub/Sub topic (topic-clickstream). The pipeline uses Dataflow to read events, join with user profile data from Cloud Bigtable, compute recommendations using a machine learning model hosted on Cloud Run, and write results to a BigQuery table for analytics. The pipeline has been running smoothly for months, but recently the Dataflow job started failing with the error: "Workflow failed. Causes: S01:ReadPubSub/Read+Transform/ParDo(ExtractUserID)+ ... (5a3b2c1d) The job failed because a worker encountered an out-of-memory error." The Dataflow job uses the Streaming Engine feature with a worker type of n2-standard-8 (8 vCPU, 32 GB memory) and autoscaling from 2 to 20 workers. The clickstream event rate has increased from 500 events/second to 5000 events/second over the past week. The user profile data in Bigtable has also grown, with average row size increasing from 1 KB to 10 KB due to additional fields. You need to resolve the out-of-memory errors without completely redesigning the pipeline. What should you do?

Accepted Answer

Change the worker machine type to n2-highmem-8 (8 vCPU, 64 GB memory) in the Dataflow job configuration.. Option B is correct because the out-of-memory error is caused by the increased per-worker memory load from larger Bigtable rows (1 KB to 10 KB) and higher event throughput (500 to 5000 events/sec). Switching to n2-highmem-8 doubles the memory from 32 GB to 64 GB, giving each worker more headroom to cache user profiles and process larger batches without OOM. This directly addresses the root cause without redesigning the pipeline.

Answer

Increase the maximum number of workers in autoscaling from 20 to 50.

Answer

Reduce the batch size in the Dataflow pipeline by setting the `max_batch_size` parameter to a lower value.

Answer

Increase the number of Bigtable nodes to improve read throughput.

Question 3

Drag and drop the steps to create a Cloud Storage bucket with uniform bucket-level access into the correct order.

Question 4

Drag and drop the steps to configure a VPC network with private Google access for on-premises connectivity using Cloud VPN into the correct order.

Question 5

Drag and drop the steps to set up a Pub/Sub topic with a push subscription to an HTTPS endpoint into the correct order.

Question 6

Match each Google Cloud data service to its primary use case.

Question 7

Match each Google Cloud service to its data processing capability.

Question 8

Match each Google Cloud monitoring/logging service to its function.

Question 9

A company uses Dataflow to process streaming data from Pub/Sub. They notice increased processing latency. What is the most likely cause?

Accepted Answer

Insufficient workers. In Dataflow, processing latency increases most commonly due to insufficient workers, as the streaming pipeline cannot keep up with the incoming data rate when the number of Compute Engine instances is too low. This causes backpressure from Pub/Sub, leading to growing unacknowledged messages and higher end-to-end latency. Autoscaling may be delayed or limited by max worker count settings, making manual or configuration-based worker scaling the primary corrective action.

Answer

Pub/Sub subscription issue

Answer

Too many shards

Answer

Wrong machine type

Question 10

A data pipeline uses Cloud Composer to orchestrate Dataflow and BigQuery jobs. The pipeline fails intermittently with dependency errors. Which design change can improve reliability?

Accepted Answer

Use retries with exponential backoff. Cloud Composer (Apache Airflow) tasks can fail due to transient issues like API rate limits or resource contention. Implementing retries with exponential backoff allows the DAG to automatically re-attempt failed tasks with increasing delays, reducing the impact of intermittent failures without manual intervention. This is a standard Airflow pattern for improving reliability in orchestrated pipelines.

Answer

Switch to Cloud Functions for orchestration

Answer

Increase worker count in Dataflow

Answer

Use a simpler DAG with fewer dependencies

Question 11

A company needs to process sensitive data in BigQuery with column-level security. They want to allow analysts to see aggregated data but not individual records. What approach?

Accepted Answer

Use authorized views with aggregation functions. Option C is correct because authorized views in BigQuery allow you to define SQL queries that aggregate data (e.g., using SUM, COUNT, AVG) and expose only the aggregated results to analysts, while hiding individual records. This approach enforces column-level security by granting access to the view rather than the underlying table, ensuring analysts cannot query the raw data directly. It meets the requirement of seeing aggregated data without seeing individual records, leveraging BigQuery's native authorization and SQL capabilities.

Answer

Use table-level access controls

Answer

Use column-level access controls with masking

Answer

Use Cloud Data Loss Prevention to de-identify data

Question 12

A company uses Dataproc for transient clusters. Which TWO actions can reduce costs?

Accepted Answer

Set cluster autoscaling to minimize idle resources. Option B is correct because Dataproc cluster autoscaling automatically adjusts the number of worker nodes based on the YARN memory and CPU utilization metrics. By scaling down during idle periods, you avoid paying for unused compute capacity, directly reducing costs for transient clusters that have variable workloads.

Answer

Increase master node size

Answer

Use standard VMs for all nodes

Answer

Use persistent clusters to avoid creation overhead

Question 13

A data engineer is migrating on-premises Hadoop jobs to Dataproc. Which TWO considerations are important?

Accepted Answer

Use Preemptible VMs for worker nodes to reduce cost. Options A and C are correct: use Cloud Storage instead of HDFS for scalability and cost, and use Preemptible VMs for transient tasks. Option B is wrong because HDFS is not recommended on Dataproc. Option D is wrong because master nodes should be on-demand for reliability. Option E is wrong because Cloud Storage connector is necessary.

Answer

Avoid using Cloud Storage connector to prevent overhead

Answer

Keep HDFS for better performance

Answer

Use on-demand VMs for master node to ensure availability

Question 14

A company building a real-time analytics pipeline with Pub/Sub and Dataflow. Which THREE best practices should they follow?

Accepted Answer

Use event time processing with watermarks and allowed lateness. Option A is correct because in streaming pipelines, event time processing with watermarks and allowed lateness is essential for handling out-of-order data. Watermarks track the progress of event time, and allowed lateness specifies how long to wait for late-arriving data before considering it as late, ensuring accurate windowed aggregations.

Answer

Use exactly-once processing for all transforms

Answer

Use event time processing only for batch pipelines

Question 15

A data engineer tries to grant a service account read access to a Cloud Storage bucket using the IAM policy above. The service account still cannot read objects. What is the most likely reason?

Accepted Answer

The condition prevents access because the request time is after 2023. Option B is correct because the IAM condition explicitly restricts access to requests made before January 1, 2023. Since the current time is after that date, the condition evaluates to false, denying the service account's read access regardless of the role binding. IAM conditions are evaluated at request time, and if the condition is not met, the permission is not granted.

Answer

The role does not include the necessary permission

Answer

The service account is misspelled

Answer

The role should be roles/storage.admin

PDE Designing data processing systems Practice Test 3 — 15 Questions

Exhibit

PDE Designing data processing systems Practice Test 3 — 15 Questions

Exhibit