PDE Building and operationalizing data processing systems Practice Test 5 — 15 Questions

Question 1

You are optimizing a Dataflow pipeline that performs a group-by-key transformation on a large, skewed dataset. The pipeline is experiencing high latency due to data skew (some keys have many more values). Which TWO actions can help mitigate the skew? (Choose two.)

Accepted Answer

Use hot key detection and split the hot key into multiple sub-keys (e.g., append a random number).. Option A is correct because splitting a hot key into multiple sub-keys (e.g., by appending a random number) distributes the values across multiple shards during the shuffle phase, reducing the load on any single worker. This technique, often called "salting," is a standard pattern in Dataflow and Apache Beam to handle data skew by breaking the bottleneck caused by a single key with disproportionately many values.

Answer

Enable the Dataflow service's automatic reshuffling feature.

Answer

Use CoGroupByKey to reduce the number of keys.

Answer

Increase the number of worker machines.

Question 2

Your company is building a data processing system that ingests sensor data from millions of devices, processes it in near real-time to detect anomalies, and stores raw and processed data for long-term analytics. The system must meet a 99.9% uptime SLA and minimize data loss. Which THREE design choices are best? (Choose three.)

Accepted Answer

Use Cloud Pub/Sub as the ingestion layer with a dead-letter topic to capture unprocessed messages.. Cloud Pub/Sub with a dead-letter topic ensures that messages that cannot be processed are captured and not lost, directly supporting the requirement to minimize data loss. The dead-letter topic allows for later reprocessing or analysis of failed messages, which is critical for meeting a 99.9% uptime SLA by preventing message backlogs from blocking the ingestion pipeline.

Answer

Store raw data in Cloud Bigtable and processed data in Cloud Storage.

Answer

Use a global Cloud Load Balancer in front of the Dataflow workers.

Question 3

Your company is evaluating managed messaging services for a new event-driven application. The application requires pub/sub semantics, high throughput (millions of messages per second), and integration with Google Cloud services like Cloud Functions and Dataflow. Which TWO services should you consider? (Choose two.)

Accepted Answer

Cloud Pub/Sub. Cloud Pub/Sub (A) is the correct choice because it provides fully managed, highly scalable pub/sub messaging with exactly-once delivery semantics and support for millions of messages per second. It integrates natively with Cloud Functions and Dataflow, making it ideal for event-driven architectures requiring high throughput and decoupled communication.

Answer

Cloud Scheduler

Answer

Cloud Functions

Answer

Cloud Tasks

Question 4

A company runs a Dataflow pipeline that reads from Pub/Sub, aggregates events in a 10-minute fixed window, and writes to BigQuery. Recently, the pipeline has been failing with 'high uncommitted bytes' errors during periods of high traffic. What is the most likely cause and recommended action?

Accepted Answer

Use a global window with a trigger that fires early based on element count to reduce the number of open windows.. The 'high uncommitted bytes' error in Dataflow occurs when the system holds too much data in memory across many open windows, exceeding the default 200 MB limit. Using a global window with an early trigger based on element count reduces the number of simultaneous open windows and allows data to be committed more frequently, preventing memory pressure. This approach is recommended over reducing window size or scaling workers because the root cause is window fan-out, not throughput or parallelism.

Answer

Reduce the window size from 10 minutes to 1 minute to decrease the amount of data per window.

Answer

Increase the number of worker machines to handle higher throughput.

Answer

Set a maximum number of workers and use a Pub/Sub flow control setting to limit incoming messages.

Question 5

A data engineer needs to process large CSV files (hundreds of GB) stored in Cloud Storage using Spark on a Dataproc cluster. The job performs a series of transformations and aggregations. Which configuration is most cost-effective and operationally efficient?

Accepted Answer

Use a cluster with a standard master node and 10 preemptible worker nodes (n1-standard-4).. Option B is correct because preemptible workers are significantly cheaper (about 80% discount) and ideal for batch processing of large CSV files where fault tolerance is built into Spark via RDD lineage. Using standard nodes for the master ensures cluster stability, while preemptible workers handle the distributed transformations and aggregations cost-effectively. This configuration balances cost and operational efficiency for ephemeral, fault-tolerant workloads.

Answer

Use a cluster with 10 high-memory (n1-highmem-8) VMs as workers to improve shuffle performance.

Answer

Use a single-node cluster with a high-memory machine type.

Answer

Use a cluster with 10 standard (n1-standard-4) VMs as master and worker nodes, all non-preemptible.

Question 6

A company uses Cloud Composer (Airflow) to orchestrate a data pipeline. One DAG has many tasks that run in parallel and dependencies that span multiple days. Recently, the DAG started failing with 'DagRun already exists' errors. What is the most likely cause?

Accepted Answer

The DAG has max_active_runs_per_dag set to a low number, causing overlapping runs to be rejected.. The 'DagRun already exists' error occurs when Airflow attempts to create a new DAG run for a logical date that already has an active or completed run, and the DAG's concurrency settings prevent overlapping runs. Setting max_active_runs_per_dag to a low number (e.g., 1) restricts the number of concurrent runs, so if a previous run hasn't finished or been cleared, a new run for the same or overlapping schedule interval is rejected with this error. This is the most likely cause given the DAG has dependencies spanning multiple days, which can cause runs to overlap if not properly configured.

Answer

The DAG has a large number of tasks, overwhelming the Airflow scheduler.

Answer

The DAG's schedule interval is too short, causing task instances to be created with duplicate run IDs.

Answer

The DAG has a depends_on_past set to True, causing upstream failures to block new runs.

Question 7

A team wants to ingest streaming data from millions of IoT devices and store historical data in BigQuery for analysis. They need near real-time analytics on the most recent data, with sub-second latency. Which architecture should they use?

Accepted Answer

Use Pub/Sub, then a Dataflow pipeline that filters and transforms data, writing to Cloud Bigtable for real-time queries and to Cloud Storage for periodic BigQuery loads.. Option B is correct because it uses Cloud Bigtable for sub-second latency on recent data, which is ideal for near real-time analytics on streaming IoT data. Dataflow provides the necessary stream processing, filtering, and transformation before writing to Bigtable for low-latency queries and to Cloud Storage for periodic batch loads into BigQuery for historical analysis. This architecture decouples real-time and historical paths, meeting both latency and storage requirements.

Answer

Use Pub/Sub to receive data, then stream directly into BigQuery using the streaming API, and use standard SQL queries for real-time analytics.

Answer

Use Pub/Sub to ingest data into a Dataproc Spark Streaming job that writes to both Bigtable and BigQuery.

Answer

Use Cloud SQL to store the latest data and periodically move historical data to BigQuery via cron jobs.

Question 8

A data engineer needs to design a batch processing pipeline using Cloud Data Fusion. The pipeline should read data from Cloud Storage, perform transformations (join, filter, aggregate), and write to BigQuery. What is the most efficient way to handle the transformations?

Accepted Answer

Use Data Fusion Wrangler to visually design the transformations and then run the pipeline on a Dataproc cluster.. Option A is correct because Cloud Data Fusion Wrangler provides a visual, no-code interface for designing transformations (join, filter, aggregate) that are then compiled into an Apache Spark or MapReduce program and executed on a Dataproc cluster. This approach leverages Data Fusion's native integration with Dataproc for efficient, scalable batch processing without manual coding, while keeping the pipeline fully managed within the Data Fusion ecosystem.

Answer

Use SQL queries in BigQuery to perform the transformations after loading raw data into staging tables.

Answer

Use custom Python scripts in a Cloud Function triggered after the files land in Cloud Storage.

Answer

Use Apache Spark on Dataproc to code the transformations manually, bypassing Data Fusion.

Question 9

A company's Dataflow pipeline uses the PubSubIO source to read messages and writes to BigQuery via the BigQueryIO sink. The pipeline is running in Streaming mode with exactly-once semantics enabled. Occasionally, duplicate rows appear in BigQuery. What is the most likely reason?

Accepted Answer

The user-provided record ID for deduplication in BigQuery's streaming inserts is not being set for all messages, leading to duplicate rows.. In Dataflow streaming pipelines with exactly-once semantics, BigQuery's streaming inserts use user-provided record IDs for deduplication. If the record ID is not set for all messages, BigQuery cannot identify duplicates, and retries or redeliveries from Pub/Sub can result in duplicate rows. This is the most common cause of duplicates in this scenario.

Answer

The pipeline is using the WriteResult method with WRITE_APPEND in batch mode, which can cause duplicates if retries happen.

Answer

The pipeline is experiencing the 'dataflow streaming log processing' bug, causing duplicate logs to be written.

Answer

The PubSubIO source is configured with a dead-letter queue and messages are being redelivered without proper deduplication.

Question 10

A retail company is building a recommendation engine that requires processing customer clickstream data in near real-time. The data is ingested via Pub/Sub, and must be joined with a lookup table of product details (updated daily) before being used for model inference. Which design pattern should they use?

Accepted Answer

Use a Dataflow pipeline that reads from Pub/Sub and uses a side input from a regularly refreshed PCollection loaded from Cloud Storage.. Option B is correct because Dataflow can read streaming data from Pub/Sub and use a side input from a regularly refreshed PCollection loaded from Cloud Storage. This pattern allows the product lookup table (updated daily) to be periodically reloaded into the pipeline as a side input, enabling efficient, low-latency enrichment of each event without per-event external calls or batch delays.

Answer

Enrich the stream by querying BigQuery for each event using a Cloud Function.

Answer

Store product details in Cloud Memorystore (Redis) and have the streaming application look up each event.

Answer

Write events to BigQuery and use scheduled queries to join with the product table in batch.

Question 11

A team is setting up a Dataflow pipeline for a time-sensitive ETL job that must complete within a specific time window. Which monitoring metric should they use to determine if the pipeline is on track to finish on time?

Accepted Answer

The job's estimated time to completion shown in the Dataflow monitoring interface.. Option D is correct because the Dataflow monitoring interface provides an estimated time to completion for the pipeline, which is the most direct metric for determining if the job will finish within the required time window. This estimate is calculated based on current throughput, backlog, and resource utilization, making it the appropriate choice for time-sensitive ETL jobs. Other metrics like system lag or element counts do not directly predict job completion time.

Answer

The number of failed elements and retries.

Answer

The system lag metric, which measures the time between event occurrence and processing.

Answer

The number of elements processed in the current window.

Question 12

An organization uses Cloud Dataproc to run Spark jobs that process sensitive data. They need to ensure data is encrypted at rest and that only specific service accounts can access the data on cluster disks. What should they do?

Accepted Answer

Use customer-managed encryption keys (CMEK) for the cluster's persistent disks and assign a dedicated service account to the cluster with minimal IAM roles.. Option D is correct because using customer-managed encryption keys (CMEK) allows the organization to control and manage the encryption keys for persistent disks attached to the Dataproc cluster, ensuring data at rest is encrypted. Assigning a dedicated service account with minimal IAM roles ensures that only that service account can access the data on the cluster disks, following the principle of least privilege.

Answer

Rely on the default encryption at rest and use VPC Service Controls to limit data exfiltration.

Answer

Use customer-supplied encryption keys (CSEK) and write a startup script to mount encrypted disks.

Answer

Enable encryption at rest using Google-managed encryption keys and grant all users the Dataproc Editor role.

Question 13

Which TWO actions should be taken to optimize a Dataflow streaming pipeline that is experiencing high system lag and backpressure? (Choose two.)

Accepted Answer

Increase the number of worker threads by adjusting the streaming worker's parallelism hint.. Option B is correct because increasing the parallelism hint allows each worker to process more bundles concurrently, which can reduce backpressure by improving throughput without adding more workers. Option C is correct because enabling autoscaling and increasing the maximum number of workers allows the pipeline to dynamically scale out to handle increased load, directly mitigating high system lag and backpressure.

Answer

Use a higher memory machine type for all workers.

Answer

Reduce the number of workers to decrease cost.

Answer

Set maxNumWorkers to 1 to force single-worker processing.

Question 14

Which THREE features of Cloud Pub/Sub guarantee at-least-once delivery and enable exactly-once processing downstream? (Choose three.)

Accepted Answer

Subscriber-retry policy with exponential backoff.. Option A is correct because a subscriber-retry policy with exponential backoff ensures that messages that fail to be processed are retried with increasing delays, preventing transient failures from causing message loss. This mechanism, combined with Pub/Sub's persistent storage, guarantees that each message is delivered at least once, as the subscriber will keep retrying until it acknowledges the message.

Answer

Message ordering by message key.

Answer

Cloud Dataproc integration for message replay.

Question 15

Which THREE practices are recommended when designing a Cloud Data Fusion pipeline to ensure efficient execution and monitoring? (Choose three.)

Accepted Answer

Limit the memory and disk usage per stage to avoid Dataproc node resource exhaustion.. Option B is correct because Cloud Data Fusion pipelines run on Dataproc clusters, and limiting memory and disk usage per stage prevents resource exhaustion on worker nodes. This ensures that no single stage consumes all available resources, which could cause the pipeline to fail or degrade performance. Proper resource limits help maintain stable execution and avoid out-of-memory errors.

Answer

Manually partition input files to control parallelism.

Answer

Schedule pipeline runs using Cloud Scheduler and Pub/Sub triggers to avoid manual starts.