PDE Designing data processing systems Practice Test 4 — 15 Questions

Question 1

A BigQuery query fails with the error shown in the exhibit. What is the most likely cause?

Accepted Answer

The query scans too many partitions or data without efficient pruning. The error indicates that the query attempted to scan too many partitions or a large amount of data without effective partition pruning. BigQuery charges based on the amount of data processed, and queries that scan all partitions of a large table can hit limits or incur high costs. The most likely cause is that the query's WHERE clause does not filter on the partitioning column, forcing a full table scan across all partitions.

Answer

The table has too many partitions

Answer

The user does not have permission to query the table

Answer

Insufficient slot capacity in the project

Question 2

A Dataflow pipeline as described in the exhibit has increasing lag. Which optimization is most likely to reduce the lag?

Accepted Answer

Use FileLoads instead of StreamingInserts for BigQuery output. The exhibit shows increasing lag in a Dataflow pipeline writing to BigQuery. StreamingInserts (the default) use the BigQuery Storage Write API, which can throttle under high throughput, causing backpressure and lag. Switching to FileLoads writes data to temporary files in Cloud Storage and then loads them into BigQuery via batch load jobs, which decouples the write path from the streaming insert quota and reduces lag by avoiding per-row insert limits.

Answer

Increase the number of workers

Answer

Use global windows instead of fixed windows

Answer

Add additional ParDo transforms

Question 3

A company needs to process large files (100GB each) from Cloud Storage using Dataproc. They want to minimize job execution time. Which configuration is most appropriate?

Accepted Answer

Use a cluster with preemptible worker nodes and high-CPU machine types. Option B is correct because preemptible worker nodes are significantly cheaper than standard nodes, allowing you to scale out the cluster with many more workers for the same cost, which directly reduces job execution time for embarrassingly parallel data processing tasks. High-CPU machine types are ideal for compute-intensive Dataproc jobs like data transformation or machine learning, as they provide more vCPUs per core for parallel processing. This combination maximizes parallelism and minimizes wall-clock time for large-scale batch jobs.

Answer

Use a single-node cluster

Answer

Use HDFS for input data to avoid network latency

Answer

Use a cluster with many standard worker nodes

Question 4

A data pipeline uses Cloud Pub/Sub to ingest events and Cloud Functions to transform and write to BigQuery. The system is experiencing data loss during Pub/Sub subscription outages. Which design change improves reliability?

Accepted Answer

Use Dataflow with at-least-once delivery and checkpointing. Dataflow with at-least-once delivery and checkpointing ensures that messages are not lost during Pub/Sub subscription outages because Dataflow tracks processing progress via checkpoints and can replay unacknowledged messages from the last checkpoint. This decouples the processing from the subscription's transient failures, providing fault-tolerant, exactly-once or at-least-once semantics depending on the sink.

Answer

Use a pull subscription with a custom app that polls frequently

Answer

Use long ack deadlines to keep messages in the subscription

Answer

Increase the timeout in Cloud Functions

Question 5

A company wants to implement a near-real-time lake architecture using Cloud Storage and BigQuery. They need to enable queries on data within 5 minutes of arrival. Which approach meets the requirement with minimal operational overhead?

Accepted Answer

Use BigQuery Omni with external tables pointing to Cloud Storage. Option D is correct because BigQuery Omni with external tables can query data directly from Cloud Storage without loading. Option A is wrong because Cloud Storage FUSE adds a filesystem layer that may not be fast enough. Option B is wrong because streaming inserts into a separate table and then merging adds complexity and latency. Option C is wrong because scheduled batch loads have a minimum 10-minute interval, not meeting the 5-minute requirement.

Answer

Set up a Cloud Function to trigger BigQuery load jobs every 5 minutes

Answer

Use Cloud Storage FUSE to mount buckets and query with Spark on Dataproc

Answer

Stream data into a BigQuery table via streaming inserts, then use a scheduled query to merge into the main table

Question 6

A data engineer needs to design a batch pipeline that processes daily log files from Cloud Storage and writes aggregated results to BigQuery. Which service is most appropriate for this ETL job?

Accepted Answer

Dataproc with PySpark. Dataproc with PySpark is the most appropriate choice because it provides a managed Spark/Hadoop environment that can efficiently process large daily log files stored in Cloud Storage using distributed computing. PySpark's native integration with BigQuery via the Spark BigQuery connector allows direct writing of aggregated results, making it ideal for batch ETL workloads that require complex transformations and high throughput.

Answer

Cloud Pub/Sub with Cloud Functions

Answer

Cloud Composer

Answer

Cloud Data Fusion

Question 7

A company uses BigQuery to run reporting queries on a table that is partitioned by date and clustered by customer_id. Queries filtering by customer_id and a date range are performing poorly. What is the most likely cause?

Accepted Answer

The date range filter is too wide, causing scans of many partitions. Option D is correct because when a table is partitioned by date and clustered by customer_id, queries that filter on both columns can still perform poorly if the date range filter is too wide, causing BigQuery to scan many partitions. Even with clustering, scanning a large number of partitions negates the benefit of clustering, as clustering only reduces the data scanned within each partition. The query optimizer must read all partitions that fall within the date range, and if that range is broad, the scan overhead dominates.

Answer

The project lacks sufficient BigQuery slot capacity

Answer

The table is too large for BigQuery

Answer

Clustering column order should be date first, then customer_id

Question 8

A financial services company needs to process high-frequency trading data with strict ordering guarantees. They use Pub/Sub with ordering keys and Dataflow. The pipeline occasionally produces out-of-order results. What is the most likely cause?

Accepted Answer

The window trigger allows late data to be included after the main output. Option D is correct because Dataflow's default window trigger behavior allows late data to arrive after the main pane is emitted. When using Pub/Sub with ordering keys, late-arriving events (e.g., due to network delays or retries) can be assigned to the correct window but emitted in a separate pane, causing the final output to appear out-of-order relative to the event time. This is a known behavior when combining event-time windows with late data handling.

Answer

Dataflow does not preserve order when using multiple workers

Answer

Dataflow uses at-least-once processing, which can reorder events

Answer

Pub/Sub does not guarantee message ordering

Question 9

An e-commerce company processes real-time clickstream data using Pub/Sub and Dataflow. They want to ensure that if a Dataflow worker fails, the pipeline can resume processing from the point of failure without data loss. Which feature should they enable?

Accepted Answer

Snapshot-based recovery. Snapshot-based recovery (Option C) is the correct feature because Dataflow snapshots capture the entire pipeline state, including the current position in each Pub/Sub subscription and the state of all transforms. If a worker fails, the pipeline can be resumed from the exact snapshot point, ensuring no data loss and exactly-once processing semantics for the recovered data.

Answer

At-least-once delivery mode

Answer

Exactly-once processing mode

Answer

Streaming engine

Question 10

A financial services company uses Cloud Composer to orchestrate daily batch jobs. One job extracts data from MongoDB to Cloud Storage, then loads into BigQuery, and finally runs a Dataflow pipeline for aggregations. The Dataflow job fails intermittently. They want to automatically restart only the failed Dataflow job without re-running the earlier extraction and load. Which Airflow operator configuration should they use?

Accepted Answer

Set retries=2 on the Dataflow operator. Option C is correct because setting retries=2 on the Dataflow operator instructs Airflow to automatically restart only that specific task upon failure, without affecting upstream tasks (MongoDB extraction, BigQuery load). This isolates the retry to the Dataflow job, preserving the earlier completed work and avoiding redundant data movement.

Answer

Implement a SlaMiss sensor

Answer

Use a DAG with depends_on_past=True

Answer

Set trigger_rule='one_success' for downstream tasks

Question 11

A company uses Cloud Storage to store IoT sensor data in JSON format. The data is ingested using a Cloud Function triggered by Cloud Storage events. They notice that when many files are uploaded simultaneously, some files are not processed and the Cloud Function logs show 'function execution timeout'. What is the most likely cause and solution?

Accepted Answer

The Cloud Function's timeout is too short; increase timeout beyond 540 seconds.. The Cloud Function logs explicitly show 'function execution timeout', which indicates the function is exceeding its configured maximum runtime. The default Cloud Functions timeout is 60 seconds, and the maximum is 540 seconds (9 minutes). When many files are uploaded simultaneously, each function invocation may take longer due to increased processing load, causing timeouts. Increasing the timeout to the maximum of 540 seconds gives the function more time to complete processing, directly addressing the logged error.

Answer

The Cloud Function is not idempotent; implement idempotency.

Answer

The Cloud Storage event notification is unreliable; switch to Pub/Sub notifications.

Answer

The Cloud Function has too few instances; increase max instances.

Question 12

An online retailer uses BigQuery for analytics. They have a time-series table with 5 billion rows and new data arrives every day. They want to optimize query performance and reduce costs by ensuring that queries scan only the partitions they need. Which table design should they use?

Accepted Answer

Use a table partitioned on the timestamp column.. Partitioning on the timestamp column allows BigQuery to perform partition pruning, so queries with filters on that column only scan the relevant partitions. This directly reduces the amount of data read, lowering both query cost (pay-per-byte) and improving performance. For a 5-billion-row table with daily data arrival, time-unit partitioning is the standard design to meet the stated goals.

Answer

Use a table clustered on the timestamp column.

Answer

Use a table with no partitioning but use LIMIT in queries.

Answer

Use a table partitioned by ingestion time with a partition expiration.

Question 13

A data engineering team needs to process a large volume of CSV files stored in Cloud Storage using Dataproc. The files are generated hourly and each contains millions of rows. They want to minimize the number of Dataproc cluster nodes to reduce cost while processing within an hour. Which configuration should they recommend?

Accepted Answer

Use a cluster with many small worker nodes and use Spark dynamic allocation.. Option D is correct because using many small worker nodes with Spark dynamic allocation allows the cluster to scale resources precisely to the workload, minimizing idle capacity and cost. Dynamic allocation enables executors to be added or removed based on the processing demands of the hourly CSV files, ensuring the job completes within the hour without over-provisioning nodes.

Answer

Use a cluster with preemptible worker nodes only.

Answer

Use a cluster with local SSDs for temporary storage.

Answer

Use a cluster with a few large worker nodes and use Spark static allocation.

Question 14

A gaming company uses Pub/Sub to ingest player events and Dataflow for real-time analytics. They notice that the Pub/Sub subscription backlog is growing despite the Dataflow pipeline running continuously. The pipeline has a 1-hour window for aggregations. What is the most effective way to reduce the backlog?

Accepted Answer

Increase the Dataflow pipeline's worker count via autoscaling.. Increasing the Dataflow pipeline's worker count via autoscaling directly addresses the backlog by adding more parallel processing capacity to consume messages from the Pub/Sub subscription faster. Since the pipeline is continuously running but the backlog grows, the bottleneck is processing throughput, not pipeline availability. Autoscaling allows Dataflow to dynamically allocate more workers based on the backlog size, matching consumption rate to the incoming message rate.

Answer

Use a push subscription instead of pull.

Answer

Decrease the window duration to 10 minutes.

Answer

Enable Pub/Sub topic retention.

Question 15

A startup wants to build a data lake on Google Cloud using Cloud Storage. They need to store raw data in its original format for future analysis. Which storage class should they use to optimize for cost given that data will be accessed occasionally after the first month?

Accepted Answer

Nearline storage class. Nearline storage class is the optimal choice because it offers low-cost storage for data accessed less than once a month, with a 30-day minimum storage duration. Since the data is accessed occasionally after the first month, Nearline provides significant cost savings over Standard while still offering low-latency access (milliseconds) suitable for analytics. Coldline and Archive have lower storage costs but impose higher retrieval fees and minimum storage durations (90 and 365 days respectively), making them more expensive for data that is accessed occasionally within the first year.

Answer

Coldline storage class

Answer

Standard storage class

Answer

Archive storage class

PDE Designing data processing systems Practice Test 4 — 15 Questions

Exhibit

PDE Designing data processing systems Practice Test 4 — 15 Questions

Exhibit