PDE Designing data processing systems Practice Test 9 — 15 Questions

Question 1

A data engineer is designing a BigQuery table for time-series data that will be queried frequently by time range and also by a customer_id. Which TWO design decisions will improve query performance and manage costs? (Choose two.)

Accepted Answer

Partition the table by day on the timestamp column. Partitioning the table by day on the timestamp column allows BigQuery to prune partitions when queries filter by a time range, scanning only the relevant partitions instead of the entire table. This directly reduces the amount of data read, improving query performance and lowering costs.

Answer

Disable automatic reclustering to save costs

Answer

Set partition expiration to 1 year

Answer

Use nested repeated fields for customer data

Question 2

A company uses Cloud Pub/Sub with pull subscriptions to process orders. The application requires at-least-once delivery and the ability to process orders in order per customer_id. Which THREE features should they configure? (Choose three.)

Accepted Answer

Configure a dead letter topic. Correct answers are B, C, and E. Enabling ordering keys ensures messages with the same key are delivered in order. Setting exactly-once delivery on subscriptions provides at-least-once with deduplication; but exactly-once delivery on subscription actually reduces duplicates. However, the question says at-least-once, so exactly-once delivery would be too strong? Actually, exactly-once delivery on subscription ensures no duplicates, so it's even better. But if they need at-least-once, they could enable it and it still satisfies. Dead letter topics allow failed messages to be isolated and reprocessed later without blocking. A (ackDeadline of 600 seconds) is too long. D (using a push subscription) does not inherently improve ordering.

Answer

Use a push subscription with a HTTPS endpoint

Answer

Set the subscription's ackDeadline to 600 seconds

Question 3

A company uses Cloud Dataflow to process streaming data. They notice that the pipeline's throughput is lower than expected and the system is experiencing high latency. What is the most likely cause?

Accepted Answer

Too few workers. Option A is correct because insufficient workers are a common cause of low throughput and high latency. Autoscaling may not be enabled or workers are too few. Option B is wrong because batch mode is not used in streaming. Option C is incorrect; watermark settings affect late data, not throughput. Option D is wrong; too many workers would not cause high latency.

Answer

Using batch mode instead of streaming mode

Answer

Too many workers

Answer

Incorrect watermark setting

Question 4

A data engineer needs to design a data processing system that ingests large volumes of sensor data from IoT devices. The data should be stored in a schema-less format and allow for real-time analytics. Which Google Cloud service is most appropriate?

Accepted Answer

Cloud Bigtable. Cloud Bigtable is the most appropriate choice because it is a fully managed, scalable NoSQL database designed for large-scale analytical and operational workloads. It supports schema-less storage of time-series sensor data and integrates with real-time analytics tools like BigQuery and Dataflow via the HBase API, meeting the requirements for high-throughput ingestion and low-latency queries.

Answer

Cloud Spanner

Answer

Firestore

Answer

Cloud SQL

Question 5

A company is migrating their on-premises Apache Spark jobs to Dataproc. They want to minimize code changes and take advantage of serverless infrastructure. Which Dataproc feature should they use?

Accepted Answer

Dataproc Serverless Spark. Dataproc Serverless Spark is the correct choice because it allows the company to run Spark workloads without provisioning or managing clusters, minimizing code changes by using the same Spark APIs and libraries. This serverless infrastructure automatically scales resources and handles failures, aligning with the goal of reducing operational overhead while maintaining compatibility with existing Spark jobs.

Answer

Dataproc clusters with preemptible VMs

Answer

Dataproc Workflow Templates

Answer

Dataproc Jobs API with custom machine types

Question 6

A data pipeline using Cloud Pub/Sub and Cloud Dataflow is experiencing duplicate messages. The source system publishes messages at least once. What Dataflow technique ensures exactly-once processing?

Accepted Answer

Use idempotent sinks. Option A is correct because idempotent sinks ensure that even if Cloud Pub/Sub delivers the same message multiple times (due to its at-least-once delivery semantics), the Dataflow pipeline can deduplicate or safely reapply the same data without causing duplicates in the output. This is achieved by designing the sink (e.g., BigQuery with insertId, Cloud Storage with unique filenames) to recognize and ignore repeated writes, effectively providing exactly-once processing semantics downstream.

Answer

Use GlobalWindows

Answer

Set watermark threshold

Answer

Enable streaming engine

Question 7

A company processes financial transactions using Cloud Dataflow. They need to ensure that late-arriving data is handled correctly for fraud detection. The pipeline uses event time processing. Which approach should they use to handle late data?

Accepted Answer

Fixed windows with allowed lateness. Option C is correct because fixed windows with allowed lateness are the standard approach in Cloud Dataflow (Apache Beam) for handling late-arriving data in event-time processing. By specifying an allowed lateness duration, the pipeline retains the window state for that period, allowing late events to be correctly assigned to their original window and triggering recomputation of results. This ensures fraud detection pipelines can account for delayed transactions without missing or misordering data.

Answer

Sliding windows with early firing

Answer

Session windows with gap duration

Answer

Global windows with triggers

Question 8

A data engineer is designing a batch ETL pipeline using Cloud Composer and Dataflow. The pipeline must be self-healing and retry on failures. Which Composer feature should they configure?

Accepted Answer

Retry policy on the DAG. Option B is correct because Cloud Composer (based on Apache Airflow) allows you to configure a retry policy directly on the DAG or individual tasks. This enables the pipeline to automatically retry failed tasks according to parameters like `retries`, `retry_delay`, and `retry_exponential_backoff`, making the ETL pipeline self-healing without external services.

Answer

Use Cloud Tasks for retries

Answer

Cloud Composer with high availability

Answer

Dataflow retries

Question 9

A team is using BigQuery to analyze petabyte-scale data. They notice that queries are slow and expensive due to full table scans. They have already partitioned by date. What additional optimization should they implement?

Accepted Answer

Cluster by frequently filtered columns. Clustering by frequently filtered columns (option B) organizes data within each partition based on the sort order of those columns. This allows BigQuery to prune blocks during query execution, significantly reducing the amount of data scanned and improving both performance and cost. Since the table is already partitioned by date, clustering adds a secondary ordering that targets the most common filter predicates, avoiding full table scans within each partition.

Answer

Use materialized views

Answer

Convert to native tables

Answer

Use query caching

Question 10

A company needs to stream real-time user click events from a web application to BigQuery for analysis. Which Google Cloud architecture is most suitable?

Accepted Answer

App Engine -> Pub/Sub -> Dataflow -> BigQuery. Option A is correct because it provides a fully managed, scalable, and decoupled architecture for ingesting real-time click events. Pub/Sub acts as a durable, asynchronous message buffer that can handle high-throughput streams, Dataflow (Apache Beam) processes the events in near real-time with exactly-once semantics, and BigQuery serves as the analytics warehouse. This pattern is the recommended Google Cloud approach for streaming analytics, as it decouples producers from consumers and supports auto-scaling.

Answer

Cloud Scheduler -> BigQuery

Answer

Compute Engine -> Cloud Storage -> BigQuery

Answer

Cloud Functions -> BigQuery

Question 11

A data pipeline reading from Cloud Storage and writing to BigQuery using Dataflow is experiencing high cost. The data is CSV and needs schema inference. What change reduces cost?

Accepted Answer

Use BigQuery load jobs with schema auto-detection. Option C is correct because BigQuery load jobs with schema auto-detection can directly ingest CSV files from Cloud Storage without the need for a Dataflow pipeline, eliminating the compute cost associated with Dataflow. Schema auto-detection infers column names and types from the CSV header and data, matching the requirement for schema inference while being a serverless, no-cost-for-compute operation (you only pay for storage and querying). This reduces cost by removing the Dataflow processing step entirely.

Answer

Use Dataproc instead of Dataflow

Answer

Use Cloud Functions to transform data

Answer

Use BigQuery Data Transfer Service

Question 12

A data engineer is designing a streaming pipeline with Cloud Pub/Sub and Cloud Dataflow. They need to guarantee at-least-once delivery and handle occasional duplicates. Which TWO configurations should they implement?

Accepted Answer

Use idempotent sinks. Option A is correct because idempotent sinks (e.g., BigQuery with insertId, Cloud Storage with object generation numbers) allow the pipeline to safely process duplicate records without causing data corruption or double-counting. In a streaming pipeline with at-least-once semantics, duplicates are inevitable, and idempotent sinks ensure that repeated writes produce the same result as a single write, maintaining data consistency.

Answer

Use global windows with triggers

Answer

Use fixed windows

Answer

Enable Dataflow Streaming Engine

Question 13

A company uses Cloud Composer to orchestrate Dataproc and BigQuery jobs. They need to implement retry logic for transient failures. Which THREE features can help?

Accepted Answer

DAG retry_delay. Option B is correct because Cloud Composer (Apache Airflow) allows setting `retry_delay` at the DAG level to define the time delay between task retries. This is a native Airflow feature that helps handle transient failures by automatically retrying failed tasks after a specified delay, reducing manual intervention.

Answer

Dataflow pipeline retries

Answer

Cloud Composer high availability

Question 14

A data warehouse in BigQuery is experiencing performance issues. Which THREE techniques can improve performance without moving data to a different storage system?

Accepted Answer

Partition by date. Partitioning by date in BigQuery allows the query engine to prune entire partitions that do not match the query's date filter, significantly reducing the amount of data scanned and improving performance. This technique works without moving data to a different storage system because it is a metadata-level reorganization of the existing table.

Answer

Use streaming buffer

Answer

Use BigQuery slots

Question 15

A company runs a streaming data pipeline on Google Cloud using Cloud Pub/Sub, Cloud Dataflow, and BigQuery. The pipeline processes real-time sensor data for predictive maintenance. Recently, the Dataflow job's lag has increased from seconds to minutes, and the system shows backpressure. The pipeline uses fixed windows of 1 minute and writes results to BigQuery. The data volume has doubled. The team has already increased the number of workers. What should they do next? Options: A. Use session windows instead of fixed windows. B. Enable Streaming Engine and use Upsert to BigQuery. C. Decrease the window duration. D. Use Cloud Storage as temporary sink.

Accepted Answer

Enable Streaming Engine and use Upsert to BigQuery. The correct answer is A because enabling Streaming Engine offloads the heavy shuffle and state management from the worker VMs to the backend service, reducing the impact of backpressure. Using Upsert to BigQuery allows the pipeline to handle late-arriving data within the fixed windows without requiring a full table rewrite, which is critical when data volume has doubled and lag has increased.

Answer

Decrease the window duration

Answer

Use session windows instead of fixed windows

Answer

Use Cloud Storage as temporary sink