PDE Designing Data Processing Systems Practice Test 10 — 15 Questions

Question 1

A developer wants to create a BigQuery table that automatically expires data older than 30 days to reduce storage costs. Which table design feature should be used?

Accepted Answer

Partitioned table with partition expiration. BigQuery partitioned tables support partition expiration, which automatically deletes partitions older than a specified number of days. Setting a partition expiration of 30 days on a time-partitioned table will drop data older than 30 days, reducing storage costs. This is the native, cost-effective way to enforce data retention.

Answer

Authorized view

Answer

Clustered table

Answer

Materialized view

Question 2

A company runs Apache Spark jobs on Dataproc. They want to reduce costs by using preemptible instances for worker nodes. The jobs are fault-tolerant and can handle occasional node loss. However, the cluster must remain available for interactive querying during business hours. Which Dataproc cluster configuration meets these requirements?

Accepted Answer

Use a standard cluster with preemptible instances as secondary workers. Dataproc supports preemptible VMs as secondary workers, which are added to a cluster alongside non-preemptible primary workers. Secondary workers handle the bulk of shuffle and task execution, so if a preemptible node is reclaimed, the job can recover while the primary workers and master keep the cluster alive and available for interactive queries. This gives the cost benefit of preemptible pricing without risking cluster or master availability.

Answer

Use a single-node cluster that automatically scales with preemptible instances

Answer

Use standard cluster with master and worker nodes as preemptible instances

Answer

Use a high-availability cluster with preemptible instances for primary workers

Question 3

A data engineer needs to design a streaming pipeline that ingests events from multiple sources, enriches them with a lookup table stored in BigQuery (updated every hour), and writes the results to a BigQuery table for real-time dashboards. The pipeline must handle late-arriving data up to 1 hour. Which Dataflow feature should be configured to manage late data?

Accepted Answer

Allowed lateness on the window. Allowed lateness on the window is the Dataflow feature that explicitly defines how long the pipeline will retain window state and accept late-arriving elements after the watermark has passed the end of the window. Setting allowed lateness to 1 hour lets Dataflow keep the window's state for that duration, so events arriving up to 60 minutes late are still processed and emitted with correct results. This directly satisfies the requirement to handle late data up to 1 hour without dropping or misassigning events.

Answer

A custom watermark estimation function

Answer

Using side inputs with a periodic refresh

Answer

A trigger that fires on every late element

Question 4

Which Google Cloud service provides a serverless Spark environment where you can run Spark jobs without provisioning or managing a cluster?

Accepted Answer

Dataproc Serverless. Dataproc Serverless is a Google Cloud service that allows you to run Spark jobs without provisioning or managing a cluster. It automatically scales resources and charges only for the duration of the job, making it ideal for serverless Spark workloads. This matches the requirement exactly.

Answer

Dataflow

Answer

Dataprep

Answer

Cloud Data Fusion

Question 5

A company is using Pub/Sub to ingest clickstream events. They need to ensure that events are delivered to a subscriber at least once, but duplicates can be tolerated. They also need to filter events by type before processing. Which subscription configuration should be used?

Accepted Answer

Pull subscription with a filter on event type attribute. A pull subscription with a filter on the event type attribute satisfies both requirements: Pub/Sub's at-least-once delivery is the default for pull subscriptions, so duplicates are tolerated, and message filtering via subscription filter expressions (on attributes like event type) prevents unwanted events from being delivered. This is the only option that combines at-least-once semantics with server-side filtering.

Answer

Pull subscription with exactly-once delivery enabled

Answer

Push subscription with no filter

Answer

Push subscription with a dead letter topic

Question 6

A data pipeline uses Cloud Data Fusion to perform ETL jobs. The pipeline reads from BigQuery, transforms data using Wrangler, and writes to Cloud Storage. The team notices that the pipeline runs slower than expected. They suspect the Data Fusion instance is under-provisioned. Which action should be taken to improve performance?

Accepted Answer

Change the Data Fusion instance type from Basic to Enterprise. Cloud Data Fusion instance types (Basic, Enterprise, Developer) determine the compute resources available to run pipelines. Upgrading from Basic to Enterprise increases the number of Dataproc worker nodes and resources available to the CDAP runtime, improving pipeline throughput and reducing runtime. This directly addresses the under-provisioned instance suspicion.

Answer

Add more Dataproc Metastore instances

Answer

Enable Data Fusion accelerator for BigQuery

Answer

Rewrite the pipeline using Cloud Dataprep instead

Question 7

A company uses Pub/Sub to ingest events from multiple sources. They need to ensure that messages from a specific source are processed in order (per source partition). They also need to deduplicate messages. Which TWO features should they use?

Accepted Answer

Enable exactly-once delivery on the subscription. Option E is correct because Pub/Sub message ordering is enabled by setting an ordering key on messages; messages sharing the same ordering key are delivered to subscribers in the order they were published, which satisfies the per-source-partition ordering requirement. Option C is correct because enabling exactly-once delivery on the subscription ensures that messages are not redelivered after successful acknowledgment, providing the required deduplication behavior. Option A is incorrect because schemas validate message format and structure, not ordering. Option B is incorrect because a dead letter topic only captures messages that cannot be successfully processed after retries, and does not reorder or deduplicate messages. Option D is incorrect because a large ack deadline only extends the processing window to reduce premature redelivery; it does not guarantee ordering or exactly-once deduplication.

Answer

Set a message schema to enforce ordering

Answer

Use a dead letter topic to handle out-of-order messages

Answer

Use a pull subscription with a large ack deadline

Question 8

A company is evaluating BigQuery for a data warehouse migration. They have a mix of reporting queries and ad-hoc analytical queries. They want to control query costs and prevent runaway queries. Which THREE strategies should they implement?

Accepted Answer

Set a custom quota for concurrent queries. Option B is correct because setting a custom quota for concurrent queries (via BigQuery custom quotas in Cloud Console/IAM) caps the number of simultaneously running queries per project or user, preventing a flood of ad-hoc queries from exhausting slots and causing runaway costs. Option C is correct because partitioning (e.g., by ingestion/date column) and clustering (e.g., by frequently filtered columns) prune the data scanned, directly reducing bytes processed and therefore on-demand query cost. Option E is correct because BigQuery reservations with flex slots (or committed slots) let predictable reporting workloads run on dedicated capacity with fixed pricing, isolating them from unpredictable ad-hoc on-demand costs and giving cost predictability. Option A is not correct because authorized views control data access/visibility, not query cost or runaway query prevention. Option D is not correct because materialized views can accelerate specific recurring queries but do not by themselves control costs or prevent runaway queries across a mixed workload.

Answer

Grant authorized view access to limit data visibility

Answer

Create materialized views for all reporting queries

Question 9

A data team is building a near-real-time dashboard that displays aggregated metrics from Kafka topics. They want to use Pub/Sub as a managed messaging service and Dataflow for stream processing. They need to ingest data from Kafka into Pub/Sub with minimal custom code. Which THREE Google Cloud services should they use together? (Choose three.)

Accepted Answer

Dataflow. Option A, Dataflow, is correct because it is Google Cloud's managed Apache Beam service for stream processing, and it can run a streaming pipeline that reads from Pub/Sub, performs windowed aggregations, and writes results for the near-real-time dashboard. Option B, Pub/Sub, is correct because it serves as the managed messaging service that decouples the Kafka ingestion layer from the Dataflow processing layer, buffering messages and enabling reliable, scalable delivery. Option C, Kafka Connect (with Pub/Sub connector), is correct because Kafka Connect provides a configuration-driven, low-code way to move data from Kafka topics into Pub/Sub using a Pub/Sub sink connector, satisfying the requirement for minimal custom code. Option D, Cloud NAT, is not relevant because it provides outbound internet address translation for private VMs and does not ingest Kafka data into Pub/Sub. Option E, Cloud Functions, is not appropriate here because it is an event-driven serverless compute service for lightweight functions, not the managed stream-processing engine needed for aggregated Kafka metrics.

Answer

Cloud NAT

Answer

Cloud Functions

Question 10

A retail company ingests point-of-sale clickstream events into Cloud Pub/Sub at roughly 200,000 messages per second during flash sales. Analysts need near-real-time dashboards that aggregate revenue by product category over sliding 5-minute windows, with results visible in BigQuery within 30 seconds of the event. The pipeline must handle occasional bursts up to 3x the normal rate without dropping messages, and the team wants to minimize operational overhead. Which design should the data engineer use?

Accepted Answer

Create a Cloud Dataflow streaming pipeline in the Apache Beam Java SDK using Pub/Sub as the source, apply windowing with a 5-minute sliding window and a 30-second allowed lateness, and write aggregated results to BigQuery using the Storage Write API.. The requirement for sliding 5-minute windows with sub-30-second BigQuery visibility under bursty load points to a managed stream processor with native windowing and autoscaling. Dataflow's sliding windows, allowed lateness, and BigQuery Storage Write API together meet latency, correctness, and operational-overhead constraints. Cluster-based and file-based approaches add latency or management burden that the scenario rules out.

Answer

Use Cloud Pub/Sub push subscriptions delivering events directly to a Cloud Run service that aggregates in memory and calls the BigQuery streaming insert API once per minute.

Answer

Deploy a Dataproc cluster with Spark Structured Streaming reading from Pub/Sub and writing micro-batches to BigQuery every 10 seconds, sizing the cluster for the 3x burst rate.

Answer

Load raw events into Cloud Storage using a Pub/Sub subscription with a Cloud Storage sink, then run a BigQuery scheduled query every 5 minutes that reads the new files and computes the sliding window aggregates.

Question 11

A startup wants to analyze user clickstream data stored in Cloud Storage in Parquet format. They need to run ad-hoc SQL queries without managing any servers and want to pay only for the queries they run. Which Google Cloud service should they use?

Accepted Answer

BigQuery. BigQuery provides serverless SQL analytics with pay-per-query pricing and can query external Parquet data in Cloud Storage. It requires no infrastructure management, perfectly matching the startup's needs. Other services either require provisioning, are not designed for SQL analytics, or lack external data querying.

Answer

Dataproc

Answer

Cloud Bigtable

Answer

Cloud SQL

Question 12

A retail company ingests point-of-sale events from thousands of stores into Cloud Pub/Sub. They need to process these events in a streaming Dataflow pipeline that enriches each event with store metadata from a slowly changing BigQuery table. The enrichment table is updated only once per day. The pipeline must minimize latency and avoid querying BigQuery for every event. Which approach should they use?

Accepted Answer

Load the BigQuery metadata table into a side input as a key-value map, refreshing it periodically via a scheduled pipeline.. The correct approach is to load the slowly changing metadata into a side input that is refreshed periodically. This allows the streaming pipeline to enrich each event without querying BigQuery per event, keeping latency low. Side inputs are a standard Dataflow pattern for enriching streaming data with reference data that changes infrequently, and they avoid the high cost and latency of per-element external calls.

Answer

Use a CoGroupByKey transform to join the streaming events with a bounded read of the metadata table.

Answer

Configure the pipeline to call the BigQuery Storage Read API for each event to fetch the latest metadata.

Answer

Use BigQueryIO.Read with a query that joins the events to the metadata table, and apply the join in the pipeline.

Question 13

Your team is designing a data processing system that ingests JSON messages from millions of IoT devices. The ingestion rate is highly variable, with spikes up to 500,000 messages per second. You need a fully managed, serverless messaging service that can buffer messages and decouple producers from consumers. Which Google Cloud service should you choose?

Accepted Answer

Cloud Pub/Sub. Cloud Pub/Sub is the correct choice because it is a serverless, fully managed messaging service that automatically scales to handle variable ingestion rates, buffers messages durably, and decouples producers from consumers. It is purpose-built for high-throughput event ingestion and integrates seamlessly with downstream processing services like Dataflow.

Answer

Cloud Bigtable

Answer

Cloud Dataflow

Answer

Cloud Dataproc

Question 14

A financial services firm needs to design a batch processing system on Google Cloud to analyze large volumes of historical transaction data stored in Cloud Storage. The data is in Parquet format and must be processed using Apache Spark. The firm wants to minimize operational overhead and only pay for the resources used during job execution. Which Google Cloud service should they use?

Accepted Answer

Cloud Dataproc Serverless for Spark. Cloud Dataproc Serverless for Spark provides a fully managed Spark environment without cluster provisioning. It charges only for the resources used during job execution, aligning with the cost and operational requirements. It supports reading Parquet from Cloud Storage and is designed for batch processing workloads, making it the best fit for analyzing historical transaction data with Spark while minimizing overhead.

Answer

Cloud Dataflow with a custom Apache Spark runner

Answer

BigQuery with external tables over Cloud Storage

Answer

Cloud Dataproc with a long-running cluster

Question 15

A financial services company needs to process credit card transactions in real time to detect fraudulent patterns. The pipeline must handle late-arriving data (up to 2 hours) and produce accurate results. They want to use a unified programming model that works for both batch and streaming. Which Google Cloud service should they use?

Accepted Answer

Cloud Dataflow with Apache Beam. Cloud Dataflow with Apache Beam is the correct choice because it offers a unified batch and streaming programming model, supports event-time processing with watermarks and triggers, and can handle late-arriving data accurately through configurable allowed lateness. This makes it well-suited for real-time fraud detection with data arriving up to 2 hours late.

Answer

Cloud Pub/Sub with Cloud Functions

Answer

BigQuery with scheduled queries

Answer

Cloud Dataproc with Spark Streaming