PDE Designing Data Processing Systems Practice Test 7 — 15 Questions

Question 1

A data pipeline ingests streaming events into Pub/Sub and needs to join them with a slowly updating reference table (few thousand rows) from a Cloud Storage CSV file. The pipeline runs on Dataflow with Apache Beam. Which approach is most cost-effective and operationally simple?

Accepted Answer

Use a side input that reads the CSV once and broadcasts it to all workers. Side inputs allow you to read the reference data once and distribute it to all workers, avoiding repeated lookups to an external service. For a small reference table, this is efficient and simple. Option A is complex and overkill; Option B adds latency; Option D is not a native Beam feature.

Answer

Read the CSV in a DoFn and perform a BigQuery query each time an event is processed

Answer

Implement a custom sink that writes events to Cloud SQL and performs a SQL JOIN there

Answer

Use CoGroupByKey to join the stream and batch PCollections by a common key after reading the CSV into a batch PCollection each window

Question 2

A Dataflow pipeline using Apache Beam processes unbounded data from Pub/Sub. The pipeline uses fixed windows of 1 minute and a trigger that fires early every 30 seconds and at watermark. The team observes that the output pane for window [10:00:00, 10:01:00) contains events with timestamps from 10:00:15 and 10:00:45, but also an event with timestamp 10:02:00. What is the most likely cause?

Accepted Answer

Allowed lateness is set to more than 1 minute, so late data is still included in its original window. Late data can arrive after the watermark has passed, and with allowed lateness, it can be included in the original window. The event with timestamp 10:02:00 is late data that arrived after the watermark for window [10:00:00, 10:01:00), but within the allowed lateness period. It is not a trigger issue because the trigger fires correctly; the event is simply late.

Answer

The trigger is firing too early, causing the window to close prematurely

Answer

The watermark is incorrectly estimated, allowing late data to be included

Answer

The window duration is actually 2 minutes due to a misconfiguration

Question 3

A developer wants to create a BigQuery table that automatically expires data older than 30 days to reduce storage costs. Which table design feature should be used?

Accepted Answer

Partitioned table with partition expiration. Partitioned tables with a partition expiration allow automatic deletion of partitions. Clustering does not affect data expiration. Materialized views are for pre-computed aggregates, not data lifecycle. Authorized views control access.

Answer

Authorized view

Answer

Clustered table

Answer

Materialized view

Question 4

A company runs Apache Spark jobs on Dataproc. They want to reduce costs by using preemptible instances for worker nodes. The jobs are fault-tolerant and can handle occasional node loss. However, the cluster must remain available for interactive querying during business hours. Which Dataproc cluster configuration meets these requirements?

Accepted Answer

Use a standard cluster with preemptible instances as secondary workers. Dataproc clusters support multiple node types. Primary workers run the NodeManager and DataNode daemons; using preemptible instances for them is risky because they are critical for HDFS and YARN. However, secondary workers (preemptible workers) are designed for stateless processing and can be lost without affecting cluster availability. Preemptible instances cannot be used for master nodes.

Answer

Use a single-node cluster that automatically scales with preemptible instances

Answer

Use standard cluster with master and worker nodes as preemptible instances

Answer

Use a high-availability cluster with preemptible instances for primary workers

Question 5

A data engineer needs to design a streaming pipeline that ingests events from multiple sources, enriches them with a lookup table stored in BigQuery (updated every hour), and writes the results to a BigQuery table for real-time dashboards. The pipeline must handle late-arriving data up to 1 hour. Which Dataflow feature should be configured to manage late data?

Accepted Answer

Allowed lateness on the window. Allowed lateness in Apache Beam specifies the maximum delay of late data to be included in the window. Watermark estimation does not control late data acceptance; triggers specify when output is emitted, not how late data is handled. Side inputs are for enrichment, not late data management.

Answer

A custom watermark estimation function

Answer

Using side inputs with a periodic refresh

Answer

A trigger that fires on every late element

Question 6

Which Google Cloud service provides a serverless Spark environment where you can run Spark jobs without provisioning or managing a cluster?

Accepted Answer

Dataproc Serverless. Dataproc Serverless allows you to submit Spark jobs that run on auto-scaled infrastructure without cluster management. Dataflow is for Beam pipelines. Dataprep is for data wrangling. Data Fusion is for visual ETL.

Answer

Dataflow

Answer

Dataprep

Answer

Cloud Data Fusion

Question 7

A company is using Pub/Sub to ingest clickstream events. They need to ensure that events are delivered to a subscriber at least once, but duplicates can be tolerated. They also need to filter events by type before processing. Which subscription configuration should be used?

Accepted Answer

Pull subscription with a filter on event type attribute. Pub/Sub provides at-least-once delivery for both pull and push subscriptions. Filtering by attributes is supported at subscription level. Pull subscriptions are typically used when the subscriber controls the pace. Push subscriptions are also possible, but the question does not specify delivery method preference. The key is to enable message filtering on the subscription.

Answer

Pull subscription with exactly-once delivery enabled

Answer

Push subscription with no filter

Answer

Push subscription with a dead letter topic

Question 8

A data pipeline uses Cloud Data Fusion to perform ETL jobs. The pipeline reads from BigQuery, transforms data using Wrangler, and writes to Cloud Storage. The team notices that the pipeline runs slower than expected. They suspect the Data Fusion instance is under-provisioned. Which action should be taken to improve performance?

Accepted Answer

Change the Data Fusion instance type from Basic to Enterprise. Cloud Data Fusion uses Dataproc clusters for execution. The instance type (basic, standard, enterprise) determines the Dataproc cluster configuration. Upgrading to a higher edition or increasing the number of worker nodes directly improves throughput. Wrangler transforms are executed on the Dataproc cluster, so more workers help.

Answer

Add more Dataproc Metastore instances

Answer

Enable Data Fusion accelerator for BigQuery

Answer

Rewrite the pipeline using Cloud Dataprep instead

Question 9

A company wants to use Pub/Sub Lite to reduce costs for a high-throughput, low-latency streaming pipeline. However, they have a requirement to retain messages for up to 7 days for reprocessing. Which Pub/Sub Lite configuration supports this retention?

Accepted Answer

Set the retention duration on the Pub/Sub Lite topic to 7 days. Pub/Sub Lite topics are the only entity where retention duration is configured; messages are retained in the topic's storage for the specified duration, allowing subscribers to replay messages within that window. Setting the retention duration to 7 days on the topic ensures messages are available for reprocessing for up to 7 days, meeting the requirement.

Answer

Set the retention duration on the Pub/Sub Lite subscription to 7 days

Answer

Enable exactly-once delivery on the Pub/Sub Lite topic to retain messages for 7 days

Answer

Use a Pub/Sub Lite reservation with 7-day retention

Question 10

A data engineer needs to create a BigQuery table that is optimized for queries that filter on a 'customer_id' column and sort by 'transaction_date'. The table will be used for interactive analysis. Which combination of table features should be used?

Accepted Answer

Partition by transaction_date and cluster by customer_id. Clustering sorts data based on one or more columns, improving query performance for filters and sorts on those columns. Partitioning by date/timestamp can further improve performance for time-range queries. For this scenario, clustering on customer_id and partitioning by transaction_date is optimal.

Answer

Partition by customer_id and cluster by transaction_date

Answer

Cluster by both customer_id and transaction_date

Answer

Use a materialized view with customer_id and transaction_date

Question 11

A data team is migrating an on-premises Hadoop cluster to Dataproc. The cluster runs a mix of long-running services (Hive, HBase) and transient Spark jobs. They want to minimize cost while maintaining performance. Which TWO strategies should they implement?

Accepted Answer

Use preemptible instances for worker nodes in the transient Spark cluster. Preemptible workers are cost-effective for fault-tolerant Spark jobs. Separating long-running services into a separate cluster avoids interference and allows independent scaling. Using a single-node cluster for services is not practical. Dataproc on GKE adds complexity. Standard persistent disks are fine for HDFS.

Answer

Consolidate all workloads into a single high-availability cluster

Answer

Use local SSDs for all nodes to improve I/O performance

Answer

Use Dataproc on GKE to run long-running services

Question 12

A company uses Pub/Sub to ingest events from multiple sources. They need to ensure that messages from a specific source are processed in order (per source partition). They also need to deduplicate messages. Which TWO features should they use?

Accepted Answer

Enable exactly-once delivery on the subscription. Pub/Sub ordering keys ensure messages with the same ordering key are delivered in order. Message deduplication is achieved via exactly-once delivery (Pub/Sub now supports this). Dead letter topics are for undeliverable messages. Schemas enforce structure, not ordering.

Answer

Set a message schema to enforce ordering

Answer

Use a dead letter topic to handle out-of-order messages

Answer

Use a pull subscription with a large ack deadline

Question 13

A data engineer is designing a streaming pipeline using Dataflow with Apache Beam. The pipeline reads from Pub/Sub, performs a stateful transformation (e.g., session windowing), and writes to BigQuery. The pipeline must handle late data and ensure exactly-once semantics. Which THREE configurations are required?

Accepted Answer

Set allowed lateness on the window to accommodate late data. Exactly-once sink (BigQuery) ensures no duplicates. Idempotent writes are built into Dataflow's BigQuery sink when using exactly-once mode. Setting allowed lateness handles late data. Watermark estimation is automatic; custom is not required. Triggers are optional; default trigger works. Windowing is inherent.

Answer

Use the File Loads write method for BigQuery

Answer

Use a custom watermark estimation function for Pub/Sub source

Question 14

A company is evaluating BigQuery for a data warehouse migration. They have a mix of reporting queries and ad-hoc analytical queries. They want to control query costs and prevent runaway queries. Which THREE strategies should they implement?

Accepted Answer

Set a custom quota for concurrent queries. Custom quotas cap query usage. Reservation models (flex slots) provide dedicated resources and predictable pricing. Partitioning and clustering reduce data scanned, lowering cost. Authorized views control access but not cost. Materialized views can reduce cost but are not a cost control mechanism per se.

Answer

Grant authorized view access to limit data visibility

Answer

Create materialized views for all reporting queries

Question 15

A company uses Cloud Data Fusion for ETL pipelines. They need to transform sensitive data (PII) by masking certain columns before writing to BigQuery. They also need to ensure the pipeline can be monitored and restarted from failure points. Which THREE features should they use?

Accepted Answer

Enable pipeline monitoring with alerts in Cloud Data Fusion. Data Fusion Wrangler provides a step-by-step recipe for transformations, including masking. Pipeline monitoring and restart from failure are supported by the orchestration framework and checkpointing. Dataproc Metastore is for Hive metadata, not relevant. Data Fusion Studio is the UI, not a feature for monitoring. Cloud Composer is for workflow orchestration, not needed if Data Fusion handles it.

Answer

Use Cloud Composer to schedule and retry the pipeline

Answer

Create a Dataproc Metastore service to store pipeline metadata