PDE Designing Data Processing Systems Practice Test 5 — 15 Questions

Question 1

A company needs to process streaming sensor data and run both real-time analytics and batch reanalysis on historical data. They want to minimize infrastructure management. Which architecture and service combination is MOST suitable?

Accepted Answer

Kappa architecture with Pub/Sub and Dataflow for both real-time and batch processing. Kappa architecture processes all data as a stream, avoiding separate batch/speed layers. Pub/Sub ingests streaming data, and Dataflow with Apache Beam can handle both real-time and batch (replay) pipelines, minimizing infrastructure management.

Answer

Lambda architecture with Pub/Sub for streaming and Cloud Storage for batch, processed by Dataflow

Answer

Batch processing only with Dataflow and Cloud Storage, ignoring real-time needs

Answer

Kappa architecture with Pub/Sub Lite and Dataflow Serverless

Question 2

You are designing a BigQuery data warehouse for a retail company. Queries frequently filter on order_date and customer_id. To optimize query performance and cost, which table design should you use?

Accepted Answer

Partition by order_date and cluster by customer_id. Partitioning on order_date limits scans to relevant date ranges. Clustering on customer_id further organizes data within partitions, improving filter and aggregation queries on customer_id.

Answer

Cluster by order_date and partition by customer_id

Answer

Partition by ingestion_time and cluster by order_date

Answer

Use a clustered table without partitioning

Question 3

A Dataflow streaming pipeline reads from Pub/Sub, processes events with a fixed window of 1 minute, and writes to BigQuery. Some events arrive late due to network issues. You need to ensure late events are still included in the correct window but the pipeline must not wait indefinitely. What configuration should you use?

Accepted Answer

Set allowed lateness to 5 minutes and use the default trigger. Setting a watermark estimate and allowed lateness with a trigger controls how long the pipeline waits for late data. The default trigger fires at the end of the window, and with allowed lateness, late events are still processed until the allowed time expires.

Answer

Use a sliding window of 1 minute with a 1-minute period

Answer

Use a global window with a trigger that fires every 10 seconds

Answer

Increase the watermark estimate to 10 minutes

Question 4

You need to process a large Spark ML training job on a Dataproc cluster. The job is fault-tolerant and can handle occasional node failures. To reduce costs, which type of worker nodes should you use?

Accepted Answer

Preemptible worker nodes. Preemptible VMs are significantly cheaper but can be terminated at any time. Since the job is fault-tolerant, preemptible workers can be used for cost savings.

Answer

Standard worker nodes

Answer

High-memory worker nodes

Answer

Sole-tenant nodes

Question 5

Your company uses Pub/Sub to ingest clickstream data. Messages must be processed in order for the same user_id. How should you configure the Pub/Sub subscription to guarantee ordering?

Accepted Answer

Use a pull subscription with enable_message_ordering=true. Pub/Sub ordering keys allow messages with the same key to be delivered in order to subscribers. The subscription must be created with enable_message_ordering set to true.

Answer

Use a pull subscription with exactly-once delivery enabled

Answer

Use a push subscription with acknowledgement deadline set to 600 seconds

Answer

Use a push subscription with a dead letter topic

Question 6

A Dataflow pipeline with multiple steps uses a side input from a slowly changing reference table stored in BigQuery. The side input is updated every hour. To avoid reprocessing the entire pipeline on each update, which approach should you use?

Accepted Answer

Use a side input with a custom 'AsIterable' and a 'Repeatable' trigger that refreshes the side input every hour. The side input should be read periodically using a side input pattern with a Repeatable trigger. This refreshes the side input without restarting the pipeline.

Answer

Use a side input with a periodic refresh via a DoFn that reads BigQuery on each element

Answer

Use the side input with a default trigger and 'withAllowedLateness'

Answer

Use a global window with a trigger that fires every hour

Question 7

You need to transform and clean messy CSV data using a visual interface without writing code. The transformation should be scheduled to run weekly. Which Google Cloud service should you use?

Accepted Answer

Cloud Dataprep. Cloud Dataprep (Trifacta) provides a visual interface for data wrangling, allowing users to create recipes and schedule jobs.

Answer

Cloud Dataflow

Answer

Dataproc

Answer

Cloud Data Fusion

Question 8

Your team wants to share a BigQuery dataset with another project while ensuring that users from that project can only query specific tables. Which BigQuery feature should you use?

Accepted Answer

Create an authorised view in your dataset and share the view with the other project. Authorised views allow you to share query results with users in other projects without giving them direct access to the underlying tables.

Answer

Use a materialised view and share the underlying table

Answer

Grant the BigQuery Data Viewer role to the other project's service account

Answer

Export the table to Cloud Storage and share the bucket

Question 9

A company uses Dataproc Serverless for Spark batch jobs. They notice that some jobs are failing due to out-of-memory (OOM) errors. Which configuration parameter should they adjust to allocate more memory per executor?

Accepted Answer

Set spark.executor.memory to a higher value, e.g., 8g. In Dataproc Serverless, Spark properties can be set via --properties. The spark.executor.memory property controls the memory per executor. Increasing it can resolve OOM errors.

Answer

Use a custom image with more memory

Answer

Set spark.driver.memory to a higher value

Answer

Increase the number of workers by setting --num-workers

Question 10

You are building a real-time fraud detection system using Dataflow. Events from Pub/Sub need to be grouped by user_id within a 5-minute window to detect suspicious patterns. Some events may be delayed by up to 2 minutes. How should you configure the window and trigger to balance accuracy and latency?

Accepted Answer

Fixed window of 5 minutes with allowed lateness of 2 minutes and early trigger every 1 minute. A fixed 5-minute window with allowed lateness of 2 minutes and a trigger that fires early every minute provides early results and captures late data within the allowed window.

Answer

Sliding window of 5 minutes with a 1-minute period and no allowed lateness

Answer

Session window with a gap duration of 5 minutes

Answer

Fixed window of 5 minutes with no allowed lateness and default trigger

Question 11

You need to choose a messaging service for a real-time streaming application that requires low cost and can tolerate occasional message loss. Which service is MOST suitable?

Accepted Answer

Pub/Sub Lite. Pub/Sub Lite offers a lower-cost option compared to Pub/Sub, with reduced reliability (e.g., at-least-once delivery but not exactly-once). It is designed for cost-sensitive streaming workloads where occasional loss is acceptable.

Answer

Cloud Scheduler

Answer

Pub/Sub

Answer

Cloud Tasks

Question 12

A data engineer needs to run an existing Spark job on Google Cloud with minimal code changes. The job requires Hive metastore access. Which Dataproc feature should they use to provide a managed Hive metastore?

Accepted Answer

Dataproc Metastore. Dataproc Metastore provides a fully managed Hive metastore that integrates with Dataproc clusters, allowing existing Spark jobs to use it without code changes.

Answer

Cloud SQL for MySQL

Answer

BigQuery as a Hive metastore

Answer

Dataproc on GKE

Question 13

You are designing a data pipeline for a financial services company that requires exactly-once processing semantics. Which TWO services or configurations provide exactly-once guarantees?

Accepted Answer

Pub/Sub with exactly-once delivery enabled. Pub/Sub with exactly-once delivery enabled provides exactly-once message delivery. Dataflow with exactly-once processing mode ensures each record is processed exactly once. Pub/Sub Lite and Dataproc do not provide exactly-once guarantees, and Cloud Storage is for storage.

Answer

Dataproc with checkpointing

Answer

Pub/Sub Lite with at-least-once delivery

Answer

Cloud Storage with object versioning

Question 14

A media company processes video metadata using a Dataflow pipeline. They need to join two streaming sources: user activity (Pub/Sub) and video catalog updates (Pub/Sub). Which THREE transforms should be used in the pipeline?

Accepted Answer

CoGroupByKey to join the two PCollections on a common key (e.g., video_id). CoGroupByKey joins multiple PCollections by key. Window into a common window ensures the joins are time-aligned. Flatten is not needed; ParDo is for element-wise processing, not joining.

Answer

Flatten to combine the two PCollections

Answer

ParDo to process each element individually

Question 15

You are designing a BigQuery data lake for a healthcare organization. The data includes patient records that must be access-controlled at the row level. Which TWO features should you use to meet this requirement?

Accepted Answer

Row-level security using row access policies. Row-level security in BigQuery allows filtering at the row level using access policies. Authorised views can also be used to expose only certain rows. Clustering and materialised views do not provide row-level access control.

Answer

Dataset-level IAM roles

Answer

Clustering on patient_id

Answer

Materialised views