PDE Designing Data Processing Systems Practice Test 2 — 15 Questions

Question 1

A data engineer is designing a real-time fraud detection system using Dataflow. The system must detect patterns across events from multiple users within a sliding window of 10 minutes. Events arrive on Pub/Sub topics per user. Which approach should they use to join the streams?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Use CoGroupByKey on the two streams using a common key like user_id. CoGroupByKey joins multiple PCollections by key. Using user_id as common key, both streams can be joined. Side inputs and Union are not for joining. Flatten merges PCollections of same type.

Answer

Use a side input to read one stream as a map and enrich the other stream

Answer

Use Flatten to merge the streams and then Partition

Answer

Use Union to combine both streams into one and then apply GroupByKey

Question 2

A company wants to use Dataprep to clean and transform raw CSV files stored in Cloud Storage before loading into BigQuery. The data quality checks show missing values and inconsistent date formats. Which Dataprep feature should they use to handle these issues?

Accepted Answer

Recipe steps. Recipe steps allow chaining transformations like fill missing values and format dates. Data quality profiling identifies issues but doesn't fix them. Scheduling automates execution. Wrangler is the UI, not a specific feature for transformations.

Answer

Data quality profiling

Answer

Scheduling

Answer

Wrangler

Question 3

A company needs a messaging service for event-driven applications that require low cost for high-throughput, but can tolerate occasional message loss. Which Pub/Sub product should they choose?

Accepted Answer

Pub/Sub Lite. Pub/Sub Lite is designed for cost-sensitive workloads with relaxed durability. Standard Pub/Sub offers at-least-once delivery and high durability. Push vs pull is irrelevant to cost.

Answer

Pub/Sub with pull subscriptions

Answer

Pub/Sub with dead letter topics

Answer

Pub/Sub with push subscriptions

Question 4

A retail company uses Dataflow to process real-time clickstream data. They need to enrich each event with customer profile data from Cloud Bigtable and session metadata from Cloud Spanner. Which two Dataflow features should they use?

Accepted Answer

ParDo. Side inputs allow reading from Bigtable and Spanner in a non-blocking way. ParDo is for per-element processing where enrichment occurs. GroupByKey and Windowing are not needed for this enrichment step.

Answer

Windowing

Answer

CoGroupByKey

Answer

GroupByKey

Question 5

A company is migrating on-premises Hadoop Hive workloads to Google Cloud. They want to use Dataproc for Spark processing and require a managed Hive metastore that can be shared across multiple Dataproc clusters. Which TWO components should they use?

Accepted Answer

Dataproc Metastore. The question specifies a managed Hive metastore that can be shared across multiple Dataproc clusters. Dataproc Metastore is a fully managed service that provides a Hive metastore, requiring no manual setup. Cloud SQL for MySQL can serve as a metastore backend, but it is not a managed service—it requires manual configuration, maintenance, and is not a dedicated metastore service. Therefore, only Dataproc Metastore meets the criteria of a managed Hive metastore.

Answer

Cloud Bigtable

Answer

Dataproc on GKE

Answer

Dataproc Serverless

Answer

Cloud SQL for MySQL

Question 6

A data engineer needs to design a BigQuery dataset for a multi-team environment. Each team should have read access only to specific tables, and the data must be protected from accidental deletion. Which THREE steps should they take?

Accepted Answer

Create authorized views for each team and grant access to the views. Authorized views allow sharing specific table data without direct table access. Dataset-level IAM is too broad. Deletion protection prevents accidental table drops. Clustering improves performance but not access control.

Answer

Cluster tables by team_id to improve performance

Answer

Grant bigquery.dataViewer on the dataset to all teams

Question 7

A company wants to design a data pipeline for real-time fraud detection. The system must process streaming financial transactions, enrich them with user profiles from a lookup table, and flag suspicious activities within seconds. Which architecture pattern would be MOST suitable?

Accepted Answer

Kappa architecture using a single stream processing framework like Apache Beam. Kappa architecture uses a single stream processing engine to handle both real-time and batch reprocessing, simplifying the pipeline. Lambda architecture requires maintaining separate batch and streaming layers, increasing complexity. The scenario only requires real-time processing with enrichment, so Kappa is more appropriate.

Answer

Pub/Sub combined with Cloud Functions for stateless processing

Answer

Batch processing with hourly micro-batches using Dataflow

Answer

Lambda architecture with a batch layer for historical analysis and a speed layer for real-time processing

Question 8

You are designing a BigQuery data warehouse for a multi-tenant SaaS application. Each tenant's data must be isolated and queried only by that tenant. You need to minimise management overhead and allow tenants to be added dynamically. Which approach should you use?

Accepted Answer

Use a single dataset with authorized views that filter by tenant_id, granting each tenant access to their view. Authorized views in a shared dataset allow you to create row-level security. By creating a view per tenant that filters by tenant_id, you can grant each tenant access only to its view. This avoids managing multiple datasets and is scalable.

Answer

Use Cloud IAM conditions on the dataset to filter by tenant_id

Answer

Create a separate dataset for each tenant and grant the tenant access to their dataset

Answer

Use a single table with a tenant_id column and enable column-level security to restrict access

Question 9

You need to process large-scale log files (hundreds of terabytes) using Apache Spark on Google Cloud. The job runs nightly and you want to minimise costs. Which Dataproc cluster configuration is MOST cost-effective?

Accepted Answer

Standard cluster with preemptible secondary workers. Preemptible VMs are significantly cheaper than standard VMs and suitable for fault-tolerant batch jobs like nightly Spark processing. Standard mode is fine but using preemptible workers reduces cost.

Answer

Single-node cluster

Answer

Standard cluster with preemptible workers for the primary worker nodes

Answer

Standard cluster with all standard (non-preemptible) workers

Question 10

A data pipeline ingests streaming events into Pub/Sub. You need to guarantee that each event is processed exactly once downstream in Dataflow. Which combination of Pub/Sub and Dataflow configurations should you use?

Accepted Answer

Use Pub/Sub with a unique message ID and Dataflow with idempotent writes or Dataflow's exactly-once sink. Pub/Sub offers at-least-once delivery. To achieve exactly-once processing, the pipeline must be idempotent or use Dataflow's exactly-once sinks. Using a unique message ID for deduplication is a common approach.

Answer

Use Pub/Sub with exactly-once delivery enabled and Dataflow with exactly-once processing

Answer

Use Pub/Sub with message deduplication and Dataflow with at-least-once processing

Answer

Use Pub/Sub with a dead letter topic and Dataflow with automatic retries

Question 11

You are designing a Dataflow pipeline that reads from Pub/Sub and writes to BigQuery. The pipeline must handle late-arriving data (up to 1 hour) and group events into 10-minute windows. Which configuration is correct?

Accepted Answer

Use fixed windows of 10 minutes with allowed lateness of 1 hour and a trigger that fires after watermark plus early firings. To handle late data, you need to set the allowed lateness to 1 hour. The trigger with AfterWatermark with early firings ensures results are emitted on time and updated when late data arrives.

Answer

Use global windows with a trigger that fires every 10 minutes

Answer

Use sliding windows of 10 minutes with a 5-minute period and allowed lateness of 1 hour

Answer

Use fixed windows of 10 minutes with allowed lateness of 0 seconds

Question 12

You are moving an on-premises Hadoop workload to Google Cloud. The workload uses Hive for metadata and HDFS for storage. Which services should you use to minimise reconfiguration?

Accepted Answer

Dataproc with Cloud Storage and Dataproc Metastore. Dataproc Metastore provides a fully managed Hive metastore service that can be used with Dataproc clusters. Cloud Storage can replace HDFS via the gs:// connector, allowing the same file paths. This minimises code changes.

Answer

Dataproc with HDFS and Cloud Bigtable for metadata

Answer

Dataproc with Cloud Storage and Cloud SQL for Hive metastore

Answer

Dataflow with Cloud Storage and BigQuery

Question 13

Which Google Cloud service provides a visual interface for building ETL pipelines using a drag-and-drop design and includes pre-built transforms from a marketplace?

Accepted Answer

Cloud Data Fusion. Cloud Data Fusion offers a visual, code-free ETL tool with a rich set of plugins from the Hub. Dataprep is for data wrangling, not full ETL.

Answer

Dataproc

Answer

Dataprep

Answer

BigQuery

Question 14

You are designing a streaming pipeline that needs to handle sudden spikes in traffic without losing data. The pipeline uses Pub/Sub and Dataflow. Which configuration ensures data is not lost if Dataflow falls behind?

Accepted Answer

Use Pub/Sub with a pull subscription and set the message retention duration to 7 days. Pub/Sub stores messages for up to 7 days, allowing Dataflow to catch up. Dataflow uses checkpointing to track progress. This combination ensures no data loss.

Answer

Use Cloud Pub/Sub Lite with a smaller retention period

Answer

Use Pub/Sub with a push subscription and increase the acknowledgment deadline

Answer

Use Pub/Sub with exactly-once delivery and Dataflow with at-least-once processing

Question 15

You need to analyse streaming data from thousands of IoT devices, each sending temperature readings every second. You want to calculate the average temperature per device over the last 5 minutes, updating every minute. Which windowing strategy should you use in Dataflow?

Accepted Answer

Sliding windows of length 5 minutes with a period of 1 minute. Sliding windows of length 5 minutes with a period of 1 minute give the desired overlapping windows: every minute, you get the average over the last 5 minutes.

Answer

Global windows with a trigger firing every minute

Answer

Fixed windows of 5 minutes

Answer

Session windows with a gap duration of 1 minute