PDE Designing Data Processing Systems Practice Test 9 — 15 Questions

Question 1

You need to choose a messaging service for a real-time streaming application that requires low cost and can tolerate occasional message loss. Which service is MOST suitable?

Accepted Answer

Pub/Sub Lite. Pub/Sub Lite is designed for high-volume, low-cost streaming where occasional message loss is acceptable. Unlike standard Pub/Sub, it uses zonal or regional storage with pre-provisioned capacity, which dramatically lowers cost but does not guarantee the same durability or at-least-once delivery semantics under all failure conditions. This makes it the right fit when cost is prioritized over guaranteed delivery.

Answer

Cloud Scheduler

Answer

Pub/Sub

Answer

Cloud Tasks

Question 2

A data engineer needs to run an existing Spark job on Google Cloud with minimal code changes. The job requires Hive metastore access. Which Dataproc feature should they use to provide a managed Hive metastore?

Accepted Answer

Dataproc Metastore. Dataproc Metastore is a fully managed, highly available Hive metastore service (based on Hive Metastore 2.3/3.1) that can be attached to Dataproc clusters, providing a persistent, serverless metastore without running a separate Hive metastore on a cluster. It allows existing Spark jobs that require Hive metastore access to run with minimal code changes, since the metastore endpoint is configured via cluster properties.

Answer

Cloud SQL for MySQL

Answer

BigQuery as a Hive metastore

Answer

Dataproc on GKE

Question 3

A media company processes video metadata using a Dataflow pipeline. They need to join two streaming sources: user activity (Pub/Sub) and video catalog updates (Pub/Sub). Which THREE transforms should be used in the pipeline?

Accepted Answer

ParDo to process each element individually. To join two streaming Pub/Sub sources in Dataflow, you need to: (1) Use ParDo to extract the key from each element (e.g., video_id), (2) Window both PCollections into a common window (e.g., fixed 1-minute) to align the data, and (3) Use CoGroupByKey to join on the common key. GroupByKey separately is not required because CoGroupByKey internally groups by key. Flatten is used to combine PCollections of the same type, which is not applicable here.

Answer

Flatten to combine the two PCollections

Answer

GroupByKey on each PCollection separately before joining

Question 4

You are designing a BigQuery data lake for a healthcare organization. The data includes patient records that must be access-controlled at the row level. Which TWO features should you use to meet this requirement?

Accepted Answer

Row-level security using row access policies. Row-level security using row access policies (A) is correct because BigQuery row access policies let you define a filter predicate (e.g., based on a patient_id or facility column) that restricts which rows each user or group can see, enforcing row-level access control directly on the table. Authorised views with row filters (B) are also correct because you can create a view that applies a WHERE clause to expose only permitted rows, then authorise that view on the dataset so users query the view without direct table access, achieving row-level control. Dataset-level IAM roles (C) only grant access at the dataset, table, or view level, not per row, so they cannot satisfy row-level requirements. Clustering on patient_id (D) is a physical storage optimization that improves query performance and reduces scanned bytes but provides no access control. Materialised views (E) cache precomputed query results for performance and do not enforce row-level security.

Answer

Dataset-level IAM roles

Answer

Clustering on patient_id

Answer

Materialised views

Question 5

A data engineer needs to process streaming data from thousands of IoT devices and generate real-time dashboards. The data volume is low but requires exactly-once processing semantics. Which Google Cloud service combination should they use?

Accepted Answer

Cloud Pub/Sub + Cloud Dataflow. Dataflow supports exactly-once processing via its streaming engine and checkpointing. Pub/Sub is the ingest service. Together they provide the required semantics.

Answer

Cloud Pub/Sub + Cloud Data Fusion

Answer

Cloud Pub/Sub + Cloud Dataproc

Answer

Cloud Pub/Sub + Cloud Dataprep

Question 6

A company has a BigQuery dataset containing sensitive customer data. They want to share a subset of this data with external partners, ensuring that partners can only see specific columns and rows. Which BigQuery feature should they use?

Accepted Answer

Authorized views. Authorized views in BigQuery allow a view to be created in one dataset that reads from a source dataset, and then access to the view is granted to external partners without granting access to the underlying source data. This enables column- and row-level filtering (via the view's SQL) so partners see only the permitted subset. This is the standard BigQuery pattern for sharing restricted data.

Answer

Materialized views

Answer

Clustered tables

Answer

Dataset-level access controls

Question 7

An organization runs periodic Apache Spark jobs on Dataproc to process data from Cloud Storage. They want to reduce costs by using preemptible instances for worker nodes. What is a key consideration when using preemptible instances in Dataproc?

Accepted Answer

Jobs must be designed to handle node preemption, and overall job runtime may increase. Preemptible VMs in Dataproc can be reclaimed by Compute Engine at any time (with a 30-second shutdown notice), so Spark jobs must be designed to tolerate worker loss — for example, by using checkpointing, retries, or resilient data pipelines. Because preempted nodes are removed and replacements take time to spin up, overall job runtime typically increases compared to a cluster of standard VMs. This trade-off is the key operational consideration when choosing preemptible workers for cost savings.

Answer

Preemptible instances cannot be used with the standard cluster mode

Answer

Preemptible instances are only available in certain regions

Answer

Jobs will automatically restart from the last checkpoint without any performance impact

Question 8

A company needs to process high-throughput streaming data with low latency. They are considering Cloud Pub/Sub for ingestion and Cloud Dataflow for processing. However, they are concerned about cost. Which alternative to Cloud Pub/Sub would reduce costs while still meeting the throughput requirements?

Accepted Answer

Cloud Pub/Sub Lite. Cloud Pub/Sub Lite is a zonal, cost-optimized messaging service designed for high-throughput streaming with predictable, low latency. Unlike standard Pub/Sub, which replicates data across zones and charges for data transfer and storage, Pub/Sub Lite lets you choose a capacity (in MiB/s) and charges a flat hourly rate, significantly reducing costs for steady, high-volume workloads. It integrates with Cloud Dataflow via the Pub/Sub Lite I/O connector, so the processing pipeline remains viable. Thus, it meets the throughput and latency requirements while addressing cost concerns.

Answer

Cloud Pub/Sub with pull subscriptions

Answer

Cloud Tasks

Answer

Cloud Pub/Sub with push subscriptions

Question 9

A data engineer is designing a pipeline that reads from Cloud Pub/Sub, aggregates events into 5-minute windows, and writes the results to BigQuery. The engineer wants to ensure that late-arriving data (up to 2 minutes late) is included in the correct window. Which Dataflow feature should they configure?

Accepted Answer

Set the allowed lateness to 2 minutes with a trigger that fires on late data. Dataflow's allowed lateness feature (set to 2 minutes) ensures that late-arriving data within that threshold is still assigned to the correct 5-minute window. Combined with a trigger that fires on late data, the pipeline can emit updated results for the window after the watermark passes, which is exactly what the engineer needs to handle late-arriving events up to 2 minutes late.

Answer

Use a sliding window of 5 minutes with 2-minute slide

Answer

Set the window duration to 7 minutes to account for lateness

Answer

Use a global window and watermark

Question 10

A company is using Cloud Storage to store raw logs. They want to use Cloud Data Fusion to transform and load the data into BigQuery on a daily schedule. The transformations are complex and involve joining multiple datasets. What is the most efficient way to run these pipelines?

Accepted Answer

Use Cloud Data Fusion to design the pipeline and schedule it to run on a Dataproc cluster. Cloud Data Fusion is a fully managed, code-free ETL/ELT service built on CDAP that provides a visual pipeline designer with a rich set of plugins (including BigQuery sinks) and can execute pipelines on ephemeral Dataproc clusters. For complex transformations involving joins across multiple datasets, Data Fusion's Wrangler and Joiner transforms plus its scheduling capability make it the most efficient native choice. Scheduling the pipeline to run on a Dataproc cluster gives the compute needed for joins while keeping orchestration managed.

Answer

Use Cloud Composer to orchestrate Dataproc jobs that run the transformations

Answer

Use Cloud Functions to trigger a Dataflow job that does the transformations

Answer

Use Cloud Dataprep to design the transformation and export to BigQuery

Question 11

An organization is implementing a data lake on Google Cloud using Cloud Storage. They need to process both batch and streaming data with a unified pipeline. The team has experience with Apache Beam. Which architecture should they use to minimize operational overhead?

Accepted Answer

Kappa architecture with Cloud Dataflow using the same pipeline for batch and streaming. Kappa architecture uses a single streaming pipeline for both batch and streaming, simplifying operations. Dataflow implements Beam and supports both modes.

Answer

Use Cloud Dataproc for batch and Cloud Dataflow for streaming

Answer

Lambda architecture with Cloud Dataflow for batch and Cloud Pub/Sub for streaming

Answer

Use Cloud Data Fusion for both batch and streaming

Question 12

A company uses Cloud Dataproc to run Spark ML training jobs. They want to persist the trained models and metadata in a Hive-compatible metastore. Which Dataproc feature should they use?

Accepted Answer

Dataproc Metastore. Dataproc Metastore is a fully managed, Hive-compatible metastore service on Google Cloud that integrates natively with Dataproc clusters, allowing Spark and Hive jobs to persist and share table metadata and schemas. It provides a serverless, scalable alternative to running a self-managed Hive metastore on a cluster, and it supports the Hive Metastore Thrift API so existing Spark ML workflows can store model metadata without code changes. This directly meets the requirement for a Hive-compatible metastore for trained models and metadata.

Answer

Cloud Hive Metastore (self-managed)

Answer

Cloud Bigtable

Answer

Cloud Data Catalog

Question 13

An engineer needs to create a Pub/Sub subscription that sends messages to an HTTPS endpoint. The endpoint must be able to acknowledge messages individually. Which type of subscription should they use?

Accepted Answer

Push subscription. Push subscriptions deliver messages to a configured HTTPS endpoint. The endpoint can acknowledge by returning a 200 status.

Answer

Pull subscription

Answer

BigQuery subscription

Answer

Cloud Storage subscription

Question 14

An organization is using BigQuery for analytics. They have a table that is 500 GB and is frequently queried by 'date' and 'region'. They want to optimize query performance and reduce costs. Which TWO actions should they take?

Accepted Answer

Cluster the table by region. Option E is correct because partitioning the table by date means BigQuery only scans the partitions that match the query's date filter, drastically reducing bytes processed and cost for a 500 GB table frequently filtered by date. Option D is correct because clustering the table by region physically sorts data within each partition by region, so queries filtering on region benefit from block pruning and scan less data. Together, partitioning by date and clustering by region match the two most common query predicates and are the standard BigQuery optimization pattern. Option A (authorized view) only controls access to data and does not improve query performance or reduce scan costs. Option B (wildcard table) is used to query multiple similarly named tables with a UNION-like syntax and does not optimize a single table's scans. Option C (materialized views) can help some workloads but is not the primary optimization for a frequently filtered base table and adds storage and refresh overhead, so it is not one of the two best actions here.

Answer

Use an authorized view

Answer

Use a wildcard table

Answer

Use materialized views

Question 15

A data pipeline ingests streaming events into Pub/Sub and needs to join them with a slowly updating reference table (few thousand rows) from a Cloud Storage CSV file. The pipeline runs on Dataflow with Apache Beam. Which approach is most cost-effective and operationally simple?

Accepted Answer

Use a side input that reads the CSV once and broadcasts it to all workers. Option B is correct because a Beam side input reads the small Cloud Storage CSV once and broadcasts the reference table to all workers, letting each streaming element be enriched in memory without per-event external calls, which is both cheap and operationally simple. Since the table has only a few thousand rows, it easily fits in worker memory as a side input. Option A is wrong because issuing a BigQuery query per event is slow, expensive, and adds an external dependency. Option C is wrong because routing events through Cloud SQL and doing SQL JOINs adds infrastructure and cost. Option D is wrong because CoGroupByKey requires reading the CSV into a batch PCollection per window and shuffling both sides, which is more complex and less efficient than a side input for a small, slowly changing table.

Answer

Read the CSV in a DoFn and perform a BigQuery query each time an event is processed

Answer

Implement a custom sink that writes events to Cloud SQL and performs a SQL JOIN there

Answer

Use CoGroupByKey to join the stream and batch PCollections by a common key after reading the CSV into a batch PCollection each window