PDE Designing Data Processing Systems — Timed 40 Questions

Question 1

A data engineer needs to design a stream processing pipeline that reads events from Pub/Sub, enriches them with data from a Cloud Storage file, and writes aggregated results to BigQuery. The pipeline must handle late-arriving events up to 1 hour. Which Dataflow feature should be used to manage late data?

Accepted Answer

Watermarks. Watermarks track event time progress and allow specifying allowed lateness. Triggers control when results are emitted, but watermarks handle late data.

Answer

Triggers

Answer

Side inputs

Answer

Windowing

Question 2

A company uses Dataproc to run daily Spark ML jobs. The jobs run for 2 hours each day. The team wants to reduce costs without changing job characteristics. Which strategy is MOST cost-effective?

Accepted Answer

Use preemptible instances for worker nodes. Preemptible VMs are up to 80% cheaper and can handle job interruptions as Spark is fault-tolerant. Single-node is for testing, not production. High-availability is for long-running clusters with HA requirements. Standard nodes are more expensive.

Answer

Use a single-node cluster to eliminate overhead

Answer

Enable high-availability mode to avoid restarts

Answer

Increase the number of standard workers to finish faster

Question 3

A financial services company stream trades into Pub/Sub and processes with Dataflow. The pipeline must ensure exactly-once processing of each trade for regulatory compliance. However, Pub/Sub guarantees at-least-once delivery. Which combination of features should the Dataflow pipeline use to achieve exactly-once semantics?

Accepted Answer

Use Dataflow's exactly-once processing mode and implement idempotent writes in the sink. Dataflow's exactly-once sink combined with idempotent writes ensures exactly-once output. Pub/Sub cannot guarantee exactly-once delivery, but Dataflow can deduplicate using unique IDs. Idempotent writes prevent duplicates even if Dataflow retries.

Answer

Enable Pub/Sub message deduplication and use at-most-once delivery

Answer

Use global windowing and discard late data

Answer

Use Pub/Sub Lite with exactly-once delivery guarantee

Question 4

A data engineer needs to create a BigQuery table that is partitioned by ingestion time and clustered by customer_id and transaction_date. They also want to limit access so that only users from a specific domain can query the table. Which approach should they use?

Accepted Answer

Create the table with partitioning and clustering, then create an authorized view on the table and grant the view access to the domain users. Authorized views allow sharing query results with specific users/groups without giving direct table access. Clustering and partitioning are defined at table creation. IAM roles at dataset level are too broad. Row-level security filters rows but doesn't restrict domain.

Answer

Create the table with partitioning only, then use a materialized view to restrict access

Answer

Create the table without clustering, use row-level security to filter by domain, and grant access to the table

Answer

Create the table with partitioning and clustering, then grant bigquery.dataViewer to the domain via IAM at the dataset level

Question 5

A startup needs a fully managed, serverless Spark service to run occasional data processing jobs without managing clusters. They want to pay only for the resources used during job execution. Which Google Cloud service should they use?

Accepted Answer

Dataproc Serverless. Dataproc Serverless provides a serverless Spark environment where you pay per job execution. Cloud Data Fusion is for visual ETL. Dataproc is managed but not serverless. Dataflow is serverless for Beam, not Spark.

Answer

Dataflow

Answer

Cloud Data Fusion

Answer

Dataproc

Question 6

A company wants to use Cloud Data Fusion to build ETL pipelines. They need to connect to a legacy on-premises database using JDBC and also want to use prebuilt transforms from the Hub. Which two features should they use?

Accepted Answer

CDAP JDBC plugin and the Hub. Cloud Data Fusion uses CDAP plugins for JDBC connections and the Hub provides prebuilt transforms. Plugins are the mechanism; Hub is where they are sourced. Wrangler is for data preparation, not sink. Dataproc is not needed as Data Fusion runs on its own infrastructure.

Answer

Cloud SQL JDBC driver and Cloud Functions

Answer

Dataproc Metastore and Cloud Storage sink

Answer

Wrangler and Dataproc

Question 7

A company uses Pub/Sub with push subscriptions to deliver events to a Cloud Run service. Recently, the service has been returning HTTP 429 (Too Many Requests), causing messages to be retried and eventually sent to the dead letter topic. What is the MOST likely cause?

Accepted Answer

The Cloud Run service needs more instances to handle the incoming request rate. Push subscriptions can be rate limited by the receiving service. Increasing ackDeadline gives more time but doesn't reduce rate. Using pull subscriptions shifts the rate control to the subscriber. Adjusting max delivery attempts only affects how many retries before dead letter, not rate limiting.

Answer

The subscription ackDeadline is set too low, causing messages to be redelivered

Answer

The push endpoint is not acknowledging messages quickly enough, causing a backlog

Answer

The dead letter topic is misconfigured, causing messages to be sent to it prematurely

Question 8

A data engineer needs to process data in a Dataflow pipeline that reads from a Pub/Sub topic. The pipeline must group events into 5-minute windows and compute the average value per key. Which Beam transform should they use after windowing?

Accepted Answer

Combine.perKey. Combine.perKey with an averaging function computes per-key average. GroupByKey groups by key but requires manual combination. ParDo is for element-wise processing. CoGroupByKey joins multiple PCollections.

Answer

ParDo

Answer

GroupByKey

Answer

CoGroupByKey

Question 9

A company uses BigQuery for analytics. They have a table that is queried frequently by date range. To reduce costs, they want to ensure queries only scan the relevant partitions. They also want to improve performance for queries filtering on a specific customer_id. Which table design should they use?

Accepted Answer

Partition by date column and cluster by customer_id. Partitioning by date allows pruning irrelevant partitions; clustering on customer_id orders data within partitions for efficient filtering. Clustering alone doesn't prune partitions. Ingestion-time partitioning is based on arrival time, not logical date.

Answer

Partition by ingestion time and cluster by customer_id

Answer

Use a materialized view that filters by date and customer_id

Answer

Cluster by date column and partition by customer_id

Question 10

A data engineer is designing a real-time fraud detection system using Dataflow. The system must detect patterns across events from multiple users within a sliding window of 10 minutes. Events arrive on Pub/Sub topics per user. Which approach should they use to join the streams?

Accepted Answer

Use CoGroupByKey on the two streams using a common key like user_id. CoGroupByKey joins multiple PCollections by key. Using user_id as common key, both streams can be joined. Side inputs and Union are not for joining. Flatten merges PCollections of same type.

Answer

Use a side input to read one stream as a map and enrich the other stream

Answer

Use Flatten to merge the streams and then Partition

Answer

Use Union to combine both streams into one and then apply GroupByKey

Question 11

A company wants to use Dataprep to clean and transform raw CSV files stored in Cloud Storage before loading into BigQuery. The data quality checks show missing values and inconsistent date formats. Which Dataprep feature should they use to handle these issues?

Accepted Answer

Recipe steps. Recipe steps allow chaining transformations like fill missing values and format dates. Data quality profiling identifies issues but doesn't fix them. Scheduling automates execution. Wrangler is the UI, not a specific feature for transformations.

Answer

Data quality profiling

Answer

Scheduling

Answer

Wrangler

Question 12

A company needs a messaging service for event-driven applications that require low cost for high-throughput, but can tolerate occasional message loss. Which Pub/Sub product should they choose?

Accepted Answer

Pub/Sub Lite. Pub/Sub Lite is designed for cost-sensitive workloads with relaxed durability. Standard Pub/Sub offers at-least-once delivery and high durability. Push vs pull is irrelevant to cost.

Answer

Pub/Sub with pull subscriptions

Answer

Pub/Sub with dead letter topics

Answer

Pub/Sub with push subscriptions

Question 13

A retail company uses Dataflow to process real-time clickstream data. They need to enrich each event with customer profile data from Cloud Bigtable and session metadata from Cloud Spanner. Which two Dataflow features should they use?

Accepted Answer

ParDo. Side inputs allow reading from Bigtable and Spanner in a non-blocking way. ParDo is for per-element processing where enrichment occurs. GroupByKey and Windowing are not needed for this enrichment step.

Answer

Windowing

Answer

CoGroupByKey

Answer

GroupByKey

Question 14

A company is migrating on-premises Hadoop Hive workloads to Google Cloud. They want to use Dataproc for Spark processing and require a managed Hive metastore that can be shared across multiple Dataproc clusters. Which TWO components should they use?

Accepted Answer

Dataproc Metastore. Dataproc Metastore provides a managed Hive metastore service that can be used by multiple Dataproc clusters. Using Dataproc on GKE allows flexibility but is not required for metastore sharing. Cloud SQL can host a Hive metastore but is not managed for Hive. Dataproc Serverless does not provide a shared metastore.

Answer

Cloud Bigtable

Answer

Dataproc on GKE

Answer

Dataproc Serverless

Question 15

A data engineer needs to design a BigQuery dataset for a multi-team environment. Each team should have read access only to specific tables, and the data must be protected from accidental deletion. Which THREE steps should they take?

Accepted Answer

Create authorized views for each team and grant access to the views. Authorized views allow sharing specific table data without direct table access. Dataset-level IAM is too broad. Deletion protection prevents accidental table drops. Clustering improves performance but not access control.

Answer

Cluster tables by team_id to improve performance

Answer

Grant bigquery.dataViewer on the dataset to all teams

Question 16

A company wants to design a data pipeline for real-time fraud detection. The system must process streaming financial transactions, enrich them with user profiles from a lookup table, and flag suspicious activities within seconds. Which architecture pattern would be MOST suitable?

Accepted Answer

Kappa architecture using a single stream processing framework like Apache Beam. Kappa architecture uses a single stream processing engine to handle both real-time and batch reprocessing, simplifying the pipeline. Lambda architecture requires maintaining separate batch and streaming layers, increasing complexity. The scenario only requires real-time processing with enrichment, so Kappa is more appropriate.

Answer

Pub/Sub combined with Cloud Functions for stateless processing

Answer

Batch processing with hourly micro-batches using Dataflow

Answer

Lambda architecture with a batch layer for historical analysis and a speed layer for real-time processing

Question 17

You are designing a BigQuery data warehouse for a multi-tenant SaaS application. Each tenant's data must be isolated and queried only by that tenant. You need to minimise management overhead and allow tenants to be added dynamically. Which approach should you use?

Accepted Answer

Use a single dataset with authorized views that filter by tenant_id, granting each tenant access to their view. Authorized views in a shared dataset allow you to create row-level security. By creating a view per tenant that filters by tenant_id, you can grant each tenant access only to its view. This avoids managing multiple datasets and is scalable.

Answer

Use Cloud IAM conditions on the dataset to filter by tenant_id

Answer

Create a separate dataset for each tenant and grant the tenant access to their dataset

Answer

Use a single table with a tenant_id column and enable column-level security to restrict access

Question 18

You need to process large-scale log files (hundreds of terabytes) using Apache Spark on Google Cloud. The job runs nightly and you want to minimise costs. Which Dataproc cluster configuration is MOST cost-effective?

Accepted Answer

Standard cluster with preemptible secondary workers. Preemptible VMs are significantly cheaper than standard VMs and suitable for fault-tolerant batch jobs like nightly Spark processing. Standard mode is fine but using preemptible workers reduces cost.

Answer

Single-node cluster

Answer

Standard cluster with preemptible workers for the primary worker nodes

Answer

Standard cluster with all standard (non-preemptible) workers

Question 19

A data pipeline ingests streaming events into Pub/Sub. You need to guarantee that each event is processed exactly once downstream in Dataflow. Which combination of Pub/Sub and Dataflow configurations should you use?

Accepted Answer

Use Pub/Sub with a unique message ID and Dataflow with idempotent writes or Dataflow's exactly-once sink. Pub/Sub offers at-least-once delivery. To achieve exactly-once processing, the pipeline must be idempotent or use Dataflow's exactly-once sinks. Using a unique message ID for deduplication is a common approach.

Answer

Use Pub/Sub with exactly-once delivery enabled and Dataflow with exactly-once processing

Answer

Use Pub/Sub with message deduplication and Dataflow with at-least-once processing

Answer

Use Pub/Sub with a dead letter topic and Dataflow with automatic retries

Question 20

You are designing a Dataflow pipeline that reads from Pub/Sub and writes to BigQuery. The pipeline must handle late-arriving data (up to 1 hour) and group events into 10-minute windows. Which configuration is correct?

Accepted Answer

Use fixed windows of 10 minutes with allowed lateness of 1 hour and a trigger that fires after watermark plus early firings. To handle late data, you need to set the allowed lateness to 1 hour. The trigger with AfterWatermark with early firings ensures results are emitted on time and updated when late data arrives.

Answer

Use global windows with a trigger that fires every 10 minutes

Answer

Use sliding windows of 10 minutes with a 5-minute period and allowed lateness of 1 hour

Answer

Use fixed windows of 10 minutes with allowed lateness of 0 seconds