PDE Designing Data Processing Systems Practice Test 2 — 15 Questions

Question 1

A company wants to design a data pipeline for real-time fraud detection. The system must process streaming financial transactions, enrich them with user profiles from a lookup table, and flag suspicious activities within seconds. Which architecture pattern would be MOST suitable?

Accepted Answer

Kappa architecture using a single stream processing framework like Apache Beam. Kappa architecture uses a single stream processing engine to handle both real-time and batch reprocessing, simplifying the pipeline. Lambda architecture requires maintaining separate batch and streaming layers, increasing complexity. The scenario only requires real-time processing with enrichment, so Kappa is more appropriate.

Answer

Pub/Sub combined with Cloud Functions for stateless processing

Answer

Batch processing with hourly micro-batches using Dataflow

Answer

Lambda architecture with a batch layer for historical analysis and a speed layer for real-time processing

Question 2

You are designing a BigQuery data warehouse for a multi-tenant SaaS application. Each tenant's data must be isolated and queried only by that tenant. You need to minimise management overhead and allow tenants to be added dynamically. Which approach should you use?

Accepted Answer

Use a single dataset with authorized views that filter by tenant_id, granting each tenant access to their view. Authorized views in a shared dataset allow you to create row-level security. By creating a view per tenant that filters by tenant_id, you can grant each tenant access only to its view. This avoids managing multiple datasets and is scalable.

Answer

Use Cloud IAM conditions on the dataset to filter by tenant_id

Answer

Create a separate dataset for each tenant and grant the tenant access to their dataset

Answer

Use a single table with a tenant_id column and enable column-level security to restrict access

Question 3

You need to process large-scale log files (hundreds of terabytes) using Apache Spark on Google Cloud. The job runs nightly and you want to minimise costs. Which Dataproc cluster configuration is MOST cost-effective?

Accepted Answer

Standard cluster with preemptible secondary workers. Preemptible VMs are significantly cheaper than standard VMs and suitable for fault-tolerant batch jobs like nightly Spark processing. Standard mode is fine but using preemptible workers reduces cost.

Answer

Single-node cluster

Answer

Standard cluster with preemptible workers for the primary worker nodes

Answer

Standard cluster with all standard (non-preemptible) workers

Question 4

A data pipeline ingests streaming events into Pub/Sub. You need to guarantee that each event is processed exactly once downstream in Dataflow. Which combination of Pub/Sub and Dataflow configurations should you use?

Accepted Answer

Use Pub/Sub with a unique message ID and Dataflow with idempotent writes or Dataflow's exactly-once sink. Pub/Sub offers at-least-once delivery. To achieve exactly-once processing, the pipeline must be idempotent or use Dataflow's exactly-once sinks. Using a unique message ID for deduplication is a common approach.

Answer

Use Pub/Sub with exactly-once delivery enabled and Dataflow with exactly-once processing

Answer

Use Pub/Sub with message deduplication and Dataflow with at-least-once processing

Answer

Use Pub/Sub with a dead letter topic and Dataflow with automatic retries

Question 5

You are designing a Dataflow pipeline that reads from Pub/Sub and writes to BigQuery. The pipeline must handle late-arriving data (up to 1 hour) and group events into 10-minute windows. Which configuration is correct?

Accepted Answer

Use fixed windows of 10 minutes with allowed lateness of 1 hour and a trigger that fires after watermark plus early firings. To handle late data, you need to set the allowed lateness to 1 hour. The trigger with AfterWatermark with early firings ensures results are emitted on time and updated when late data arrives.

Answer

Use global windows with a trigger that fires every 10 minutes

Answer

Use sliding windows of 10 minutes with a 5-minute period and allowed lateness of 1 hour

Answer

Use fixed windows of 10 minutes with allowed lateness of 0 seconds

Question 6

You are moving an on-premises Hadoop workload to Google Cloud. The workload uses Hive for metadata and HDFS for storage. Which services should you use to minimise reconfiguration?

Accepted Answer

Dataproc with Cloud Storage and Dataproc Metastore. Dataproc Metastore provides a fully managed Hive metastore service that can be used with Dataproc clusters. Cloud Storage can replace HDFS via the gs:// connector, allowing the same file paths. This minimises code changes.

Answer

Dataproc with HDFS and Cloud Bigtable for metadata

Answer

Dataproc with Cloud Storage and Cloud SQL for Hive metastore

Answer

Dataflow with Cloud Storage and BigQuery

Question 7

Which Google Cloud service provides a visual interface for building ETL pipelines using a drag-and-drop design and includes pre-built transforms from a marketplace?

Accepted Answer

Cloud Data Fusion. Cloud Data Fusion offers a visual, code-free ETL tool with a rich set of plugins from the Hub. Dataprep is for data wrangling, not full ETL.

Answer

Dataproc

Answer

Dataprep

Answer

BigQuery

Question 8

You are designing a streaming pipeline that needs to handle sudden spikes in traffic without losing data. The pipeline uses Pub/Sub and Dataflow. Which configuration ensures data is not lost if Dataflow falls behind?

Accepted Answer

Use Pub/Sub with a pull subscription and set the message retention duration to 7 days. Pub/Sub stores messages for up to 7 days, allowing Dataflow to catch up. Dataflow uses checkpointing to track progress. This combination ensures no data loss.

Answer

Use Cloud Pub/Sub Lite with a smaller retention period

Answer

Use Pub/Sub with a push subscription and increase the acknowledgment deadline

Answer

Use Pub/Sub with exactly-once delivery and Dataflow with at-least-once processing

Question 9

You need to analyse streaming data from thousands of IoT devices, each sending temperature readings every second. You want to calculate the average temperature per device over the last 5 minutes, updating every minute. Which windowing strategy should you use in Dataflow?

Accepted Answer

Sliding windows of length 5 minutes with a period of 1 minute. Sliding windows of length 5 minutes with a period of 1 minute give the desired overlapping windows: every minute, you get the average over the last 5 minutes.

Answer

Global windows with a trigger firing every minute

Answer

Fixed windows of 5 minutes

Answer

Session windows with a gap duration of 1 minute

Question 10

A company uses BigQuery with partitioned tables by ingestion time. They notice that queries scanning recent partitions are fast but queries scanning older partitions are slow. What is the most likely cause?

Accepted Answer

The older partitions lack clustering metadata because clustering was enabled after data was ingested. Clustering can improve query performance by sorting data within partitions. If older partitions were created before clustering was enabled, they are not clustered, leading to slower scans. Re-clustering only applies to new data unless you manually rewrite older partitions.

Answer

Older partitions are stored on slower storage tiers

Answer

The table has too many partitions, causing high metadata overhead

Answer

Queries are using different SQL syntax for older partitions

Question 11

You need to run a one-time data transformation job on a small CSV file (100 MB) using a visual, code-free interface. Which Google Cloud service is designed for this?

Accepted Answer

Dataprep. Dataprep (Trifacta) is a visual data wrangling tool for exploring and transforming data without code. It's ideal for ad-hoc, small to medium datasets.

Answer

Dataflow

Answer

Cloud Data Fusion

Answer

Dataproc

Question 12

You are designing a Dataflow pipeline that joins two unbounded PCollections from different sources. Which transform should you use?

Accepted Answer

CoGroupByKey. CoGroupByKey performs a key-based join of multiple PCollections. It can handle unbounded streams with appropriate windowing.

Answer

ParDo

Answer

Flatten

Answer

GroupByKey

Question 13

A company wants to build a real-time dashboard for monitoring application logs. The logs are ingested via Pub/Sub and must be processed with low latency (sub-second). You need to enrich the logs with user metadata from Cloud SQL and store the results in BigQuery for analysis. Which TWO services should be used for the stream processing? (Choose two.)

Accepted Answer

Dataflow. Dataflow can read from Pub/Sub, enrich with side inputs from Cloud SQL, and write to BigQuery. Pub/Sub is the ingestion point. The pipeline uses Dataflow for stream processing.

Answer

Dataproc

Answer

Cloud Data Fusion

Answer

Cloud Functions

Question 14

A data pipeline processes sensitive customer data. You need to ensure that only authorised users can query the data in BigQuery, and that the data is encrypted at rest and in transit. Which THREE steps should you take? (Choose three.)

Accepted Answer

Create authorised views in BigQuery to restrict access to sensitive columns. Encryption at rest is enabled by default with CMEK or CSEK. In-transit encryption is default for BigQuery. Authorised views provide fine-grained access control. IAM roles control dataset access.

Answer

Use a Cloud VPN to encrypt data in transit between on-premises and Google Cloud

Answer

Grant the bigquery.dataViewer role at the dataset level to all users

Question 15

You are designing a Dataflow pipeline for processing real-time clickstream data. The pipeline must group events into 30-second windows and handle late data up to 5 minutes. You want to output partial results every 10 seconds for low-latency monitoring. Which TWO configurations should you use? (Choose two.)

Accepted Answer

Use fixed windows of 30 seconds. Fixed windows of 30 seconds with allowed lateness 5 minutes. Early firings every 10 seconds produce partial results.

Answer

Use sliding windows of 30 seconds with a 10-second period

Answer

Use a trigger that fires after the end of the window