PDE Designing Data Processing Systems Practice Test 2 — 15 Questions

Question 1

A company wants to build a real-time dashboard for monitoring application logs. The logs are ingested via Pub/Sub and must be processed with low latency (sub-second). You need to enrich the logs with user metadata from Cloud SQL and store the results in BigQuery for analysis. Which TWO services should be used for the stream processing? (Choose two.)

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Dataflow. Option C (Dataflow) is correct because it is Google Cloud's fully managed Apache Beam service designed for low-latency, sub-second stream processing, and it natively supports reading from Pub/Sub, enriching records via external sources such as Cloud SQL (e.g., using JdbcIO), and writing results into BigQuery with exactly-once semantics. Option E (Pub/Sub) is correct because it is the ingestion and messaging backbone specified in the scenario: it receives the application logs and delivers them as a continuous stream to the processing layer, providing the durable, scalable transport that enables real-time consumption. Together, Pub/Sub feeds the stream and Dataflow transforms and enriches it before loading BigQuery. Option A (Dataproc) is not appropriate here because it is a batch-oriented Hadoop/Spark managed cluster service, not a serverless low-latency streaming pipeline for this use case. Option B (Cloud Data Fusion) is a codeless ETL/ELT tool built on CDAP that is better suited to batch or scheduled data integration than sub-second streaming enrichment. Option D (Cloud Functions) is event-driven and can react to Pub/Sub messages, but it is not designed for stateful, high-throughput stream processing with windowing and enrichment at scale.

Answer

Dataproc

Answer

Cloud Data Fusion

Answer

Cloud Functions

Question 2

A data pipeline processes sensitive customer data. You need to ensure that only authorised users can query the data in BigQuery, and that the data is encrypted at rest and in transit. Which THREE steps should you take? (Choose three.)

Accepted Answer

Create authorised views in BigQuery to restrict access to sensitive columns. Option C is correct because authorised views in BigQuery let you expose only specific columns or rows to users, so sensitive fields remain hidden while authorised users can still query the permitted data. Option D is correct because enabling CMEK for BigQuery ensures data at rest is encrypted with keys you control, satisfying the encryption-at-rest requirement. Option E is correct because IAM conditions can restrict access based on attributes such as the requester's IP address, adding context-aware authorisation so only approved networks can query the data. Option A is not correct because Cloud VPN encrypts traffic between on-premises and Google Cloud, but it does not govern BigQuery query authorisation or BigQuery data-at-rest encryption. Option B is not correct because granting bigquery.dataViewer at the dataset level to all users would broadly allow data access rather than restricting it to authorised users.

Answer

Use a Cloud VPN to encrypt data in transit between on-premises and Google Cloud

Answer

Grant the bigquery.dataViewer role at the dataset level to all users

Question 3

You are designing a batch data pipeline that runs daily to ingest data from an on-premises database into BigQuery. The ingestion volume is approximately 50 GB per day. The data must be available in BigQuery by 6 AM each day. The on-premises database supports change data capture (CDC) via logs. Which approach minimizes operational cost and complexity?

Accepted Answer

Use Cloud Dataflow with a JDBC source in batch mode to read CDC logs and write to BigQuery. Cloud Dataflow in batch mode with a JDBC source reading CDC logs and writing to BigQuery is the lowest-cost, lowest-complexity option for a daily 50 GB batch ingestion. It uses a fully managed, serverless batch pipeline that reads from the on-premises database via JDBC, processes CDC records, and loads BigQuery — no streaming infrastructure, no cluster management, and no always-on resources. This matches the daily batch requirement without over-engineering.

Answer

Use Cloud Dataproc with Spark Streaming to ingest CDC logs

Answer

Use Pub/Sub with a Dataflow streaming pipeline

Answer

Use Cloud Data Fusion with a batch pipeline

Question 4

Your company uses Cloud Data Fusion to build ETL pipelines. You have a pipeline that reads from Cloud Storage, transforms data using a custom Wrangler recipe, and writes to BigQuery. The pipeline is failing with an error indicating that the Wrangler directive is invalid. You have verified the recipe works in the Cloud Data Fusion Studio. What is the most likely cause of the failure?

Accepted Answer

The Wrangler plugin is not deployed in the Cloud Data Fusion instance. The Wrangler plugin must be deployed to the Cloud Data Fusion instance before a pipeline can execute directives that depend on it. Even if the recipe previews correctly in Studio (which may use a different runtime context), the pipeline execution fails when the Wrangler transform artifact is missing from the instance's plugin registry. Deploying the Wrangler plugin from the Hub resolves the 'invalid directive' error.

Answer

The pipeline is using a different version of the Wrangler plugin

Answer

The Cloud Storage bucket is in a different region than the Data Fusion instance

Answer

The service account used in the pipeline does not have permissions to write to BigQuery

Question 5

Your team is migrating a legacy batch processing system that uses Apache Spark on-premises. The migration must be completed with minimal code changes and support both batch and streaming in the future. You want to use a fully managed service. Which Google Cloud service is most appropriate?

Accepted Answer

Cloud Dataproc (standard cluster). Cloud Dataproc (standard cluster) runs managed Apache Spark/Hadoop clusters, so existing on-premises Spark code can be lifted and shifted with minimal changes, and it supports both batch and streaming (Spark Streaming/Structured Streaming). It is the most appropriate fully managed service for a Spark migration.

Answer

Cloud Data Fusion

Answer

Cloud Dataflow

Answer

Cloud Dataproc Serverless

Question 6

You are designing a Dataflow pipeline that reads from Pub/Sub, aggregates events into 10-minute windows, and writes the results to BigQuery. The pipeline must reliably handle late-arriving data (up to 1 hour) and prevent duplicate aggregations. Which combination of pipeline options should you use?

Accepted Answer

Use exactly-once processing by setting the pipeline's streaming engine to exactly-once and using a BigQuery sink with exactly-once semantics. To prevent duplicate aggregations, you need exactly-once processing. Dataflow supports exactly-once sinks (like BigQuery) when using the FILE_LOADS method or streaming inserts with exactly-once semantics. Using at-least-once with deduplication in BigQuery is not reliable. Exactly-once semantics require idempotent writes; the recommended approach is to use the BigQuery sink with exactly-once support (by setting the trigger frequency appropriately and using a dedup key). However, the simplest way is to use the Dataflow streaming engine and the BigQuery sink with exactly-once enabled. Among the options, using the BigQuery streaming inserts with exactly-once semantics (available in Dataflow) is correct.

Answer

Use at-least-once processing and rely on BigQuery's automatic deduplication

Answer

Use exactly-once processing and write results to a staging table, then use a scheduled merge query to combine with the main table

Answer

Use at-most-once processing to guarantee no duplicates, and accept data loss

Question 7

Your organization is designing a data lake on Google Cloud using Cloud Storage. You need to choose a file format for storing raw data that supports schema evolution, is splittable for parallel processing, and is optimized for query performance in BigQuery. Which TWO formats meet these requirements? (Choose 2.)

Accepted Answer

Avro. Avro (A) is correct because it stores the schema alongside the data in the file header, which enables robust schema evolution, and its block-based container format is splittable, allowing parallel processing across workers. Parquet (C) is correct because it is a columnar format that supports schema evolution via schema merging, is splittable, and provides excellent query performance in BigQuery through column pruning and predicate pushdown. CSV (B) is not suitable because it is row-based, has no embedded schema, and is poorly optimized for BigQuery query performance. JSON newline-delimited (D) lacks an enforced schema and is not efficiently splittable or columnar, hurting query performance. ORC (E) is columnar and splittable but is not natively supported by BigQuery as an external table format, so it does not meet the BigQuery optimization requirement.

Answer

CSV

Answer

JSON (newline-delimited)

Answer

ORC

Question 8

A data engineer wants to create a BigQuery table that is partitioned by day and clustered by user_id and product_id. Which SQL statement should they use?

Accepted Answer

CREATE TABLE mydataset.table (user_id INT64, product_id INT64, event_date DATE) PARTITION BY event_date CLUSTER BY user_id, product_id;. The correct syntax for creating a partitioned and clustered table in BigQuery requires PARTITION BY followed by CLUSTER BY in that order. Since `event_date` is of type DATE, `PARTITION BY event_date` correctly partitions by day. Option A unnecessarily wraps `event_date` in `DATE()`, which is only needed for TIMESTAMP columns. Option C is a duplicate of B and cannot be correct in a single-select question. Option D incorrectly places `CLUSTER BY` before `PARTITION BY`, which is invalid.

Answer

CREATE TABLE mydataset.table (user_id INT64, product_id INT64, event_date DATE) PARTITION BY DATE(event_date) CLUSTER BY user_id, product_id;

Answer

CREATE TABLE mydataset.table (user_id INT64, product_id INT64, event_date DATE) PARTITION BY event_date CLUSTER BY user_id, product_id;

Answer

CREATE TABLE mydataset.table (user_id INT64, product_id INT64, event_date DATE) CLUSTER BY user_id, product_id PARTITION BY event_date;

Question 9

A data pipeline uses Dataflow to read from Pub/Sub, window messages into 1-minute fixed windows, and write to BigQuery. The pipeline occasionally has late-arriving data. How should they configure the pipeline to allow late data up to 5 minutes and then trigger a final pane?

Accepted Answer

withAllowedLateness(Duration.standardMinutes(5)).triggering(AfterWatermark.pastEndOfWindow().withLateFirings(AfterPane.elementCountAtLeast(1))). The correct configuration pairs withAllowedLateness(Duration.standardMinutes(5)) with a trigger that fires the on-time pane at the watermark and then fires late panes as data arrives. AfterWatermark.pastEndOfWindow().withLateFirings(...) is the canonical Apache Beam trigger for this pattern: it emits a final result at end-of-window and continues emitting late panes for any element arriving within the 5-minute allowed lateness. This satisfies the requirement to allow late data up to 5 minutes and then produce a final pane.

Answer

withAllowedLateness(Duration.standardMinutes(5)).triggering(Repeatedly.forever(AfterPane.elementCountAtLeast(1)))

Answer

triggering(AfterWatermark.pastEndOfWindow()).withAllowedLateness(Duration.standardMinutes(5))

Answer

withAllowedLateness(Duration.standardMinutes(5)).accumulatingFiredPanes()

Question 10

A Dataflow pipeline processes a high-volume stream of JSON events. The pipeline has a bottleneck where a ParDo transformation performs an external API call for each element, causing high latency. Which strategy would BEST improve throughput without sacrificing correctness?

Accepted Answer

Use a GroupByKey to group elements with the same key and then make one API call per group.. Grouping elements by key with GroupByKey allows batching multiple elements that require the same API call, reducing the number of external calls and improving throughput. Side inputs are not designed for dynamic batching; they provide static data to all elements. Increasing workers or switching to MapElements does not address the core issue of per-element API calls.

Answer

Increase the number of workers in the pipeline.

Answer

Switch from ParDo to MapElements.

Answer

Use a side input to batch elements and make fewer API calls.

Question 11

A company needs to process data from a legacy system that outputs CSV files daily. They want to visually build transformations without writing code. Which Google Cloud service should they use?

Accepted Answer

Dataprep. Dataprep by Trifacta is a Google Cloud service designed for visually exploring, cleaning, and transforming data without writing code. It supports CSV inputs from Cloud Storage and provides a graphical interface for building transformation recipes, making it the right choice for a no-code ETL requirement on daily CSV files.

Answer

Dataproc

Answer

Dataflow

Answer

Cloud Data Fusion

Question 12

A company is designing a data pipeline using the lambda architecture. They need to process both real-time streams and batch historical data. Which THREE components are essential for a lambda architecture on Google Cloud?

Accepted Answer

Dataflow for batch processing. The three essential components for a lambda architecture on Google Cloud are: Dataflow for batch processing (batch layer), Cloud Pub/Sub for real-time ingestion (used by both batch and speed layers), and Dataflow for stream processing (speed layer). These form the batch, ingestion, and speed layers. The serving layer (e.g., BigQuery) is also important but not among the three essential components listed here.

Answer

Cloud SQL as the serving database

Answer

Cloud Bigtable for serving

Question 13

A company needs to process streaming sensor data and run both real-time analytics and batch reanalysis on historical data. They want to minimize infrastructure management. Which architecture and service combination is MOST suitable?

Accepted Answer

Kappa architecture with Pub/Sub and Dataflow for both real-time and batch processing. Kappa architecture uses a single streaming pipeline for both real-time and reprocessing, and Pub/Sub + Dataflow (with Beam) supports unified batch and stream processing with serverless, autoscaling infrastructure. This minimizes infrastructure management while meeting both real-time and historical reanalysis needs.

Answer

Lambda architecture with Pub/Sub for streaming and Cloud Storage for batch, processed by Dataflow

Answer

Batch processing only with Dataflow and Cloud Storage, ignoring real-time needs

Answer

Kappa architecture with Pub/Sub Lite and Dataflow Serverless

Question 14

A Dataflow streaming pipeline reads from Pub/Sub, processes events with a fixed window of 1 minute, and writes to BigQuery. Some events arrive late due to network issues. You need to ensure late events are still included in the correct window but the pipeline must not wait indefinitely. What configuration should you use?

Accepted Answer

Set allowed lateness to 5 minutes and use the default trigger. Setting allowed lateness to 5 minutes with the default trigger ensures that late-arriving events are still assigned to their correct 1-minute fixed window and emitted when the window closes plus the allowed lateness period. The default trigger fires when the watermark passes the end of the window, and allowed lateness extends the window's acceptance of late data without waiting indefinitely. This balances correctness for late events with a bounded wait, satisfying both requirements.

Answer

Use a sliding window of 1 minute with a 1-minute period

Answer

Use a global window with a trigger that fires every 10 seconds

Answer

Increase the watermark estimate to 10 minutes

Question 15

A Dataflow pipeline with multiple steps uses a side input from a slowly changing reference table stored in BigQuery. The side input is updated every hour. To avoid reprocessing the entire pipeline on each update, which approach should you use?

Accepted Answer

Use a side input with a custom 'AsIterable' and a 'Repeatable' trigger that refreshes the side input every hour. In Apache Beam/Dataflow, a side input that must be refreshed periodically without reprocessing the entire pipeline should use a custom windowing/trigger strategy — specifically a side input with a repeatable trigger that fires every hour, so the side input is re-read and re-emitted on that cadence while the main pipeline continues processing. This avoids restarting or re-running the whole pipeline when the reference table changes.

Answer

Use a side input with a periodic refresh via a DoFn that reads BigQuery on each element

Answer

Use the side input with a default trigger and 'withAllowedLateness'

Answer

Use a global window with a trigger that fires every hour