PDE Designing Data Processing Systems Practice Test 4 — 15 Questions

Question 1

You are designing a Dataflow pipeline that reads from Pub/Sub, aggregates events into 10-minute windows, and writes the results to BigQuery. The pipeline must reliably handle late-arriving data (up to 1 hour) and prevent duplicate aggregations. Which combination of pipeline options should you use?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Use exactly-once processing by setting the pipeline's streaming engine to exactly-once and using a BigQuery sink with exactly-once semantics. To prevent duplicate aggregations, you need exactly-once processing. Dataflow supports exactly-once sinks (like BigQuery) when using the FILE_LOADS method or streaming inserts with exactly-once semantics. Using at-least-once with deduplication in BigQuery is not reliable. Exactly-once semantics require idempotent writes; the recommended approach is to use the BigQuery sink with exactly-once support (by setting the trigger frequency appropriately and using a dedup key). However, the simplest way is to use the Dataflow streaming engine and the BigQuery sink with exactly-once enabled. Among the options, using the BigQuery streaming inserts with exactly-once semantics (available in Dataflow) is correct.

Answer

Use at-least-once processing and rely on BigQuery's automatic deduplication

Answer

Use exactly-once processing and write results to a staging table, then use a scheduled merge query to combine with the main table

Answer

Use at-most-once processing to guarantee no duplicates, and accept data loss

Question 2

You need to create a BigQuery table that stores customer transaction data. The table will be queried frequently by a customer_id column to retrieve recent transactions (last 30 days). Which table design optimizes query performance and cost?

Accepted Answer

Partition by transaction_date and cluster by customer_id. Partitioning by transaction_date allows queries to scan only relevant partitions. Clustering by customer_id sorts data within each partition by customer_id, further reducing the amount of data scanned for queries filtering on customer_id. This combination is best for time-range queries with frequent customer_id filters.

Answer

Partition by customer_id and cluster by transaction_date

Answer

Partition by ingestion_time and cluster by customer_id

Answer

Cluster by transaction_date and customer_id without partitioning

Question 3

Your company is building a real-time anomaly detection system for financial transactions. The system must process streams of transactions and flag anomalies within seconds. The volume is moderate (5000 transactions per second). You want a fully managed solution that integrates with BigQuery for historical analysis. Which service should you use for stream processing?

Accepted Answer

Cloud Dataflow. Cloud Dataflow is a fully managed service ideal for real-time stream processing with low latency. It can read from Pub/Sub, perform transformations (e.g., anomaly detection), and write to BigQuery for historical analysis. Dataproc requires cluster management; Data Fusion is batch-oriented; Pub/Sub alone does not process data.

Answer

Cloud Pub/Sub with push subscriptions

Answer

Cloud Dataproc with Spark Streaming

Answer

Cloud Data Fusion

Question 4

Your organization is designing a data lake on Google Cloud using Cloud Storage. You need to choose a file format for storing raw data that supports schema evolution, is splittable for parallel processing, and is optimized for query performance in BigQuery. Which TWO formats meet these requirements? (Choose 2.)

Accepted Answer

Avro. Both Avro and Parquet support schema evolution (through schemas) and are splittable. Parquet is columnar and highly optimized for BigQuery performance. Avro is row-oriented but also splittable and supports schema evolution. CSV and JSON do not natively support schema evolution and are less performant for BigQuery. ORC is not natively supported by BigQuery.

Answer

CSV

Answer

JSON (newline-delimited)

Answer

ORC

Question 5

Your company runs a Dataflow streaming pipeline that processes user activity from Pub/Sub and writes aggregated results to BigQuery. Lately, the pipeline is experiencing high latency and backlog growth during peak hours. You need to troubleshoot and improve performance. Which THREE actions should you take? (Choose 3.)

Accepted Answer

Change the worker machine type to a higher CPU/memory configuration. Increasing the number of workers allows the pipeline to process more data in parallel. Using streaming engine can improve throughput and reduce latency by offloading state management. Adjusting the worker machine type to use more CPU/memory can help if the processing is compute-intensive. Adding more subscriptions would not help because the pipeline reads from a single subscription. Changing the window size affects business logic but not necessarily performance. Combining these three optimizations addresses common bottlenecks.

Answer

Decrease the window duration to reduce data per window

Answer

Add additional Pub/Sub subscriptions to the same topic

Question 6

Your team is using Cloud Dataprep to clean and transform a dataset. Which TWO features of Cloud Dataprep help you understand data quality issues before running the pipeline? (Choose 2.)

Accepted Answer

Column histograms. Data quality profiling provides statistics and distributions to identify anomalies. Column histograms visualize data distribution and outliers. Scheduling and recipe steps are execution features, not exploratory analysis. Joins are transformations, not profiling.

Answer

Scheduling data quality jobs

Answer

Joining datasets

Answer

Recipe steps

Question 7

A company needs to process streaming sensor data from millions of devices with sub-second latency, apply transformations, and write results to BigQuery for real-time dashboards. The data volume varies, and they want to avoid managing servers. Which service should they use?

Accepted Answer

Dataflow. Dataflow is a fully managed, serverless stream and batch processing service that can handle high-throughput streaming with sub-second latency.

Answer

Cloud Data Fusion

Answer

Dataproc

Answer

Dataprep

Question 8

A company uses Cloud Pub/Sub to ingest events from multiple sources. They need to guarantee that each event is processed exactly once by downstream consumers. However, Pub/Sub guarantees at-least-once delivery. Which additional steps should they implement to achieve exactly-once processing?

Accepted Answer

Store each message's unique ID in a database and ignore duplicates.. Pub/Sub only provides at-least-once; exactly-once requires consumer-side deduplication using a unique message ID.

Answer

Set the subscription's acknowledgment deadline to 0.

Answer

Enable message deduplication on the subscription.

Answer

Use a dead letter topic to capture duplicates.

Question 9

A data pipeline uses Dataflow to read from Pub/Sub, window messages into 1-minute fixed windows, and write to BigQuery. The pipeline occasionally has late-arriving data. How should they configure the pipeline to allow late data up to 5 minutes and then trigger a final pane?

Accepted Answer

withAllowedLateness(Duration.standardMinutes(5)).triggering(AfterWatermark.pastEndOfWindow().withLateFirings(AfterPane.elementCountAtLeast(1))). In Beam, allowed lateness and triggering combine to handle late data.

Answer

withAllowedLateness(Duration.standardMinutes(5)).triggering(Repeatedly.forever(AfterPane.elementCountAtLeast(1)))

Answer

triggering(AfterWatermark.pastEndOfWindow()).withAllowedLateness(Duration.standardMinutes(5))

Answer

withAllowedLateness(Duration.standardMinutes(5)).accumulatingFiredPanes()

Question 10

Which Google Cloud service provides a fully managed, serverless Spark environment without requiring cluster provisioning?

Accepted Answer

Dataproc Serverless. Dataproc Serverless allows running Spark workloads without managing clusters.

Answer

Dataproc on GKE

Answer

Dataflow

Answer

Cloud Data Fusion

Question 11

A team wants to use Cloud Pub/Sub Lite for a high-throughput, low-cost messaging system. They need exactly-once delivery to subscribers. What should they know about Pub/Sub Lite's delivery guarantees?

Accepted Answer

Pub/Sub Lite provides at-least-once delivery, same as standard Pub/Sub.. Pub/Sub Lite offers at-least-once delivery like standard Pub/Sub; exactly-once is not guaranteed.

Answer

Pub/Sub Lite provides exactly-once delivery when using push subscriptions.

Answer

Pub/Sub Lite provides exactly-once delivery when using pull subscriptions.

Answer

Pub/Sub Lite supports exactly-once delivery by default.

Question 12

A company wants to use BigQuery materialized views to accelerate queries on a table that is updated every hour. Which statement about materialized views is true?

Accepted Answer

Materialized views are automatically updated when the base table changes.. BigQuery materialized views are automatically refreshed when base tables are changed.

Answer

Materialized views cannot be clustered.

Answer

Materialized views must be manually refreshed by the user.

Answer

Materialized views can only be created on ingestion-time partitioned tables.

Question 13

Which BigQuery feature allows you to share query results with specific users without giving them direct access to the underlying tables?

Accepted Answer

Authorized views. Authorized views allow sharing results without granting access to the base tables.

Answer

IAM roles

Answer

Dataset access controls

Answer

Materialized views

Question 14

A company wants to use Cloud Data Fusion for ETL pipelines. They need to integrate with custom transformations not available in the marketplace. What should they do?

Accepted Answer

Write a custom plugin using the CDAP SDK and deploy it.. Cloud Data Fusion supports custom plugins using the CDAP SDK.

Answer

Switch to Dataproc and write a Spark job.

Answer

Use the Data Fusion Hub to download a custom plugin.

Answer

Use Dataprep to create the transformation.

Question 15

A Dataproc cluster uses preemptible worker nodes to reduce costs. The cluster runs a long-running Spark job that occasionally experiences worker failures. How should the job be configured to handle preemptible worker failures gracefully?

Accepted Answer

Set spark.task.maxFailures to a high number to allow retries.. Spark jobs should use checkpointing and handle task retries to survive preemption.

Answer

Disable preemptible workers for the job.

Answer

Use persistent disks for preemptible workers.

Answer

Enable automatic restart of the Spark driver on failure.