PDE Designing Data Processing Systems Practice Test 4 — 15 Questions

Question 1

A company needs to process streaming sensor data from millions of devices with sub-second latency, apply transformations, and write results to BigQuery for real-time dashboards. The data volume varies, and they want to avoid managing servers. Which service should they use?

Accepted Answer

Dataflow. Dataflow is a fully managed, serverless stream and batch processing service that can handle high-throughput streaming with sub-second latency.

Answer

Cloud Data Fusion

Answer

Dataproc

Answer

Dataprep

Question 2

A data engineer wants to create a BigQuery table that is partitioned by day and clustered by user_id and product_id. Which SQL statement should they use?

Accepted Answer

CREATE TABLE mydataset.table (user_id INT64, product_id INT64, event_date DATE) PARTITION BY event_date CLUSTER BY user_id, product_id;. The correct syntax for creating a partitioned and clustered table in BigQuery uses PARTITION BY and CLUSTER BY.

Answer

CREATE TABLE mydataset.table (user_id INT64, product_id INT64, event_date DATE) PARTITION BY DATE(event_date) CLUSTER BY user_id, product_id;

Answer

CREATE TABLE mydataset.table (user_id INT64, product_id INT64, event_date DATE) CLUSTER BY user_id, product_id PARTITION BY event_date;

Question 3

A company uses Cloud Pub/Sub to ingest events from multiple sources. They need to guarantee that each event is processed exactly once by downstream consumers. However, Pub/Sub guarantees at-least-once delivery. Which additional steps should they implement to achieve exactly-once processing?

Accepted Answer

Store each message's unique ID in a database and ignore duplicates.. Pub/Sub only provides at-least-once; exactly-once requires consumer-side deduplication using a unique message ID.

Answer

Set the subscription's acknowledgment deadline to 0.

Answer

Enable message deduplication on the subscription.

Answer

Use a dead letter topic to capture duplicates.

Question 4

A data pipeline uses Dataflow to read from Pub/Sub, window messages into 1-minute fixed windows, and write to BigQuery. The pipeline occasionally has late-arriving data. How should they configure the pipeline to allow late data up to 5 minutes and then trigger a final pane?

Accepted Answer

withAllowedLateness(Duration.standardMinutes(5)).triggering(AfterWatermark.pastEndOfWindow().withLateFirings(AfterPane.elementCountAtLeast(1))). In Beam, allowed lateness and triggering combine to handle late data.

Answer

withAllowedLateness(Duration.standardMinutes(5)).triggering(Repeatedly.forever(AfterPane.elementCountAtLeast(1)))

Answer

triggering(AfterWatermark.pastEndOfWindow()).withAllowedLateness(Duration.standardMinutes(5))

Answer

withAllowedLateness(Duration.standardMinutes(5)).accumulatingFiredPanes()

Question 5

Which Google Cloud service provides a fully managed, serverless Spark environment without requiring cluster provisioning?

Accepted Answer

Dataproc Serverless. Dataproc Serverless allows running Spark workloads without managing clusters.

Answer

Dataproc on GKE

Answer

Dataflow

Answer

Cloud Data Fusion

Question 6

A team wants to use Cloud Pub/Sub Lite for a high-throughput, low-cost messaging system. They need exactly-once delivery to subscribers. What should they know about Pub/Sub Lite's delivery guarantees?

Accepted Answer

Pub/Sub Lite provides at-least-once delivery, same as standard Pub/Sub.. Pub/Sub Lite offers at-least-once delivery like standard Pub/Sub; exactly-once is not guaranteed.

Answer

Pub/Sub Lite provides exactly-once delivery when using push subscriptions.

Answer

Pub/Sub Lite provides exactly-once delivery when using pull subscriptions.

Answer

Pub/Sub Lite supports exactly-once delivery by default.

Question 7

A company wants to use BigQuery materialized views to accelerate queries on a table that is updated every hour. Which statement about materialized views is true?

Accepted Answer

Materialized views are automatically updated when the base table changes.. BigQuery materialized views are automatically refreshed when base tables are changed.

Answer

Materialized views cannot be clustered.

Answer

Materialized views must be manually refreshed by the user.

Answer

Materialized views can only be created on ingestion-time partitioned tables.

Question 8

A Dataflow pipeline processes a high-volume stream of JSON events. The pipeline has a bottleneck where a ParDo transformation performs an external API call for each element, causing high latency. Which strategy would BEST improve throughput without sacrificing correctness?

Accepted Answer

Use a side input to batch elements and make fewer API calls.. Using side inputs to batch data before API calls can reduce the number of calls and improve throughput.

Answer

Increase the number of workers in the pipeline.

Answer

Switch from ParDo to MapElements.

Answer

Use a GroupByKey to group elements with the same key and then make one API call per group.

Question 9

Which BigQuery feature allows you to share query results with specific users without giving them direct access to the underlying tables?

Accepted Answer

Authorized views. Authorized views allow sharing results without granting access to the base tables.

Answer

IAM roles

Answer

Dataset access controls

Answer

Materialized views

Question 10

A company wants to use Cloud Data Fusion for ETL pipelines. They need to integrate with custom transformations not available in the marketplace. What should they do?

Accepted Answer

Write a custom plugin using the CDAP SDK and deploy it.. Cloud Data Fusion supports custom plugins using the CDAP SDK.

Answer

Switch to Dataproc and write a Spark job.

Answer

Use the Data Fusion Hub to download a custom plugin.

Answer

Use Dataprep to create the transformation.

Question 11

A Dataproc cluster uses preemptible worker nodes to reduce costs. The cluster runs a long-running Spark job that occasionally experiences worker failures. How should the job be configured to handle preemptible worker failures gracefully?

Accepted Answer

Set spark.task.maxFailures to a high number to allow retries.. Spark jobs should use checkpointing and handle task retries to survive preemption.

Answer

Disable preemptible workers for the job.

Answer

Use persistent disks for preemptible workers.

Answer

Enable automatic restart of the Spark driver on failure.

Question 12

A company needs to process data from a legacy system that outputs CSV files daily. They want to visually build transformations without writing code. Which Google Cloud service should they use?

Accepted Answer

Dataprep. Dataprep is a visual data wrangling tool for exploring and cleaning data.

Answer

Dataproc

Answer

Dataflow

Answer

Cloud Data Fusion

Question 13

A company uses Cloud Pub/Sub for event ingestion. They want to ensure that if a subscriber fails to process a message after 5 attempts, the message is sent to a dead letter topic for analysis. Which TWO configurations are needed?

Accepted Answer

Set max delivery attempts to 5 on the subscription.. Dead letter topics require setting max delivery attempts and specifying a dead letter topic.

Answer

Set the subscription's ack deadline to 600 seconds.

Answer

Enable message ordering on the subscription.

Answer

Set the subscription type to push.

Question 14

A company is designing a data pipeline using the lambda architecture. They need to process both real-time streams and batch historical data. Which THREE components are essential for a lambda architecture on Google Cloud?

Accepted Answer

Cloud Pub/Sub for real-time ingestion. Lambda architecture has batch, speed, and serving layers.

Answer

BigQuery as the serving layer

Answer

Cloud SQL as the serving database

Answer

Cloud Bigtable for serving

Question 15

A company wants to use Dataproc Metastore to manage metadata for their Spark jobs. Which TWO benefits does Dataproc Metastore provide?

Accepted Answer

High availability with automatic failover. Dataproc Metastore offers a managed Hive metastore with high availability and compatibility.

Answer

Automatic scaling of compute resources

Answer

Integration with BigQuery

Answer

Built-in data lineage tracking