PDE Designing Data Processing Systems Practice Test 6 — 15 Questions

Question 1

A data engineer needs to process streaming data from thousands of IoT devices and generate real-time dashboards. The data volume is low but requires exactly-once processing semantics. Which Google Cloud service combination should they use?

Accepted Answer

Cloud Pub/Sub + Cloud Dataflow. Dataflow supports exactly-once processing via its streaming engine and checkpointing. Pub/Sub is the ingest service. Together they provide the required semantics.

Answer

Cloud Pub/Sub + Cloud Data Fusion

Answer

Cloud Pub/Sub + Cloud Dataproc

Answer

Cloud Pub/Sub + Cloud Dataprep

Question 2

A company has a BigQuery dataset containing sensitive customer data. They want to share a subset of this data with external partners, ensuring that partners can only see specific columns and rows. Which BigQuery feature should they use?

Accepted Answer

Authorized views. Authorized views allow you to share a query (view) that filters columns and rows, while granting access to the view only, not the underlying tables.

Answer

Materialized views

Answer

Clustered tables

Answer

Dataset-level access controls

Question 3

A data pipeline is built with Cloud Dataflow that reads from Pub/Sub, applies transformations, and writes to BigQuery. The pipeline is experiencing high latency and occasional data loss during worker failures. The engineer wants to improve reliability and performance. Which two actions should they take?

Accepted Answer

Enable Dataflow Streaming Engine and use BigQuery exactly-once sink. Enabling streaming engine moves the state management to the backend, reducing latency and improving reliability. Using exactly-once sinks (like BigQuery with exactly-once guarantees) prevents data loss.

Answer

Switch to Cloud Dataproc and use Spark Structured Streaming

Answer

Increase the number of workers and use at-least-once delivery to BigQuery

Answer

Use a global window and disable triggers

Question 4

An organization runs periodic Apache Spark jobs on Dataproc to process data from Cloud Storage. They want to reduce costs by using preemptible instances for worker nodes. What is a key consideration when using preemptible instances in Dataproc?

Accepted Answer

Jobs must be designed to handle node preemption, and overall job runtime may increase. Preemptible VMs can be terminated at any time, so Spark jobs must be fault-tolerant. Dataproc handles this by automatically rescheduling failed tasks, but the job may take longer.

Answer

Preemptible instances cannot be used with the standard cluster mode

Answer

Preemptible instances are only available in certain regions

Answer

Jobs will automatically restart from the last checkpoint without any performance impact

Question 5

A company needs to process high-throughput streaming data with low latency. They are considering Cloud Pub/Sub for ingestion and Cloud Dataflow for processing. However, they are concerned about cost. Which alternative to Cloud Pub/Sub would reduce costs while still meeting the throughput requirements?

Accepted Answer

Cloud Pub/Sub Lite. Pub/Sub Lite is a cost-effective alternative for high-throughput streaming when you don't need global availability or some advanced features of Pub/Sub.

Answer

Cloud Pub/Sub with pull subscriptions

Answer

Cloud Tasks

Answer

Cloud Pub/Sub with push subscriptions

Question 6

A data engineer is designing a pipeline that reads from Cloud Pub/Sub, aggregates events into 5-minute windows, and writes the results to BigQuery. The engineer wants to ensure that late-arriving data (up to 2 minutes late) is included in the correct window. Which Dataflow feature should they configure?

Accepted Answer

Set the allowed lateness to 2 minutes with a trigger that fires on late data. Option C is correct because Dataflow's allowed lateness feature (set to 2 minutes) ensures that late-arriving data within that threshold is still assigned to the correct 5-minute window. Combined with a trigger that fires on late data, the pipeline can emit updated results for the window after the watermark passes, which is exactly what the engineer needs to handle late-arriving events up to 2 minutes late.

Answer

Use a sliding window of 5 minutes with 2-minute slide

Answer

Set the window duration to 7 minutes to account for lateness

Answer

Use a global window and watermark

Question 7

A company is using Cloud Storage to store raw logs. They want to use Cloud Data Fusion to transform and load the data into BigQuery on a daily schedule. The transformations are complex and involve joining multiple datasets. What is the most efficient way to run these pipelines?

Accepted Answer

Use Cloud Data Fusion to design the pipeline and schedule it to run on a Dataproc cluster. Cloud Data Fusion supports scheduling pipelines and runs them on Dataproc. This is the standard way to run batch pipelines.

Answer

Use Cloud Composer to orchestrate Dataproc jobs that run the transformations

Answer

Use Cloud Functions to trigger a Dataflow job that does the transformations

Answer

Use Cloud Dataprep to design the transformation and export to BigQuery

Question 8

A company has a BigQuery table that is partitioned by ingestion time and clustered by the 'customer_id' column. They notice that queries filtering on 'customer_id' are not benefiting from clustering as expected. What is the most likely cause?

Accepted Answer

The table is too small for clustering to be effective. Clustering works best when the clustering column is used in a filter that limits data scanned. However, if the filter is on a non-clustering column, or if the clustering column has high cardinality with many distinct values, clustering may not help much. In this case, the issue could be that the filter on 'customer_id' is not selective enough, or the table is too small.

Answer

The query is using a wildcard function that prevents clustering pruning

Answer

The clustering column must be the same as the partition column

Answer

Clustering does not work with ingestion-time partitioning

Question 9

An organization is implementing a data lake on Google Cloud using Cloud Storage. They need to process both batch and streaming data with a unified pipeline. The team has experience with Apache Beam. Which architecture should they use to minimize operational overhead?

Accepted Answer

Kappa architecture with Cloud Dataflow using the same pipeline for batch and streaming. Kappa architecture uses a single streaming pipeline for both batch and streaming, simplifying operations. Dataflow implements Beam and supports both modes.

Answer

Use Cloud Dataproc for batch and Cloud Dataflow for streaming

Answer

Lambda architecture with Cloud Dataflow for batch and Cloud Pub/Sub for streaming

Answer

Use Cloud Data Fusion for both batch and streaming

Question 10

A data pipeline using Cloud Dataflow reads from a Pub/Sub subscription that has a dead letter topic configured. Some messages are being sent to the dead letter topic. Upon investigation, the engineer finds that the messages contain valid data but are malformed according to the schema. What is the most likely reason for the messages being dead-lettered?

Accepted Answer

The Pub/Sub topic has a schema that the messages do not comply with. The subscription's message schema enforcement validates incoming messages; if the message doesn't conform to the schema, it is forwarded to the dead letter topic.

Answer

The Pub/Sub topic is not configured with a schema

Answer

The Dataflow pipeline is using at-least-once delivery guarantee

Answer

The subscription's ack deadline is too short

Question 11

A company uses Cloud Dataproc to run Spark ML training jobs. They want to persist the trained models and metadata in a Hive-compatible metastore. Which Dataproc feature should they use?

Accepted Answer

Dataproc Metastore. Dataproc Metastore provides a Hive-compatible metastore that can be used across clusters and services.

Answer

Cloud Hive Metastore (self-managed)

Answer

Cloud Bigtable

Answer

Cloud Data Catalog

Question 12

An engineer needs to create a Pub/Sub subscription that sends messages to an HTTPS endpoint. The endpoint must be able to acknowledge messages individually. Which type of subscription should they use?

Accepted Answer

Push subscription. Push subscriptions deliver messages to a configured HTTPS endpoint. The endpoint can acknowledge by returning a 200 status.

Answer

Pull subscription

Answer

BigQuery subscription

Answer

Cloud Storage subscription

Question 13

A company is migrating their on-premises Hadoop workloads to Google Cloud. They want to use Dataproc for data processing and need to minimize costs for non-critical batch jobs that can tolerate interruptions. Which TWO configurations should they use?

Accepted Answer

Use preemptible instances for worker nodes. Preemptible instances are cheaper and can be preempted, suitable for fault-tolerant batch jobs. Single-node clusters are cost-effective for small jobs.

Answer

Enable high-availability mode

Answer

Use standard (non-preemptible) instances for all nodes

Answer

Use Dataproc on GKE

Question 14

A data engineering team is designing a streaming pipeline using Cloud Dataflow. They need to join two unbounded PCollections based on a common key. The join must handle late data up to 10 minutes. Which THREE components should they use?

Accepted Answer

CoGroupByKey transform. CoGroupByKey joins two PCollections by key. Window into fixed windows of appropriate duration. Allowed lateness and triggers handle late data.

Answer

Use a global window with triggers

Answer

Flatten transform

Question 15

An organization is using BigQuery for analytics. They have a table that is 500 GB and is frequently queried by 'date' and 'region'. They want to optimize query performance and reduce costs. Which TWO actions should they take?

Accepted Answer

Cluster the table by region. Partitioning by date enables partition pruning to scan only relevant partitions. Clustering by region further reduces data scanned within partitions.

Answer

Use an authorized view

Answer

Use a wildcard table

Answer

Use materialized views