PDE Designing Data Processing Systems Practice Test 5 — 15 Questions

Question 1

A company needs to process data from a legacy system that outputs CSV files daily. They want to visually build transformations without writing code. Which Google Cloud service should they use?

Accepted Answer

Dataprep. Dataprep is a visual data wrangling tool for exploring and cleaning data.

Answer

Dataproc

Answer

Dataflow

Answer

Cloud Data Fusion

Question 2

A company uses Cloud Pub/Sub for event ingestion. They want to ensure that if a subscriber fails to process a message after 5 attempts, the message is sent to a dead letter topic for analysis. Which TWO configurations are needed?

Accepted Answer

Set max delivery attempts to 5 on the subscription.. Dead letter topics require setting max delivery attempts and specifying a dead letter topic.

Answer

Set the subscription's ack deadline to 600 seconds.

Answer

Enable message ordering on the subscription.

Answer

Set the subscription type to push.

Question 3

A company wants to use Dataproc Metastore to manage metadata for their Spark jobs. Which TWO benefits does Dataproc Metastore provide?

Accepted Answer

High availability with automatic failover. Dataproc Metastore offers a managed Hive metastore with high availability and compatibility.

Answer

Automatic scaling of compute resources

Answer

Integration with BigQuery

Answer

Built-in data lineage tracking

Question 4

A company needs to process streaming sensor data and run both real-time analytics and batch reanalysis on historical data. They want to minimize infrastructure management. Which architecture and service combination is MOST suitable?

Accepted Answer

Kappa architecture with Pub/Sub and Dataflow for both real-time and batch processing. Kappa architecture processes all data as a stream, avoiding separate batch/speed layers. Pub/Sub ingests streaming data, and Dataflow with Apache Beam can handle both real-time and batch (replay) pipelines, minimizing infrastructure management.

Answer

Lambda architecture with Pub/Sub for streaming and Cloud Storage for batch, processed by Dataflow

Answer

Batch processing only with Dataflow and Cloud Storage, ignoring real-time needs

Answer

Kappa architecture with Pub/Sub Lite and Dataflow Serverless

Question 5

You are designing a BigQuery data warehouse for a retail company. Queries frequently filter on order_date and customer_id. To optimize query performance and cost, which table design should you use?

Accepted Answer

Partition by order_date and cluster by customer_id. Partitioning on order_date limits scans to relevant date ranges. Clustering on customer_id further organizes data within partitions, improving filter and aggregation queries on customer_id.

Answer

Cluster by order_date and partition by customer_id

Answer

Partition by ingestion_time and cluster by order_date

Answer

Use a clustered table without partitioning

Question 6

A Dataflow streaming pipeline reads from Pub/Sub, processes events with a fixed window of 1 minute, and writes to BigQuery. Some events arrive late due to network issues. You need to ensure late events are still included in the correct window but the pipeline must not wait indefinitely. What configuration should you use?

Accepted Answer

Set allowed lateness to 5 minutes and use the default trigger. Setting a watermark estimate and allowed lateness with a trigger controls how long the pipeline waits for late data. The default trigger fires at the end of the window, and with allowed lateness, late events are still processed until the allowed time expires.

Answer

Use a sliding window of 1 minute with a 1-minute period

Answer

Use a global window with a trigger that fires every 10 seconds

Answer

Increase the watermark estimate to 10 minutes

Question 7

You need to process a large Spark ML training job on a Dataproc cluster. The job is fault-tolerant and can handle occasional node failures. To reduce costs, which type of worker nodes should you use?

Accepted Answer

Preemptible worker nodes. Preemptible VMs are significantly cheaper but can be terminated at any time. Since the job is fault-tolerant, preemptible workers can be used for cost savings.

Answer

Standard worker nodes

Answer

High-memory worker nodes

Answer

Sole-tenant nodes

Question 8

Your company uses Pub/Sub to ingest clickstream data. Messages must be processed in order for the same user_id. How should you configure the Pub/Sub subscription to guarantee ordering?

Accepted Answer

Use a pull subscription with enable_message_ordering=true. Pub/Sub ordering keys allow messages with the same key to be delivered in order to subscribers. The subscription must be created with enable_message_ordering set to true.

Answer

Use a pull subscription with exactly-once delivery enabled

Answer

Use a push subscription with acknowledgement deadline set to 600 seconds

Answer

Use a push subscription with a dead letter topic

Question 9

A Dataflow pipeline with multiple steps uses a side input from a slowly changing reference table stored in BigQuery. The side input is updated every hour. To avoid reprocessing the entire pipeline on each update, which approach should you use?

Accepted Answer

Use a side input with a custom 'AsIterable' and a 'Repeatable' trigger that refreshes the side input every hour. The side input should be read periodically using a side input pattern with a Repeatable trigger. This refreshes the side input without restarting the pipeline.

Answer

Use a side input with a periodic refresh via a DoFn that reads BigQuery on each element

Answer

Use the side input with a default trigger and 'withAllowedLateness'

Answer

Use a global window with a trigger that fires every hour

Question 10

You need to transform and clean messy CSV data using a visual interface without writing code. The transformation should be scheduled to run weekly. Which Google Cloud service should you use?

Accepted Answer

Cloud Dataprep. Cloud Dataprep (Trifacta) provides a visual interface for data wrangling, allowing users to create recipes and schedule jobs.

Answer

Cloud Dataflow

Answer

Dataproc

Answer

Cloud Data Fusion

Question 11

Your team wants to share a BigQuery dataset with another project while ensuring that users from that project can only query specific tables. Which BigQuery feature should you use?

Accepted Answer

Create an authorised view in your dataset and share the view with the other project. Authorised views allow you to share query results with users in other projects without giving them direct access to the underlying tables.

Answer

Use a materialised view and share the underlying table

Answer

Grant the BigQuery Data Viewer role to the other project's service account

Answer

Export the table to Cloud Storage and share the bucket

Question 12

A company uses Dataproc Serverless for Spark batch jobs. They notice that some jobs are failing due to out-of-memory (OOM) errors. Which configuration parameter should they adjust to allocate more memory per executor?

Accepted Answer

Set spark.executor.memory to a higher value, e.g., 8g. In Dataproc Serverless, Spark properties can be set via --properties. The spark.executor.memory property controls the memory per executor. Increasing it can resolve OOM errors.

Answer

Use a custom image with more memory

Answer

Set spark.driver.memory to a higher value

Answer

Increase the number of workers by setting --num-workers

Question 13

You are building a real-time fraud detection system using Dataflow. Events from Pub/Sub need to be grouped by user_id within a 5-minute window to detect suspicious patterns. Some events may be delayed by up to 2 minutes. How should you configure the window and trigger to balance accuracy and latency?

Accepted Answer

Fixed window of 5 minutes with allowed lateness of 2 minutes and early trigger every 1 minute. A fixed 5-minute window with allowed lateness of 2 minutes and a trigger that fires early every minute provides early results and captures late data within the allowed window.

Answer

Sliding window of 5 minutes with a 1-minute period and no allowed lateness

Answer

Session window with a gap duration of 5 minutes

Answer

Fixed window of 5 minutes with no allowed lateness and default trigger

Question 14

You need to choose a messaging service for a real-time streaming application that requires low cost and can tolerate occasional message loss. Which service is MOST suitable?

Accepted Answer

Pub/Sub Lite. Pub/Sub Lite offers a lower-cost option compared to Pub/Sub, with reduced reliability (e.g., at-least-once delivery but not exactly-once). It is designed for cost-sensitive streaming workloads where occasional loss is acceptable.

Answer

Cloud Scheduler

Answer

Pub/Sub

Answer

Cloud Tasks

Question 15

A data engineer needs to run an existing Spark job on Google Cloud with minimal code changes. The job requires Hive metastore access. Which Dataproc feature should they use to provide a managed Hive metastore?

Accepted Answer

Dataproc Metastore. Dataproc Metastore provides a fully managed Hive metastore that integrates with Dataproc clusters, allowing existing Spark jobs to use it without code changes.

Answer

Cloud SQL for MySQL

Answer

BigQuery as a Hive metastore

Answer

Dataproc on GKE