PDE Designing Data Processing Systems Practice Test 3 — 15 Questions

Question 1

You need to transform and clean messy CSV data using a visual interface without writing code. The transformation should be scheduled to run weekly. Which Google Cloud service should you use?

Accepted Answer

Cloud Dataprep. Cloud Dataprep is a fully managed, visual data preparation service that lets users clean and transform data through a point-and-click interface without writing code. It supports scheduling jobs, including weekly runs, making it the right choice for this requirement. It is built on Trifacta and integrates with BigQuery and Cloud Storage.

Answer

Cloud Dataflow

Answer

Dataproc

Answer

Cloud Data Fusion

Question 2

Your team wants to share a BigQuery dataset with another project while ensuring that users from that project can only query specific tables. Which BigQuery feature should you use?

Accepted Answer

Create an authorised view in your dataset and share the view with the other project. Authorised views allow you to share query results with users in other projects without giving them direct access to the underlying tables.

Answer

Use a materialised view and share the underlying table

Answer

Grant the BigQuery Data Viewer role to the other project's service account

Answer

Export the table to Cloud Storage and share the bucket

Question 3

You are designing a data pipeline for a financial services company that requires exactly-once processing semantics. Which TWO services or configurations provide exactly-once guarantees?

Accepted Answer

Pub/Sub with exactly-once delivery enabled. Pub/Sub with exactly-once delivery enabled provides exactly-once message delivery. Dataflow with exactly-once processing mode ensures each record is processed exactly once. Pub/Sub Lite and Dataproc do not provide exactly-once guarantees, and Cloud Storage is for storage.

Answer

Dataproc with checkpointing

Answer

Pub/Sub Lite with at-least-once delivery

Answer

Cloud Storage with object versioning

Question 4

A data pipeline is built with Cloud Dataflow that reads from Pub/Sub, applies transformations, and writes to BigQuery. The pipeline is experiencing high latency and occasional data loss during worker failures. The engineer wants to improve reliability and performance. Which two actions should they take?

Accepted Answer

Enable Dataflow Streaming Engine and use BigQuery exactly-once sink. Enabling streaming engine moves the state management to the backend, reducing latency and improving reliability. Using exactly-once sinks (like BigQuery with exactly-once guarantees) prevents data loss.

Answer

Switch to Cloud Dataproc and use Spark Structured Streaming

Answer

Increase the number of workers and use at-least-once delivery to BigQuery

Answer

Use a global window and disable triggers

Question 5

A company has a BigQuery table that is partitioned by ingestion time and clustered by the 'customer_id' column. They notice that queries filtering on 'customer_id' are not benefiting from clustering as expected. What is the most likely cause?

Accepted Answer

The table is too small for clustering to be effective. BigQuery only applies cluster pruning when the table has enough data blocks to make the clustering metadata meaningful. For small tables (typically under ~1 GB or with few rows per block), the query optimizer simply scans all data because the overhead of evaluating cluster metadata outweighs any savings, so filtering on customer_id appears to give no benefit. This is expected behavior, not a misconfiguration.

Answer

The query is using a wildcard function that prevents clustering pruning

Answer

The clustering column must be the same as the partition column

Answer

Clustering does not work with ingestion-time partitioning

Question 6

A data pipeline using Cloud Dataflow reads from a Pub/Sub subscription that has a dead letter topic configured. Some messages are being sent to the dead letter topic. Upon investigation, the engineer finds that the messages contain valid data but are malformed according to the schema. What is the most likely reason for the messages being dead-lettered?

Accepted Answer

The Pub/Sub topic has a schema that the messages do not comply with. Pub/Sub dead letter topics capture messages that cannot be successfully processed after a configured number of delivery attempts. If messages contain valid data but are malformed against a schema, the most likely cause is that the Pub/Sub topic enforces a schema and the publisher's messages fail validation, so they are routed to the dead letter topic. Schema validation failures are a common dead-letter trigger.

Answer

The Pub/Sub topic is not configured with a schema

Answer

The Dataflow pipeline is using at-least-once delivery guarantee

Answer

The subscription's ack deadline is too short

Question 7

A company is migrating their on-premises Hadoop workloads to Google Cloud. They want to use Dataproc for data processing and need to minimize costs for non-critical batch jobs that can tolerate interruptions. Which TWO configurations should they use?

Accepted Answer

Use preemptible instances for worker nodes. Option A is correct because preemptible (spot) VMs on Dataproc worker nodes cost significantly less than standard VMs and are designed for fault-tolerant, non-critical batch workloads that can tolerate interruptions, which matches the requirement to minimize costs. Option D is correct because single-node Dataproc clusters run all roles (master and worker) on one VM, eliminating the cost of additional master/worker nodes for small jobs, making them a cost-efficient choice for lightweight batch processing. Option B is not appropriate because high-availability mode adds two additional master nodes (three masters total), increasing cost and providing no benefit for interruptible batch jobs. Option C is incorrect because standard non-preemptible instances for all nodes is the most expensive configuration and contradicts the cost-minimization goal. Option E is incorrect because Dataproc on GKE adds Kubernetes cluster overhead and is not the simplest or cheapest option for this scenario.

Answer

Enable high-availability mode

Answer

Use standard (non-preemptible) instances for all nodes

Answer

Use Dataproc on GKE

Question 8

A data engineering team is designing a streaming pipeline using Cloud Dataflow. They need to join two unbounded PCollections based on a common key. The join must handle late data up to 10 minutes. Which THREE components should they use?

Accepted Answer

CoGroupByKey transform. Option A (CoGroupByKey transform) is correct because it is the standard Apache Beam/Dataflow transform for joining multiple PCollections that share a common key, grouping all values per key so the two unbounded streams can be correlated. Option B (Window into fixed windows) is correct because joining unbounded PCollections requires windowing to bound the join scope; fixed windows align both streams into discrete time intervals so matching keys land in the same window. Option E (Set allowed lateness and trigger to handle late data) is correct because allowed lateness defines how long a window accepts late-arriving elements (here 10 minutes) and triggers determine when results are emitted, which is exactly what is needed to handle late data. Option C is not correct because a global window with triggers does not bound the join by time and is not the appropriate windowing strategy for a time-bounded join with late data. Option D is not correct because Flatten merely merges PCollections of the same type without performing any key-based join.

Answer

Use a global window with triggers

Answer

Flatten transform

Question 9

A Dataflow pipeline using Apache Beam processes unbounded data from Pub/Sub. The pipeline uses fixed windows of 1 minute and a trigger that fires early every 30 seconds and at watermark. The team observes that the output pane for window [10:00:00, 10:01:00) contains events with timestamps from 10:00:15 and 10:00:45, but also an event with timestamp 10:02:00. What is the most likely cause?

Accepted Answer

Allowed lateness is set to more than 1 minute, so late data is still included in its original window. The event with timestamp 10:02:00 is late data relative to the window [10:00:00, 10:01:00). In Apache Beam, allowed lateness determines how long after the watermark passes the end of a window that late data can still be accepted and included in that window's pane. If allowed lateness is set to more than 1 minute (e.g., 2 minutes), then an event with timestamp 10:02:00 can still be added to the window [10:00:00, 10:01:00) because the watermark for that window has not yet advanced beyond the allowed lateness period. Thus, the presence of that event is expected behavior when allowed lateness exceeds the window duration.

Answer

The trigger is firing too early, causing the window to close prematurely

Answer

The watermark is incorrectly estimated, allowing late data to be included

Answer

The window duration is actually 2 minutes due to a misconfiguration

Question 10

A company wants to use Pub/Sub Lite to reduce costs for a high-throughput, low-latency streaming pipeline. However, they have a requirement to retain messages for up to 7 days for reprocessing. Which Pub/Sub Lite configuration supports this retention?

Accepted Answer

Set the retention duration on the Pub/Sub Lite topic to 7 days. Pub/Sub Lite topics are the only entity where retention duration is configured; messages are retained in the topic's storage for the specified duration, allowing subscribers to replay messages within that window. Setting the retention duration to 7 days on the topic ensures messages are available for reprocessing for up to 7 days, meeting the requirement.

Answer

Set the retention duration on the Pub/Sub Lite subscription to 7 days

Answer

Enable exactly-once delivery on the Pub/Sub Lite topic to retain messages for 7 days

Answer

Use a Pub/Sub Lite reservation with 7-day retention

Question 11

A data engineer needs to create a BigQuery table that is optimized for queries that filter on a 'customer_id' column and sort by 'transaction_date'. The table will be used for interactive analysis. Which combination of table features should be used?

Accepted Answer

Partition by transaction_date and cluster by customer_id. BigQuery best practices dictate that a table should be partitioned on a high-level, frequently filtered column (like transaction_date) to reduce the amount of data scanned, and then clustered on columns used for filtering and sorting (like customer_id) to further optimize query performance. Partitioning by transaction_date allows pruning of partitions based on date filters, while clustering by customer_id sorts data within each partition, making filters on customer_id and sorts by transaction_date more efficient. This combination is ideal for interactive analysis where queries often filter on customer_id and sort by transaction_date.

Answer

Partition by customer_id and cluster by transaction_date

Answer

Cluster by both customer_id and transaction_date

Answer

Use a materialized view with customer_id and transaction_date

Question 12

A data team is migrating an on-premises Hadoop cluster to Dataproc. The cluster runs a mix of long-running services (Hive, HBase) and transient Spark jobs. They want to minimize cost while maintaining performance. Which TWO strategies should they implement?

Accepted Answer

Use preemptible instances for worker nodes in the transient Spark cluster. Option E is correct because separating long-running services like Hive and HBase into a dedicated cluster with standard (non-preemptible) instances keeps those persistent services stable and available, while allowing the transient Spark cluster to be sized and torn down independently, which minimizes cost without sacrificing performance for the always-on services. Option C is correct because preemptible (Spot) VMs cost significantly less than standard instances, and transient Spark jobs that can tolerate interruptions are ideal candidates for preemptible worker nodes, reducing cost for the batch workload. Option A is incorrect because consolidating everything into one HA cluster forces long-running and transient workloads to share resources, preventing the transient cluster from scaling down to zero and increasing cost. Option B is incorrect because attaching local SSDs to all nodes raises cost and local SSD data is ephemeral, so it is not a cost-minimizing strategy for the whole cluster. Option D is incorrect because Dataproc on GKE is aimed at containerized, often ephemeral workloads and does not by itself provide the cost-optimal split between persistent services and transient jobs described here.

Answer

Consolidate all workloads into a single high-availability cluster

Answer

Use local SSDs for all nodes to improve I/O performance

Answer

Use Dataproc on GKE to run long-running services

Question 13

A data engineer is designing a streaming pipeline using Dataflow with Apache Beam. The pipeline reads from Pub/Sub, performs a stateful transformation (e.g., session windowing), and writes to BigQuery. The pipeline must handle late data and ensure exactly-once semantics. Which THREE configurations are required?

Accepted Answer

Set allowed lateness on the window to accommodate late data. Option B is correct because setting allowed lateness on the window (e.g., with withAllowedLateness) tells Apache Beam how long to retain window state and keep accepting late-arriving elements after the watermark passes, which is essential for handling late data in session windows. Option C is correct because triggers (e.g., AfterWatermark with early/late firings) control when window results are emitted, so late data that arrives within the allowed lateness can still produce updated outputs. Option E is correct because enabling exactly-once processing on the Dataflow pipeline (via the streaming engine's exactly-once mode) ensures no duplicate records are written to BigQuery and that state is consistent across retries. Option A is not required: the File Loads method is a BigQuery write method choice, but exactly-once and late-data handling are governed by windowing, triggers, and pipeline semantics, not by the load method. Option D is not required: Pub/Sub's default watermark estimation is generally sufficient, and a custom watermark function is not a mandatory configuration for late data or exactly-once semantics.

Answer

Use the File Loads write method for BigQuery

Answer

Use a custom watermark estimation function for Pub/Sub source

Question 14

A company uses Cloud Data Fusion for ETL pipelines. They need to transform sensitive data (PII) by masking certain columns before writing to BigQuery. They also need to ensure the pipeline can be monitored and restarted from failure points. Which THREE features should they use?

Accepted Answer

Enable pipeline monitoring with alerts in Cloud Data Fusion. Option C is correct because Cloud Data Fusion provides built-in pipeline monitoring and alerting, allowing the team to track pipeline execution status and receive notifications on failures or performance issues. Option D is correct because Cloud Data Fusion supports pipeline checkpointing, which saves the pipeline's state at intervals so the pipeline can be resumed from the last successful checkpoint rather than restarting from the beginning. Option E is correct because Wrangler is Cloud Data Fusion's interactive data preparation tool that includes masking directives (such as 'mask' or 'hash' transformations) to obfuscate PII columns before writing to BigQuery. Option A is not correct because Cloud Composer is a separate orchestration service and is not required to schedule or retry a Cloud Data Fusion pipeline, which has its own scheduling and retry capabilities. Option B is not correct because Dataproc Metastore is a metadata management service for Hive/Spark workloads and does not provide PII masking or pipeline restart functionality for Cloud Data Fusion.

Answer

Use Cloud Composer to schedule and retry the pipeline

Answer

Create a Dataproc Metastore service to store pipeline metadata

Question 15

A company is designing a data pipeline that ingests real-time events from IoT devices and must handle late-arriving data (up to 1 hour late) while minimizing duplicate processing. They plan to use Dataflow with Pub/Sub. Which combination of windowing and trigger settings should they use?

Accepted Answer

Session windows with a 10-minute gap duration and allowed lateness of 1 hour. Session windows naturally group events based on a gap duration, so late events within the gap extend the window. Setting the allowed lateness to 1 hour ensures that late events are still included in the correct session. Using withAllowedLateness(1 hour) allows the watermark to advance and the session to finalize after the gap, but late data within 1 hour will trigger a pane update.

Answer

Sliding windows of 10 minutes with allowed lateness of 30 minutes and accumulating panes

Answer

Global window with allowed lateness of 1 hour and accumulating trigger every 5 minutes

Answer

Fixed windows of 1 hour with allowed lateness of 1 hour and no accumulation