PDE Designing Data Processing Systems Practice Test 7 — 15 Questions

Question 1

Which Google Cloud service provides a visual interface for building ETL pipelines using a drag-and-drop design and includes pre-built transforms from a marketplace?

Accepted Answer

Cloud Data Fusion. Cloud Data Fusion is Google Cloud's fully managed, code-free ETL/ELT service built on the open-source CDAP project, offering a graphical drag-and-drop pipeline designer and a marketplace of pre-built plugins and transformations. It lets data engineers build batch and streaming pipelines visually and deploy them to ephemeral Dataproc clusters. This matches the requirement for a visual interface with marketplace transforms.

Answer

Dataproc

Answer

Dataprep

Answer

BigQuery

Question 2

You are designing a streaming pipeline that needs to handle sudden spikes in traffic without losing data. The pipeline uses Pub/Sub and Dataflow. Which configuration ensures data is not lost if Dataflow falls behind?

Accepted Answer

Use Pub/Sub with a pull subscription and set the message retention duration to 7 days. A Pub/Sub pull subscription with a 7-day message retention duration ensures that if Dataflow falls behind, messages are retained in the subscription backlog for up to 7 days, preventing data loss. Dataflow pulls messages and acknowledges them only after processing, so unacknowledged messages remain available for redelivery. This configuration decouples ingestion from processing and provides a buffer for spikes.

Answer

Use Cloud Pub/Sub Lite with a smaller retention period

Answer

Use Pub/Sub with a push subscription and increase the acknowledgment deadline

Answer

Use Pub/Sub with exactly-once delivery and Dataflow with at-least-once processing

Question 3

You need to analyse streaming data from thousands of IoT devices, each sending temperature readings every second. You want to calculate the average temperature per device over the last 5 minutes, updating every minute. Which windowing strategy should you use in Dataflow?

Accepted Answer

Sliding windows of length 5 minutes with a period of 1 minute. Sliding windows of length 5 minutes with a period of 1 minute produce a new window every minute, each covering the previous 5 minutes of data — exactly matching the requirement to compute a 5-minute average that updates every minute. This is the canonical use case for sliding windows in Dataflow/Beam.

Answer

Global windows with a trigger firing every minute

Answer

Fixed windows of 5 minutes

Answer

Session windows with a gap duration of 1 minute

Question 4

You are designing a Dataflow pipeline that joins two unbounded PCollections from different sources. Which transform should you use?

Accepted Answer

CoGroupByKey. CoGroupByKey performs a key-based join of multiple PCollections. It can handle unbounded streams with appropriate windowing.

Answer

ParDo

Answer

Flatten

Answer

GroupByKey

Question 5

You are designing a Dataflow pipeline for processing real-time clickstream data. The pipeline must group events into 30-second windows and handle late data up to 5 minutes. You want to output partial results every 10 seconds for low-latency monitoring. Which THREE configurations should you use? (Choose three.)

Accepted Answer

Use fixed windows of 30 seconds. Option C is correct because fixed (tumbling) 30-second windows partition the clickstream into non-overlapping 30-second intervals, which is the required grouping for this scenario. Option D is correct because setting allowed lateness to 5 minutes lets the pipeline retain window state and accept events that arrive up to 5 minutes after the window closes, matching the late-data requirement. Option E is correct because a trigger with early firings every 10 seconds emits speculative partial results before the window closes, providing the requested low-latency monitoring output. Option A is not appropriate because sliding windows with a 10-second period would create overlapping 30-second windows and duplicate events across windows, which is not the specified grouping. Option B is not appropriate because a trigger that fires only after the end of the window would produce results only at window closure and would not deliver the required 10-second partial outputs.

Answer

Use sliding windows of 30 seconds with a 10-second period

Answer

Use a trigger that fires after the end of the window

Question 6

Your data engineering team needs to process a continuous stream of clickstream events from a website and update a real-time dashboard showing user activity over the last hour. The pipeline should have minimal operational overhead and support exactly-once processing semantics. Which Google Cloud service should you use?

Accepted Answer

Cloud Dataflow with Apache Beam. Cloud Dataflow with Apache Beam is Google's fully managed, serverless stream and batch processing service, and Beam provides built-in exactly-once semantics for streaming pipelines. It integrates natively with Pub/Sub for ingestion and supports windowing, triggers, and state for real-time aggregations like a rolling 1-hour user activity view. Because Dataflow is fully managed, it meets the 'minimal operational overhead' requirement without cluster management.

Answer

Cloud Dataproc with Apache Spark Streaming

Answer

Cloud Data Fusion with batch pipelines

Answer

Cloud Pub/Sub Lite with push subscriptions

Question 7

Your company ingests millions of events per second into a Pub/Sub topic. The downstream consumer must process events with minimal latency and high throughput. However, the consumer occasionally falls behind during traffic spikes, and you need to ensure no data loss while minimizing costs. Which subscription type and configuration should you choose?

Accepted Answer

Pull subscription with flow control settings. A pull subscription with flow control settings allows the consumer to control the rate of message delivery, preventing overload during traffic spikes. Pull subscriptions are ideal for high-throughput, low-latency processing because the consumer can batch and process messages at its own pace. Flow control settings (e.g., max outstanding messages, max bytes) help avoid overwhelming the consumer, ensuring no data loss while minimizing costs.

Answer

Push subscription with a load balancer

Answer

Push subscription with endpoint on Cloud Run

Answer

Pull subscription with exactly-once delivery disabled

Question 8

You are designing a data pipeline that processes streaming events with late-arriving data (up to 2 hours late). The pipeline must compute hourly aggregations and emit results as soon as possible, but must also accurately update results when late data arrives. You want to minimize overall processing cost. Which Dataflow windowing and trigger configuration should you use?

Accepted Answer

Fixed windows of 1 hour with allowed lateness of 2 hours and trigger every 5 minutes (early) and on watermark (late) with accumulating fired panes. Fixed windows of 1 hour with allowed lateness of 2 hours and triggers every 5 minutes (early) and on watermark (late) with accumulating fired panes is the correct configuration. This setup computes hourly aggregations, emits early results every 5 minutes, and updates results when late data arrives within the 2-hour allowed lateness. Accumulating panes ensure that late data updates the previous results. This minimizes cost by using fixed windows and appropriate triggers.

Answer

Global window with triggers every 5 minutes

Answer

Sliding windows of 1 hour with 30-minute offset

Answer

Session windows with 10-minute gap duration

Question 9

You are migrating on-premises Hadoop jobs to Google Cloud. The existing jobs use Spark for ETL and Hive for querying. You want to minimize changes to the existing code and maintain the ability to use Hive queries with the same metastore across multiple clusters. Which service combination should you use?

Accepted Answer

Cloud Dataproc with Cloud Storage and Dataproc Metastore. Cloud Dataproc runs managed Spark and Hive clusters, so existing Spark ETL jobs and Hive queries migrate with minimal code changes. Pairing Dataproc with Cloud Storage for data and Dataproc Metastore for a shared Hive metastore across clusters preserves the same table definitions and schema across multiple clusters.

Answer

Cloud Dataflow with Beam SQL

Answer

Cloud Dataproc with Dataproc on GKE

Answer

Cloud BigQuery with external tables on Cloud Storage

Question 10

You have a BigQuery table that is partitioned by ingestion time and clustered on user_id. The table stores event logs and is queried frequently by user_id to analyze user behavior over the last 30 days. Queries are still scanning too many partitions. Which optimization should you apply first?

Accepted Answer

Change the partition column to a DATE column based on event_timestamp and keep clustering on user_id. Ingestion-time partitioning creates partitions based on when data arrives, not the event timestamp, so a query filtering on the last 30 days of events may still scan partitions that contain older events ingested recently. Changing the partition column to a DATE derived from event_timestamp aligns partition pruning with the query filter, and keeping clustering on user_id preserves the benefit for user-based filtering. This is the highest-impact first optimization.

Answer

Create a materialized view that pre-aggregates data by user_id and date

Answer

Remove partitioning and rely solely on clustering

Answer

Add clustering on a second column like event_type

Question 11

You need to allow a data analyst to run queries on a BigQuery dataset but prevent them from modifying the data or deleting the dataset. Which IAM role should you grant?

Accepted Answer

roles/bigquery.dataViewer. roles/bigquery.dataViewer grants read-only access to BigQuery datasets, tables, and views, allowing the analyst to run queries and view metadata but not modify data or delete the dataset. It is the least-privilege role that satisfies the requirement of querying without write or delete permissions. This role is typically paired with roles/bigquery.jobUser at the project level so the user can actually run jobs.

Answer

roles/bigquery.dataOwner

Answer

roles/bigquery.jobUser

Answer

roles/bigquery.dataEditor

Question 12

You need to create a BigQuery table that stores customer transaction data. The table will be queried frequently by a customer_id column to retrieve recent transactions (last 30 days). Which table design optimizes query performance and cost?

Accepted Answer

Partition by transaction_date and cluster by customer_id. Partitioning by transaction_date aligns with the query filter (last 30 days), so BigQuery prunes all partitions outside that window, scanning only the relevant date range. Clustering by customer_id then sorts data within each partition, so the customer_id filter benefits from block-level pruning. This combination minimizes both bytes scanned (cost) and query latency.

Answer

Partition by customer_id and cluster by transaction_date

Answer

Partition by ingestion_time and cluster by customer_id

Answer

Cluster by transaction_date and customer_id without partitioning

Question 13

Your company is building a real-time anomaly detection system for financial transactions. The system must process streams of transactions and flag anomalies within seconds. The volume is moderate (5000 transactions per second). You want a fully managed solution that integrates with BigQuery for historical analysis. Which service should you use for stream processing?

Accepted Answer

Cloud Dataflow. Cloud Dataflow is a fully managed, serverless stream and batch processing service based on Apache Beam. It natively supports real-time streaming with low latency (sub-second to seconds), integrates seamlessly with Pub/Sub for ingestion and BigQuery for output, and auto-scales to handle moderate throughput like 5000 transactions per second. Its managed nature eliminates cluster operations, making it ideal for this use case.

Answer

Cloud Pub/Sub with push subscriptions

Answer

Cloud Dataproc with Spark Streaming

Answer

Cloud Data Fusion

Question 14

Your company runs a Dataflow streaming pipeline that processes user activity from Pub/Sub and writes aggregated results to BigQuery. Lately, the pipeline is experiencing high latency and backlog growth during peak hours. You need to troubleshoot and improve performance. Which THREE actions should you take? (Choose 3.)

Accepted Answer

Change the worker machine type to a higher CPU/memory configuration. Option A is correct because moving to a higher CPU/memory worker machine type gives each worker more compute and memory to process elements, which directly addresses CPU or memory saturation that causes high latency and backlog growth during peak hours. Option C is correct because enabling Dataflow Streaming Engine moves pipeline state and windowing execution off the worker VMs to the Dataflow service, reducing worker CPU/memory pressure and improving streaming performance and autoscaling responsiveness. Option D is correct because increasing the number of workers adds horizontal processing capacity, allowing the pipeline to drain the Pub/Sub backlog faster and keep up with peak throughput. Option B is not appropriate because shortening the window duration does not reduce the total volume of data processed and can actually increase per-window overhead and output frequency, potentially worsening latency. Option E is not appropriate because adding more subscriptions to the same topic does not increase the throughput of the existing pipeline; each subscription would need its own reader/pipeline, and extra subscriptions only add fan-out consumers rather than improving the current pipeline's performance.

Answer

Decrease the window duration to reduce data per window

Answer

Add additional Pub/Sub subscriptions to the same topic

Question 15

Your team is using Cloud Dataprep to clean and transform a dataset. Which TWO features of Cloud Dataprep help you understand data quality issues before running the pipeline? (Choose 2.)

Accepted Answer

Column histograms. Option B, column histograms, is correct because Cloud Dataprep generates interactive histograms for each column that visually reveal value distributions, outliers, nulls, and anomalies, letting you spot data quality issues before executing a pipeline. Option E, data quality profiling, is correct because Cloud Dataprep's profiling feature automatically scans the dataset and reports metrics such as missing values, distinct counts, mismatched types, and invalid values, which is exactly the pre-pipeline assessment of data quality described. Option A, scheduling data quality jobs, is not correct because scheduling controls when transformation jobs run, not how you inspect data quality beforehand. Option C, joining datasets, is not correct because joins are transformation operations that combine data rather than diagnose quality issues. Option D, recipe steps, is not correct because recipe steps define the transformations to apply, not the profiling or histogram analysis used to understand data quality first.

Answer

Scheduling data quality jobs

Answer

Joining datasets

Answer

Recipe steps