PDE Designing data processing systems Practice Test 5 — 15 Questions

Question 1

A media company uses Cloud Data Loss Prevention (DLP) API to inspect and de-identify sensitive data before loading into BigQuery. They want to reduce costs by sampling the data during inspection. Which configuration should they use?

Accepted Answer

Set the sample method to 'RANDOM' with a percentage.. Option B is correct because the Cloud DLP API supports a 'sample_method' of 'RANDOM' with a 'sampling_percentage' to inspect only a random subset of rows. This directly reduces the volume of data scanned, lowering costs while still providing statistically representative coverage for sensitive data discovery.

Answer

Use the 'ROWS' limit in the inspection job.

Answer

Use a hybrid inspection with a BigQuery sample table.

Answer

Use the 'BYTES_LIMIT' parameter.

Question 2

A company runs a Dataflow streaming pipeline that reads from Pub/Sub and writes to BigQuery. They experience a sudden spike in data volume causing BigQuery write throughput to be exceeded, resulting in errors. Which strategy should they implement to handle this gracefully?

Accepted Answer

Use a BigQuery sink with 'FAIL_FAST' error handling and set a dead-letter queue for failed writes.. Using a BigQuery sink with 'FAIL_FAST' error handling and a dead-letter queue (D) allows the pipeline to route failed writes to a separate Pub/Sub topic for later retry, preventing data loss and backpressure. Option A and B change write mode but don't handle errors. Option C prevents table creation but doesn't address throughput.

Answer

Use a BigQuery sink with 'WRITE_APPEND' mode and set 'writeDisposition' to 'WRITE_APPEND'.

Answer

Use a BigQuery sink with 'WRITE_TRUNCATE' mode.

Answer

Use a BigQuery sink with 'CREATE_NEVER' write method.

Question 3

A company is designing a data processing pipeline for real-time sensor data. They want to ensure low latency and exactly-once processing semantics. Which two Google services should they combine to achieve this? (Choose 2)

Accepted Answer

Cloud Pub/Sub with exactly-once delivery. Cloud Pub/Sub with exactly-once delivery (Option C) ensures that each message is delivered to subscribers exactly once, preventing duplicates in the pipeline. Cloud Dataflow with exactly-once processing mode (Option D) provides end-to-end exactly-once semantics by leveraging consistent snapshots and idempotent sinks, which is critical for real-time sensor data pipelines requiring low latency and accuracy.

Answer

Cloud Dataproc with Spark Streaming

Answer

Cloud Functions with Cloud Pub/Sub triggers

Answer

Cloud IoT Core with device gateways

Question 4

A data warehouse team uses Cloud BigQuery for analytics. They want to optimize query performance and reduce costs. Which three actions should they take? (Choose 3)

Accepted Answer

Use partitioned tables on time columns. Option A is correct because partitioning tables on time columns (e.g., DATE, TIMESTAMP) in BigQuery allows the query engine to perform partition pruning, scanning only the relevant partitions instead of the entire table. This directly reduces the amount of data read, lowering query costs and improving performance by limiting I/O to the necessary time range.

Answer

Use automatic reclustering

Answer

Use BI Engine for all queries

Question 5

A company uses Cloud Dataproc for large-scale Spark jobs. They notice that some jobs are failing due to insufficient memory on the worker nodes. They want to improve memory management without over-provisioning. Which three configurations should they apply? (Choose 3)

Accepted Answer

Set spark.executor.memory to a value that fits within the node memory. Option A is correct because setting spark.executor.memory to a value that fits within the node memory ensures that each executor does not exceed the available RAM on a worker node, preventing out-of-memory (OOM) errors. This configuration directly controls the heap size allocated to each executor, and when combined with spark.executor.cores and spark.executor.instances, it allows precise memory budgeting per node. Over-provisioning is avoided by calculating the maximum safe executor memory as (node memory - OS overhead - HDFS cache) / number of executors per node.

Answer

Use local SSDs for temporary storage

Answer

Use preemptible worker nodes for volatile tasks

Question 6

Given the query plan, what is the most likely reason this query is efficient despite processing 10 billion rows?

Accepted Answer

The table is partitioned by sale_date.. Option B is correct because partitioning by sale_date enables partition pruning, which allows the query engine to scan only the relevant partitions instead of the entire 10-billion-row table. This drastically reduces the amount of data read and processed, making the query efficient even with a large total row count.

Answer

The query uses a wildcard function.

Answer

The table is materialized.

Answer

The table is clustered by product_id.

Question 7

What is the most likely cause of data duplication after this command?

Accepted Answer

The BigQuery sink is not idempotent.. Option D is correct because BigQuery sinks in Dataflow are not idempotent by default; if the pipeline retries writes (e.g., due to worker failures or checkpoint issues), duplicate rows can be inserted into the BigQuery table. This is a known limitation: BigQuery does not support deduplication at the sink level unless you implement custom deduplication logic or use a staging table with merge operations. The command likely triggered a retry scenario, and the non-idempotent sink caused the duplication.

Answer

The Pub/Sub source is not exactly-once.

Answer

The pipeline uses at-least-once semantics.

Answer

The snapshot was taken before scaling.

Question 8

What is the root cause of this error and the correct solution?

Accepted Answer

The Dataflow worker service account needs the BigQuery Data Viewer role.. Option D is correct because Dataflow workers execute under a specific service account (compute engine default or custom), and that service account must have the BigQuery Data Viewer role to read data from BigQuery tables. Without this permission, the workers cannot access the source data, causing the job to fail with access errors. The BigQuery User role is insufficient for reading table data, and the BigQuery Admin role is overly permissive and not required for this task.

Answer

The BigQuery table requires authorized view access.

Answer

The user running the job needs the BigQuery Admin role.

Answer

The Dataflow service account needs the BigQuery User role.

Question 9

A company is designing a streaming data pipeline to process real-time clickstream events. They need to aggregate events by session window with a 5-minute gap and enable exactly-once processing semantics. Which Google Cloud service should they use?

Accepted Answer

Cloud Dataflow with Apache Beam. Cloud Dataflow with Apache Beam is the correct choice because it provides native support for session windows with a 5-minute gap duration and exactly-once processing semantics via its sink and source integrations. Dataflow's Beam SDK allows you to define session windows using `Window.into(Sessions.withGapDuration(Duration.standardMinutes(5)))`, and its checkpointing and idempotent writes ensure exactly-once delivery even in failure scenarios.

Answer

Cloud Pub/Sub with Cloud Functions

Answer

Cloud Dataproc with Spark Streaming

Answer

Cloud Bigtable with Dataflow templates

Question 10

A data engineer is designing a batch data pipeline that reads Avro files from Cloud Storage, transforms data using Apache Beam, and writes to BigQuery. The pipeline must handle daily runs and backfills. Which runner should they use?

Accepted Answer

DataflowRunner. DataflowRunner is the correct choice because it is the fully managed service runner for Apache Beam on Google Cloud, optimized for batch and streaming pipelines. It automatically handles scaling, resource management, and exactly-once processing semantics, which are essential for reliable daily runs and backfills with Avro files from Cloud Storage and BigQuery sinks.

Answer

FlinkRunner

Answer

SparkRunner

Answer

DirectRunner

Question 11

A company processes IoT sensor data in near real-time. They ingest data via Cloud Pub/Sub, then a Dataflow streaming pipeline writes to Bigtable for low-latency queries. Recently, they observed increased Pub/Sub message backlog during traffic spikes. What is the most effective scaling strategy?

Accepted Answer

Increase Dataflow worker count and adjust autoscaling configuration. The correct answer is B because the increased Pub/Sub backlog during traffic spikes indicates that the Dataflow pipeline is unable to consume messages as fast as they are being published. Increasing the Dataflow worker count and adjusting autoscaling configuration allows the pipeline to scale horizontally, processing more messages per second and reducing the backlog. Pub/Sub itself is designed to handle high throughput, so the bottleneck is the consumer (Dataflow), not the ingestion layer.

Answer

Increase Pub/Sub subscription throughput by increasing the number of partitions

Answer

Use a Cloud Scheduler to throttle Pub/Sub publishing

Answer

Add a Cloud Function to pre-process messages before they are consumed by Dataflow

Question 12

A team needs to migrate an existing on-premises Hadoop Hive workload to Google Cloud. They want to minimize code changes and use a managed service for transient clusters. Which service should they choose?

Accepted Answer

Cloud Dataproc. Cloud Dataproc is the correct choice because it is a managed Spark and Hadoop service that supports Hive workloads natively, allowing you to run existing Hive scripts with minimal changes. It also supports transient clusters, which can be automatically scaled up and down, aligning with the requirement for transient clusters.

Answer

Cloud Dataflow

Answer

Cloud Dataprep

Answer

BigQuery

Question 13

A financial company needs to process batch trades data daily and ensure that if a transformation step fails, the entire daily run is retried from the beginning. Which design pattern is appropriate?

Accepted Answer

Use an orchestrator like Cloud Composer with retry logic. Option B is correct because the requirement states that if any transformation step fails, the entire daily run must be retried from the beginning. An orchestrator like Cloud Composer (Apache Airflow) provides native DAG-level retry logic that can be configured to restart the entire workflow on failure, ensuring atomicity of the batch run. This pattern is essential for maintaining data consistency when partial processing cannot be tolerated.

Answer

Use idempotent writes with checkpointing

Answer

Retry the failed step only

Answer

Use a transactional staging area

Question 14

A data pipeline uses Cloud Pub/Sub to ingest events, then a Dataflow job writes to Cloud Storage in Avro format. The Dataflow job uses Global windows with a 10-minute trigger. The data is later loaded into BigQuery. They notice duplicate rows in BigQuery because the trigger produced multiple panes. What should the Dataflow pipeline change to eliminate duplicates?

Accepted Answer

Enable exactly-once sink to BigQuery via Dataflow. Option A is correct because enabling exactly-once sinks in Dataflow ensures that each record is written to the sink only once, even if the pipeline produces multiple panes due to triggers. In this scenario, the 10-minute trigger on a global window causes multiple output panes, leading to duplicate rows in BigQuery. Exactly-once sinks use idempotent writes and deduplication mechanisms to prevent duplicates, directly addressing the issue without changing the windowing or trigger logic.

Answer

Use a sharded output to Cloud Storage with unique filenames

Answer

Write to a staging table and use a MERGE statement in BigQuery

Answer

Use a session window instead of global window

Question 15

A company wants to analyze server logs stored in Cloud Storage using SQL. They need to get results in seconds without setting up any clusters. Which service should they use?

Accepted Answer

BigQuery. BigQuery is a serverless, highly scalable, and cost-effective multi-cloud data warehouse designed for business agility. It allows you to analyze petabytes of data using standard SQL without needing to provision or manage any clusters, making it ideal for querying server logs stored in Cloud Storage directly via external tables or loading data into BigQuery for sub-second query performance.

Answer

Cloud Dataflow

Answer

Cloud Logging

Answer

Cloud Dataproc