PDE Designing data processing systems Practice Test 7 — 15 Questions

Question 1

A gaming company uses Avro schemas for its streaming event data. They anticipate adding new optional fields to events over time. They need to ensure backward compatibility so that existing pipelines continue to work. Which strategy should they adopt?

Accepted Answer

Use Avro with a schema registry that enforces backward-compatible changes. Option A is correct because Avro, combined with a schema registry, allows schema evolution with backward compatibility. The registry enforces rules such as adding optional fields with defaults, ensuring that consumers using older schemas can still deserialize new data without breaking. This directly addresses the requirement for existing pipelines to continue working as new optional fields are added.

Answer

Use JSON instead of Avro and ignore unknown fields

Answer

Use Protocol Buffers with breaking changes

Answer

Use FlatBuffers for performance

Question 2

A financial services company must comply with GDPR "right to be forgotten". They store customer transactions in BigQuery partitioned by date. When a user requests deletion, all their data must be removed within 48 hours. The deletion requests are received via a Pub/Sub topic. What is the most scalable and cost-effective approach?

Accepted Answer

Use a Dataflow pipeline that reads the deletion IDs from Pub/Sub, joins with the transactions table using a side input, and writes the filtered data to a new table, then swapping. Option C is correct because it uses Dataflow to process deletion requests from Pub/Sub, join them with the BigQuery transactions table via a side input, and write a filtered copy to a new table. This approach is scalable (handles high-throughput streaming deletions) and cost-effective (avoids expensive DELETE mutations on BigQuery, which consume slot resources and can be slow for large tables). Swapping the new table for the old one completes the deletion efficiently within the 48-hour SLA.

Answer

Use Cloud Functions to execute a BigQuery DELETE statement on each request

Answer

Use Cloud DLP to redact the user's data in Cloud Storage

Answer

Use BigQuery table snapshots and restore after deletion

Question 3

You are designing a streaming Dataflow pipeline that processes high-throughput data. Which two features can help minimize cost? (Choose TWO.)

Accepted Answer

Enable autoscaling based on CPU utilization. Option A is correct because enabling autoscaling based on CPU utilization allows the Dataflow pipeline to dynamically adjust the number of worker instances in response to the actual processing load. This prevents over-provisioning during low-throughput periods, directly reducing compute cost while maintaining performance during spikes.

Answer

Use batch loads to BigQuery for streaming inserts

Answer

Use preemptible VMs for all workers

Answer

Use a global window and batch output to BigQuery every hour

Question 4

A payment processing company needs to detect fraudulent transactions in real time. The system must have sub-second latency for high-value transactions and use a machine learning model. Which two components should be part of the architecture? (Choose TWO.)

Accepted Answer

Bigtable to store user profiles and transaction history for fast lookups. Bigtable is a fully managed, scalable NoSQL database that provides consistent sub-10ms latency for high-throughput read/write operations, making it ideal for real-time lookups of user profiles and transaction history in fraud detection. Its ability to handle large volumes of data with low latency supports the sub-second requirement for high-value transactions.

Answer

Cloud Storage for transaction logs

Answer

Cloud SQL to store reference data

Answer

Cloud Functions for long-running batch model training

Question 5

You are designing a streaming pipeline that must guarantee exactly-once processing. Which three services or features can help achieve this? (Choose THREE.)

Accepted Answer

Cloud Spanner for deduplication state across the pipeline. Cloud Spanner is correct because it provides globally distributed, strongly consistent transactions that can be used to maintain deduplication state across the entire streaming pipeline. By storing a unique key for each processed event in Spanner, the pipeline can atomically check and record whether an event has already been handled, ensuring exactly-once semantics even in the face of retries or failures.

Answer

Cloud Functions for post-processing

Answer

BigQuery streaming inserts with a unique key for deduplication

Question 6

What is the most likely cause of this error?

Accepted Answer

The schema of the BigQuery table expects a TIMESTAMP but the pipeline is sending a STRING. Option D is correct because the error message indicates a type mismatch: BigQuery expects a TIMESTAMP column, but the pipeline is sending a STRING. Dataflow's BigQuery sink performs automatic schema validation, and if the source data type (STRING) does not match the target column type (TIMESTAMP), the write operation fails with a mismatch error. This is a common issue when pipeline code or source data formats timestamps as strings without explicit conversion.

Answer

The BigQuery table is not partitioned

Answer

The Dataflow worker does not have the correct time zone

Answer

The pipeline is using a fixed window but the data is out of order

Question 7

The query above runs slowly on the 10 TB table. Which optimization would most improve performance?

Accepted Answer

Create a materialized view that pre-aggregates by customer_id and item category. Option C is correct because a materialized view can pre-compute and store the aggregated results by customer_id and item category, eliminating the need to scan the full 10 TB table for each query. This dramatically reduces I/O and computation time, especially when the underlying aggregation is expensive and the query pattern is predictable.

Answer

Use a subquery to filter item.category first

Answer

Cluster the table by customer_id

Answer

Partition the table by order_date

Question 8

The push endpoint is returning 500 errors. What is the most likely cause?

Accepted Answer

The push endpoint requires authentication but none is set. The push endpoint likely requires authentication, but none is configured, causing the 500 errors.

Answer

The topic has no messages

Answer

The push endpoint is not a valid HTTPS URL

Answer

The ack deadline is too short

Question 9

A company is designing a streaming pipeline using Dataflow to process real-time clickstream data. The pipeline reads from Pub/Sub, performs user sessionization using Apache Beam's Session window, and writes to BigQuery. The team notices that the pipeline's lag is growing and the worker utilization is low. What is the most likely cause and recommended fix?

Accepted Answer

The session window gap duration is too large, causing excessive state per key; reduce the gap duration.. D is correct because a large session window gap duration causes Dataflow to maintain excessive state per key (user session), leading to high memory pressure and slow processing. This results in growing pipeline lag despite low worker utilization, as workers spend more time managing state than processing data. Reducing the gap duration limits the state size and improves throughput.

Answer

Too many workers are created; reduce the number of workers.

Answer

The pipeline is not using autoscaling; enable autoscaling.

Answer

Insufficient disk space per worker; increase the boot disk size.

Question 10

A company wants to ingest IoT sensor data from thousands of devices into BigQuery for near-real-time analytics. The data volume is approximately 10 GB per hour. Which combination of Google Cloud services should they use for a cost-effective and scalable solution?

Accepted Answer

Pub/Sub → Dataflow → BigQuery. Pub/Sub provides a scalable, managed ingestion layer for high-volume IoT data, decoupling producers from consumers. Dataflow (Apache Beam) processes the streaming data in near-real-time with exactly-once semantics and auto-scaling, writing directly to BigQuery for analytics. This combination minimizes operational overhead and cost by avoiding intermediate storage and manual scaling.

Answer

Cloud IoT Core → Cloud Functions → BigQuery

Answer

Cloud IoT Core → Cloud Dataproc → BigQuery

Answer

Cloud IoT Core → Cloud Storage → BigQuery load jobs

Question 11

A Dataflow streaming pipeline uses stateful transformations with per-key state and timers. After a deployment, the team observes that the pipeline is reprocessing events from the last 30 minutes every time it restarts. The pipeline's checkpoint is configured to persist every 10 seconds. Which change should be made to prevent unnecessary reprocessing?

Accepted Answer

Enable idempotent writes to the sink by adding a unique identifier per event.. Option C is correct because enabling idempotent writes ensures that even if events are reprocessed due to pipeline restarts, the sink will deduplicate them based on the unique identifier. This prevents duplicate data from being written, which is the core issue when stateful transformations cause reprocessing of events from the last 30 minutes. The checkpoint interval (10 seconds) is already frequent enough; the problem is not checkpoint frequency but the lack of deduplication at the sink.

Answer

Use a non-volatile state backend like Cloud Bigtable for state storage.

Answer

Increase the checkpoint interval to 60 seconds to reduce frequency of checkpoints.

Answer

Decrease the checkpoint interval to 1 second to checkpoint more frequently.

Question 12

A data team uses Cloud Dataproc to run nightly Spark jobs. The job volume has increased, and the cluster is often underutilized during the day. They want to reduce costs while ensuring jobs can scale when needed. Which strategy should they adopt?

Accepted Answer

Use a cluster with a small number of primary workers and a large pool of preemptible workers, and enable autoscaling.. Option C is correct because it combines a small number of primary (non-preemptible) workers for reliability with a large pool of preemptible workers for cost-effective scaling, and enables autoscaling to dynamically adjust the cluster size based on workload. This minimizes cost during idle periods (preemptible instances are ~80% cheaper) while ensuring jobs can scale up quickly when needed, as autoscaling adds preemptible workers automatically. Preemptible workers are ideal for fault-tolerant Spark jobs that can handle node preemptions.

Answer

Use preemptible workers for both primary and secondary nodes to minimize cost.

Answer

Manually scale the cluster up before nightly jobs and down after.

Answer

Use custom machine types with local SSDs for primary workers to improve I/O.

Question 13

A company processes CSV files that are uploaded to Cloud Storage by external partners. Each file is around 500 MB, and they need to be parsed and loaded into BigQuery. The processing must start as soon as the file arrives. What is the most efficient serverless architecture?

Accepted Answer

Cloud Storage triggers a Cloud Function that publishes events to Pub/Sub; a Dataflow streaming pipeline reads from Pub/Sub and writes to BigQuery.. Option A is correct because it combines Cloud Storage event-driven triggers with Pub/Sub for reliable asynchronous message delivery, and uses Dataflow streaming with autoscaling to handle 500 MB files efficiently. This serverless architecture ensures processing starts immediately upon file arrival, scales to handle large files without manual intervention, and leverages BigQuery's streaming inserts for near-real-time data loading.

Answer

Use Cloud Scheduler to periodically check for new files and process them with Dataflow batch jobs.

Answer

Cloud Storage triggers a Dataproc job that reads the file and loads it into BigQuery.

Answer

Cloud Storage triggers a Cloud Function that directly loads the data into BigQuery using the BigQuery API.

Question 14

A company stores IoT sensor readings in BigQuery. The table is partitioned by day and clustered by sensor_id. Query performance has degraded as data grows; many queries filter by a date range and a single sensor_id. Which optimization should be applied first?

Accepted Answer

Recluster the table to ensure data is sorted by sensor_id within each partition.. Option D is correct because reclustering the table ensures that within each daily partition, the data is physically sorted by sensor_id. This optimizes the performance of queries that filter by a date range and a single sensor_id, as BigQuery can use the clustering metadata to prune blocks and read only the relevant data, reducing the amount of data scanned and improving query speed.

Answer

Remove clustering on sensor_id as it may cause overhead.

Answer

Add a WHERE clause to filter by partition date even if the query already filters by a date range.

Answer

Increase the number of BigQuery slots assigned to the project.

Question 15

A data engineering team uses Cloud Data Fusion to build ETL pipelines. They have a pipeline that reads from Cloud SQL, transforms data using Wrangler, and writes to BigQuery. The pipeline fails intermittently with a 'connection timeout' error from Cloud SQL. What is the best way to handle this?

Accepted Answer

Configure the Cloud SQL connector in Data Fusion to use retry logic and increase the connection timeout.. Option B is correct because Cloud Data Fusion's Cloud SQL connector can be configured with retry logic and an increased connection timeout to handle transient network issues. This directly addresses the intermittent 'connection timeout' error without requiring architectural changes, as the error is likely due to brief network latency or resource contention, not a persistent connectivity problem.

Answer

Use Cloud NAT to provide a static IP for Data Fusion to whitelist.

Answer

Increase the number of Data Fusion nodes to distribute the load.

Answer

Migrate Cloud SQL to Cloud Spanner to handle higher concurrency.