PDE Designing Data Processing Systems Practice Test 12 — 15 Questions

Question 1

An online gaming company runs a Dataflow streaming pipeline that aggregates player actions into per-session metrics. Sessions are defined by a gap duration of 30 minutes of inactivity, and the pipeline must emit final session results even when a session's events arrive out of order by up to 10 minutes. Late data beyond that window can be dropped. Which combination of Beam concepts should the pipeline use?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Session windows with a 30-minute gap, an allowed lateness of 10 minutes, and a trigger that fires after the watermark passes the end of the window.. Session windows are the only window type that models inactivity gaps, and combining a 30-minute gap with 10 minutes of allowed lateness keeps state open long enough for out-of-order events while dropping anything later. A watermark-based trigger emits the final result once the window is considered complete.

Answer

Fixed windows of 30 minutes with the default trigger and allowed lateness of zero.

Answer

Sliding windows of 30 minutes with a one-minute period and an early trigger firing every minute.

Answer

Global window with a repeated element-count trigger and a discard accumulation mode.

Question 2

A startup wants to build a data lake on Google Cloud to store raw JSON, CSV, and Parquet files from various sources. They need a storage solution that is highly durable, globally accessible, and integrates natively with BigQuery and Dataproc. They want to minimize management overhead. Which Google Cloud service should they use?

Accepted Answer

Cloud Storage. Cloud Storage is the foundational object store for data lakes on Google Cloud. It offers high durability, global accessibility, and native integration with analytics services like BigQuery and Dataproc. It requires no capacity planning or server management, aligning with the goal of minimal overhead. Relational, NoSQL, and NFS services are not suited for raw file storage at scale.

Answer

Filestore

Answer

Bigtable

Answer

Cloud SQL

Question 3

A startup is building a data lake on Google Cloud. They need to store raw JSON, CSV, and Parquet files from various sources. The files will be accessed by multiple analytics tools, including BigQuery and Dataproc. The startup wants a cost-effective, durable, and highly available storage solution that integrates natively with these services. Which Google Cloud service should they use?

Accepted Answer

Cloud Storage with a Standard storage class.. Cloud Storage is the correct choice for a data lake because it is a durable, highly available, and cost-effective object store that integrates natively with BigQuery, Dataproc, and other Google Cloud analytics services. It supports all file formats and can be organized into buckets with lifecycle policies for cost optimization. The Standard storage class is appropriate for frequently accessed data.

Answer

Cloud SQL for MySQL with a large SSD and binary large object (BLOB) storage.

Answer

Cloud Bigtable with a column family for each data format.

Answer

Persistent Disk attached to a Compute Engine instance, shared via NFS.

Question 4

A financial services firm ingests trade events into Cloud Storage and must load them into BigQuery. Compliance requires that each event be processed exactly once and that the load be idempotent across retries, even if a Dataflow job restarts mid-batch. The destination table must also be queryable immediately after each successful load. Which loading approach best satisfies these requirements?

Accepted Answer

Use a Dataflow pipeline with the BigQueryIO Storage Write API in exactly-once mode, writing to the table with deterministic record identifiers.. Exactly-once loading into BigQuery is achieved with the Storage Write API's exactly-once streams, where offsets are committed only on checkpoint and deterministic record IDs prevent duplicates on restart. This gives idempotent loads across retries and makes committed rows immediately queryable. Streaming inserts and periodic batch appends both risk duplicates and cannot meet the compliance constraint.

Answer

Use BigQuery's LOAD DATA statement triggered by Cloud Storage object notifications, appending each new file into the destination table.

Answer

Use a Dataproc Spark job that writes Parquet files to Cloud Storage, then run a scheduled bq load command every fifteen minutes to append into BigQuery.

Answer

Use a Dataflow pipeline that writes to BigQuery with STREAMING_INSERTS and relies on the pipeline's exactly-once semantics to deduplicate on retry.

Question 5

A retail analytics team must move 30 TB of Parquet files from an on-premises Hadoop cluster into BigQuery once, then run standard SQL dashboards. The transfer window is 48 hours and the source cluster has limited outbound bandwidth. Which approach should the data engineer choose?

Accepted Answer

Export the Parquet files to encrypted portable drives and use the Transfer Appliance to ship them to a Google ingest location, then load the objects into BigQuery.. When the data volume is large and the source's outbound bandwidth is the constraint, an offline transfer avoids the network bottleneck. Transfer Appliance ships encrypted storage to Google, where the data lands in Cloud Storage, and a BigQuery load job then ingests the Parquet files into native managed tables. Dashboards subsequently query columnar managed storage rather than external objects, so both the migration deadline and query performance requirements are met.

Answer

Use the Storage Transfer Service with a transfer job configured against the on-premises source, then load the resulting Cloud Storage objects into BigQuery using a load job.

Answer

Create a Dataproc cluster in the same region and use a Spark job to read from the on-premises Hadoop cluster over the network and write directly into BigQuery.

Answer

Run gcloud storage cp from a Compute Engine VM to upload the files into a BigQuery-managed bucket, then query them with an external table.

Question 6

A logistics company uses Cloud Dataflow to process a continuous stream of GPS events from delivery trucks. The pipeline must compute the distance traveled per truck per hour and write the results to BigQuery. The events are keyed by truck ID, and the pipeline uses windowing with a one-hour fixed window. The team notices that some trucks report events with timestamps that are several minutes late due to network delays. They want to ensure that late events are still included in the correct window and that the results are emitted only after a reasonable wait. What should they configure?

Accepted Answer

Set a trigger that fires when the watermark passes the end of the window, and set allowed lateness to a duration longer than the expected delay.. To include late events in the correct window, you must set allowed lateness to a value greater than the expected delay. The trigger on the watermark passing the window end ensures results are emitted after the window closes, and the allowed lateness keeps the window state alive to accept late data. This configuration balances timely results with completeness.

Answer

Use a session window with a gap duration equal to the expected network delay.

Answer

Set a trigger that fires when the watermark passes the end of the window, and set allowed lateness to zero.

Answer

Set a trigger that fires every minute, and set allowed lateness to zero.

Question 7

Your team is migrating an on-premises Apache Hadoop cluster to Google Cloud. The cluster runs MapReduce jobs that read and write data to HDFS. You want to minimize code changes and operational overhead. Which Google Cloud service should you use to run these jobs?

Accepted Answer

Cloud Dataproc. Cloud Dataproc is designed for running Hadoop and Spark workloads in Google Cloud. It provides a managed cluster with Hadoop, Spark, and Hive, and can use Cloud Storage as a replacement for HDFS via the Cloud Storage connector. Existing MapReduce jobs can often run with little modification, and Dataproc handles cluster management, making it the best choice for migrating on-premises Hadoop workloads with minimal code changes and operational overhead.

Answer

BigQuery

Answer

Cloud Composer

Answer

Cloud Dataflow

Question 8

A media company streams playback telemetry through Pub/Sub into a Dataflow pipeline that writes to BigQuery. During prime-time peaks, the pipeline's BigQuery write step shows growing latency and the job repeatedly reports that it is backing off on insert retries. The team wants to reduce write pressure without changing the downstream table schema or losing exactly-once semantics. What should they do?

Accepted Answer

Switch the sink to the BigQuery Storage Write API with the exactly-once stream type and group records into batches before writing.. Backoff on insert retries signals that the write path is saturating BigQuery's streaming ingestion limits. The Storage Write API's exactly-once stream type preserves exactly-once semantics through offset-based deduplication, and batching rows into larger appends cuts request volume and quota pressure. This keeps the existing table schema and the streaming model intact while removing the write bottleneck that was throttling the pipeline during peak load.

Answer

Enable Dataflow Shuffle and set the pipeline's disk size larger so the write stage can buffer more rows before retrying.

Answer

Change the pipeline to use a BigQuery load job triggered every 15 minutes by writing micro-batches to Cloud Storage first.

Answer

Increase the number of BigQuery Streaming Inserts API calls by adding more Dataflow workers so each record is inserted individually.

Question 9

You need to design a data processing system that ingests streaming data from thousands of IoT devices. The data must be processed in real-time to calculate average temperature per device over 1-minute intervals, and the results should be stored in BigQuery for analysis. You want a serverless solution with minimal management. Which combination of Google Cloud services should you use?

Accepted Answer

Cloud Pub/Sub for ingestion, Cloud Dataflow for processing, and BigQuery for storage.. The scenario requires serverless ingestion, real-time processing with windowing, and analytical storage. Pub/Sub handles ingestion at scale, Dataflow provides serverless stream processing with windowing to compute 1-minute averages, and BigQuery stores results for analysis. This combination is fully managed and requires minimal operational effort.

Answer

Cloud Pub/Sub for ingestion, Cloud Functions for processing, and BigQuery for storage.

Answer

Cloud Pub/Sub for ingestion, Cloud Dataproc for processing, and Cloud Bigtable for storage.

Answer

Cloud IoT Core for ingestion, Cloud Dataproc for processing, and BigQuery for storage.

Question 10

You are designing a data processing architecture on Google Cloud. You need to ingest data from multiple sources, including streaming events and batch files, and process them to produce a unified dataset for analytics. The solution must support both real-time and historical processing with the same codebase, and be able to handle late-arriving data. Which two Google Cloud services should you use together to achieve this? (Choose two.)

Accepted Answer

Cloud Pub/Sub. To achieve unified batch and stream processing with the same codebase and handle late data, you need a processing engine that supports Apache Beam, such as Cloud Dataflow, and a scalable ingestion service for streaming data, such as Cloud Pub/Sub. Dataflow's Beam model allows you to write one pipeline that works for both batch and streaming, with built-in support for event-time windowing and late data. Pub/Sub provides reliable, scalable ingestion for streaming events.

Answer

Cloud Composer

Answer

Cloud Dataproc

Answer

BigQuery

Question 11

A financial services firm must process payment events in strict order per account and cannot tolerate duplicates. The events arrive in Pub/Sub and must be written to BigQuery. The engineering team is designing the pipeline and wants to guarantee that each account's events are applied in the order they were published. Which approach should they take?

Accepted Answer

Enable message ordering on the Pub/Sub subscription and use the ordering key set to the account ID, then process with a Dataflow pipeline that preserves order per key.. Pub/Sub ordering keys combined with a key-aware Dataflow pipeline enforce per-account order without sacrificing parallelism across accounts. Sorting after the fact or using a single global consumer either fails to guarantee application order or imposes unacceptable throughput limits.

Answer

Write all events to a Cloud Storage bucket and run a nightly batch job that sorts by timestamp before loading into BigQuery.

Answer

Use a single Pub/Sub subscriber with one thread to process all events sequentially across all accounts.

Answer

Use a Pub/Sub pull subscription with multiple subscribers and rely on BigQuery to sort events by timestamp during queries.

Question 12

A healthcare analytics group must build a pipeline that ingests HL7 messages from an on-premises interface engine, must retain raw messages for seven years for compliance, and must expose de-identified aggregates to analysts. The security team requires that protected health information never be written to a dataset analysts can query, and that all data be encrypted with keys the organization manages and can revoke. Which two design choices satisfy these requirements? (Choose two.)

Accepted Answer

De-identify the messages in Dataflow, write only the de-identified aggregates into a separate BigQuery dataset secured with CMEK and policy tags, and grant analysts access only to that dataset.. The requirements demand a hard separation between raw PHI and anything analysts can query, plus organization-managed revocable encryption and seven-year retention. Landing raw messages in Cloud Storage with CMEK and a retention lock, accessible only to the ingestion service account, satisfies retention and key control. De-identifying in Dataflow and publishing only aggregates to a separate CMEK-protected BigQuery dataset keeps PHI out of analyst-facing storage entirely, so both boundaries hold independently.

Answer

Load the raw messages directly into a BigQuery table that analysts can query, and rely on column-level security to mask the PHI columns.

Answer

Store the raw messages in BigQuery and use authorized views so analysts query a view that filters out PHI columns instead of the base table.

Answer

Encrypt the raw messages with a customer-supplied key in the on-premises interface engine and upload only the ciphertext to Cloud Storage, decrypting in Dataflow for de-identification and discarding the key afterward.

Question 13

A logistics company collects GPS pings from delivery vehicles into Pub/Sub and needs to compute the distance traveled per vehicle per hour. The data volume is high and bursty, and the company wants a managed service that automatically scales the number of workers based on load while allowing custom windowing and stateful processing. Which service should they use?

Accepted Answer

Dataflow with a streaming pipeline using windowing and stateful DoFn. Dataflow provides managed autoscaling and native support for windowing and stateful processing, which are required to compute per-vehicle hourly distance from a bursty stream. The other options either lack stateful windowing, do not scale automatically, or are not designed for continuous stream processing.

Answer

Cloud Functions triggered by Pub/Sub messages

Answer

Dataproc with a Spark Streaming job on a fixed-size cluster

Answer

BigQuery with a scheduled query that reads from a Pub/Sub subscription

Question 14

A small startup wants to run a nightly batch job that transforms a 2 GB CSV file in Cloud Storage and writes the result back as Parquet. The team has no cluster administration experience, wants per-job pricing rather than an always-on cluster, and needs the transformation to run in under thirty minutes. Which Google Cloud approach should they choose?

Accepted Answer

Use a serverless Spark job on Dataproc Serverless, submitting the transformation and letting the platform provision and tear down resources per run.. Dataproc Serverless executes Spark workloads without any cluster provisioning, and you pay for the resources consumed while the workload runs. A nightly 2 GB CSV-to-Parquet transformation finishes well within the thirty-minute window, so the team gets per-job pricing and avoids cluster administration entirely.

Answer

Use Cloud Data Fusion to build a visual pipeline and run it on a dedicated ephemeral Dataproc cluster each night.

Answer

Create a persistent Dataproc cluster with a fixed number of worker nodes and submit the job to it every night.

Answer

Run the transformation on a Compute Engine VM with a cron job that invokes a local Spark installation each night.

Question 15

A retail company is designing a Dataflow pipeline to process point-of-sale transactions from Cloud Pub/Sub and write to BigQuery. The pipeline must handle late-arriving data up to 24 hours and ensure that all data is written to BigQuery exactly once, even in the event of worker failures. Which two features should the engineer implement to meet these requirements? (Choose two.)

Accepted Answer

Set the allowed lateness to 24 hours on the windowing strategy.. To handle late-arriving data up to 24 hours, the pipeline must set allowed lateness to 24 hours on the windowing strategy. To achieve exactly-once writes to BigQuery, the engineer should use the Storage Write API with a deterministic deduplication key and enable Dataflow's ExactlyOnce mode. Together, these features satisfy both requirements.

Answer

Use the Dataflow ExactlyOnce processing mode and enable the BigQueryIO.Write with FILE_LOADS and a trigger that fires after 24 hours.

Answer

Use BigQueryIO.Write with the UseBeamSchema option and set the write disposition to WRITE_APPEND.

Answer

Configure the pipeline to use the BigQueryIO.Write with STREAMING_INSERTS and set the insertId based on a unique transaction identifier.