PDE Designing data processing systems Practice Test 6 — 15 Questions

Question 1

A data pipeline processes streaming data from Pub/Sub to BigQuery. The pipeline needs to handle late-arriving data that is up to 1 hour late. Which Dataflow feature should be used?

Accepted Answer

Fixed windows with allowed lateness. Fixed windows with allowed lateness are the correct choice because the pipeline needs to handle late-arriving data up to 1 hour late while processing data in fixed time intervals (e.g., 1-hour windows). The `allowedLateness` parameter in Dataflow (Apache Beam) allows late data to be included in the appropriate fixed window for up to the specified duration after the watermark passes the window end. This ensures that late Pub/Sub messages are correctly joined with their original window in BigQuery.

Answer

Global windows with watermark

Answer

Session windows

Answer

Sliding windows with allowed lateness

Question 2

A company uses Cloud Dataproc to run Spark jobs on ephemeral clusters. The input data is in Cloud Storage and output is also to Cloud Storage. The cluster is created and deleted daily. The cost is high due to spinning up nodes. Which change can reduce cost without sacrificing performance?

Accepted Answer

Use a combination of standard and preemptible VMs for worker nodes. Option C is correct because using a combination of standard and preemptible VMs for worker nodes reduces cost significantly while maintaining performance. Preemptible VMs are up to 80% cheaper than standard VMs, and since Spark is fault-tolerant and can handle node preemptions via speculative execution, the job can complete without performance degradation. Standard VMs for master nodes ensure cluster stability, while preemptible workers handle the bulk of data processing.

Answer

Use standard VMs with a larger number of smaller machines

Answer

Use Cloud Dataflow instead

Answer

Use preemptible VMs for all nodes

Question 3

A company is designing a data lake on Google Cloud. They need to store raw data in multiple formats (CSV, Parquet, Avro) and allow various downstream processing frameworks. Which two storage solutions provide flexibility and scalability? (Choose two.)

Accepted Answer

BigQuery. BigQuery is correct because it can directly query raw data stored in Cloud Storage in formats like CSV, Parquet, and Avro using external tables or federated queries, without requiring data loading. This provides a flexible, serverless analytics layer that scales automatically and integrates with downstream processing frameworks like Apache Spark, Dataflow, and Dataproc.

Answer

Cloud Filestore

Answer

Cloud Spanner

Answer

Cloud Bigtable

Question 4

A streaming pipeline uses Cloud Pub/Sub and Dataflow to process financial transactions. The pipeline must guarantee that each transaction is processed exactly once and in order per customer key. Which two configurations are necessary? (Choose two.)

Accepted Answer

Use Dataflow stateful processing with event time ordering. Options A and B are correct. Pub/Sub ordering keys (A) ensure messages with the same ordering key are delivered in order. Dataflow stateful processing with event time ordering (B) allows processing events in order while maintaining exactly-once semantics. Option C (global window with trigger) does not guarantee order. Option D (keyed state) is required but is encompassed by B. Option E (session window) is not about ordering.

Answer

Use a session window with max gap duration

Answer

Use a keyed state with a value state per customer

Answer

Use a global window with a trigger

Question 5

A company is planning to migrate a legacy batch ETL pipeline to Google Cloud. The pipeline involves reading from a relational database, transforming data, and writing to a data warehouse. Which three Google Cloud services can be used as the orchestration layer? (Choose three.)

Accepted Answer

Cloud Scheduler. Cloud Scheduler is a fully managed cron job service that can trigger orchestration workflows on a schedule. It is correct because it can initiate batch ETL pipelines by sending HTTP requests to Cloud Run, Cloud Functions, or Pub/Sub, making it a lightweight orchestration trigger for scheduled batch jobs.

Answer

Cloud Dataproc

Answer

Cloud Dataflow

Question 6

A data engineer runs this Dataflow template to load CSV files from Cloud Storage into BigQuery. The job fails with a 'File pattern not matching any files' error. What is the most likely cause?

Accepted Answer

The CSV files are stored in a subdirectory that is not matched by the pattern. The error 'File pattern not matching any files' indicates that the file pattern specified in the Dataflow template does not resolve to any existing objects in Cloud Storage. If the CSV files are stored in a subdirectory (e.g., gs://bucket/subdir/*.csv) but the pattern only references the root (e.g., gs://bucket/*.csv), no files will be matched. This is the most likely cause because the pattern must explicitly include the subdirectory path.

Answer

The bucket name is incorrectly spelled

Answer

The template has a bug

Answer

The output table does not exist

Question 7

A team configured a garbage collection rule on a Cloud Bigtable column family with max_age of 100 seconds. After 2 minutes, they notice that data older than 100 seconds is still present. What is the most likely reason?

Accepted Answer

Garbage collection runs only periodically (e.g., once per day). Cloud Bigtable garbage collection (GC) is not applied in real time; it runs as a background process that typically executes once per day. Even though the max_age rule is set to 100 seconds, the actual deletion of expired data occurs only during the next scheduled GC cycle, which may not happen for up to 24 hours. Therefore, observing data older than 100 seconds after only 2 minutes is expected behavior.

Answer

They need to apply the rule using a different API

Answer

The max_age must be at least 1 hour

Answer

The rule is applied only to new data, not existing data

Question 8

A team has set up a push subscription to an HTTPS endpoint. They notice that messages are not being acknowledged and are resent every 10 seconds. What is the most likely issue?

Accepted Answer

The push endpoint is returning HTTP 500. In Google Cloud Pub/Sub push subscriptions, the subscriber must acknowledge messages by returning an HTTP 200 status code. If the endpoint returns HTTP 500, Pub/Sub interprets this as a failure and will retry delivery with exponential backoff, but the default minimum retry interval is 10 seconds. This matches the observed behavior of messages being resent every 10 seconds without acknowledgment.

Answer

The push endpoint is returning HTTP 200 but taking too long to process

Answer

The push endpoint is returning HTTP 200 with 'ack' in the body

Answer

The push endpoint is returning HTTP 400

Question 9

A company processes real-time clickstream data from websites. They need to aggregate user sessions that may span multiple hours and handle events that arrive late due to network delays. The pipeline must avoid discarding late data. Which Dataflow feature should they configure?

Accepted Answer

Use session windows with a gap duration and allow late data with a suitable allowed_lateness. Session windows are ideal for aggregating user sessions that span multiple hours, as they group events based on a gap duration of inactivity. By configuring `allowed_lateness`, the pipeline can handle late-arriving events without discarding them, ensuring completeness. This directly addresses the requirement to avoid discarding late data while aggregating sessions.

Answer

Use fixed windows with a trigger that fires after every element

Answer

Use the GlobalWindow with a watermark

Answer

Use sliding windows with no allowed lateness

Question 10

A data analyst frequently queries a BigQuery table that contains an array of structs representing product purchases. The query below runs slowly:

SELECT customer_id, COUNT(purchase) as total_purchases
FROM sales, UNNEST(purchases) as purchase
GROUP BY customer_id

What change would most improve query performance?

Accepted Answer

Create a materialized view that pre-aggregates by customer_id and purchase count. The query runs slowly because it must unnest the `purchases` array for every row and then aggregate. A materialized view pre-aggregates the data by `customer_id` and purchase count, avoiding repeated full scans and unnesting. This is the most impactful optimization because it eliminates the compute cost of UNNEST and GROUP BY at query time.

Answer

Partition the table by transaction date

Answer

Use a subquery to filter purchases first

Answer

Cluster the table by purchases.product_id

Question 11

A Dataflow pipeline reads events from Pub/Sub and transforms them. Some events contain invalid product IDs that should be filtered out. The list of valid product IDs is stored in a frequently updated BigQuery table. What is the best approach to filter out invalid events?

Accepted Answer

Read the BigQuery table as a side input and refresh it periodically using a global window with a periodic trigger. Option A is correct because reading the BigQuery table as a side input with a global window and periodic trigger allows the pipeline to refresh the list of valid product IDs at a configurable interval without reprocessing the entire stream. This pattern is idiomatic for Beam/Dataflow when the reference data changes frequently and must be kept reasonably current while maintaining low latency for streaming events.

Answer

Use a Combine.PerKey to group by product ID and then filter

Answer

Use a custom pipeline option to read the valid IDs at startup and cache them

Answer

Use a ParDo with a side input that is a MapSideInput of valid IDs, and refresh it on each element

Question 12

A manufacturing company wants to detect anomalies in sensor data from thousands of IoT devices in real time. The data is streaming into Pub/Sub. The best solution should use a machine learning model served from AI Platform that scores sensor readings aggregated over 5-minute windows. Which pipeline design meets these requirements?

Accepted Answer

Use Cloud Dataflow with sliding windows to aggregate sensor readings every 5 minutes, then call a trained model hosted on AI Platform Prediction for each window. Option C is correct because Cloud Dataflow's sliding windows natively handle the 5-minute aggregation requirement for streaming data, and its ability to call external services via a DoFn allows integration with AI Platform Prediction for real-time model scoring. This design aligns with the need for low-latency, scalable processing of Pub/Sub streams without managing infrastructure.

Answer

Use Cloud Dataproc with Spark Streaming to aggregate data, and use a Spark ML model embedded in the pipeline

Answer

Use BigQuery streaming inserts and run scheduled queries that call the ML model

Answer

Use Cloud Functions triggered by Pub/Sub to process each sensor reading individually

Question 13

A company runs a nightly Dataproc batch job to process large log files. The job is idempotent and can tolerate node failures if restarted. Minimizing cost is critical. What is the most cost-effective cluster design?

Accepted Answer

Use preemptible instances for all nodes and enable automatic restart. Preemptible instances cost about 80% less than standard instances, making them the most cost-effective choice for fault-tolerant, idempotent batch jobs. Enabling automatic restart ensures that if a preemptible instance is terminated (which can happen at any time), Dataproc will automatically recreate it, maintaining cluster capacity without manual intervention. This design minimizes cost while preserving the job's ability to complete despite node failures.

Answer

Use standard instances with autoscaling based on YARN memory

Answer

Use all preemptible instances and configure the cluster to delete after the job completes

Answer

Use a single-node cluster with a high-memory machine type

Question 14

A company wants to implement a data lake on Google Cloud to store raw sensor data (unstructured binary files) and allow data scientists to run SQL queries on processed data. They expect to store terabytes of data and have different access patterns. Which combination of GCP services best meets these requirements?

Accepted Answer

Cloud Storage for raw data and BigQuery for processed data. Cloud Storage is the ideal service for storing raw, unstructured binary sensor data at petabyte scale, offering low-cost, durable object storage with multiple access tiers. BigQuery is a serverless, highly scalable data warehouse that allows data scientists to run SQL queries on processed data, with features like columnar storage and automatic optimization for analytical workloads. This combination directly addresses the need for raw storage and SQL-based analytics on processed data.

Answer

Bigtable for raw data and Cloud Spanner for processed data

Answer

Cloud Storage for both raw and processed data

Answer

Cloud SQL for raw data and Cloud Dataproc for processing

Question 15

A data engineering team needs to build a data integration pipeline that involves connecting to multiple sources, performing data transformations with visual editing, and then running custom machine learning algorithms. The team has both data analysts and data scientists. Which approach is most suitable?

Accepted Answer

Use Cloud Data Fusion for the initial ingestion and transformations, then export the data to Cloud Dataproc for the ML algorithms. Option D is correct because it leverages Cloud Data Fusion's visual, no-code interface for data ingestion and transformation, which is ideal for data analysts, and then exports the prepared data to Cloud Dataproc, which provides native support for custom machine learning algorithms using Spark or Hadoop, meeting the data scientists' needs. This separation of concerns optimizes the pipeline for both user groups and avoids forcing all tasks into a single tool that may not excel at both visual ETL and custom ML.

Answer

Use Cloud Composer to orchestrate both Data Fusion and Dataproc

Answer

Use only Cloud Dataproc for all steps

Answer

Use only Cloud Data Fusion for all steps