PDE Building and operationalizing data processing systems Practice Test 2 — 15 Questions

Question 1

A company is building a real-time streaming pipeline using Pub/Sub and Dataflow to process clickstream data. The pipeline writes aggregated metrics to BigQuery every 10 seconds using a fixed window. During peak traffic, some windows produce duplicate rows in BigQuery. What is the most likely cause?

Accepted Answer

Dataflow is retrying BigQuery streaming inserts after a timeout, and the retries succeed even though the original insert succeeded.. Option A is correct because Dataflow uses at-least-once semantics for streaming inserts into BigQuery. When a streaming insert times out, Dataflow retries the insert, and if the original insert actually succeeded but the acknowledgment was lost, the retry produces a duplicate row. This is a known behavior of BigQuery streaming inserts with retry logic.

Answer

The pipeline uses default triggers instead of after-watermark triggers.

Answer

The fixed window duration is too short, causing overlapping windows.

Answer

The pipeline is using too many Dataflow workers, causing load balancing issues.

Question 2

A data engineer is designing a batch ETL pipeline that reads CSV files from Cloud Storage, transforms them using Dataproc, and writes the results to BigQuery. The data volume is expected to grow 10x in the next year. Which design approach best balances cost and performance?

Accepted Answer

Use a Dataproc cluster with preemptible worker nodes and autoscaling enabled.. Option C is correct because preemptible worker nodes significantly reduce cost (up to 80% discount) while autoscaling dynamically adjusts cluster size to match the growing workload, ensuring performance without over-provisioning. This combination handles the 10x data growth efficiently by scaling out during peak loads and scaling in during lulls, using preemptible instances for fault-tolerant tasks like transformation.

Answer

Create a single large persistent Dataproc cluster to handle the peak load.

Answer

Use Cloud Data Fusion to visually design the pipeline and run it on Dataproc.

Answer

Migrate the pipeline to Dataflow with Apache Beam and use flexRS for cost savings.

Question 3

A data team uses Cloud Composer to orchestrate Airflow DAGs. They need to ensure that a downstream task runs only if at least two out of three upstream sensor tasks succeed. Which TWO configurations should they combine?

Accepted Answer

Set trigger_rule to 'none_failed_or_skipped' and use a condition.. Option A is correct because the 'none_failed_or_skipped' trigger rule triggers the downstream task when all upstream tasks have succeeded or been skipped. Combined with a condition (e.g., using a PythonOperator or BranchPythonOperator) that checks whether at least two of the three sensor tasks succeeded, this ensures the downstream task runs only when the required threshold is met. This approach leverages Airflow's built-in trigger rules and conditional logic to implement a quorum-based dependency.

Answer

Set trigger_rule to 'one_success'.

Answer

Set trigger_rule to 'all_done'.

Answer

Set trigger_rule to 'none_failed'.

Question 4

A Dataflow pipeline reads log files from Cloud Storage, parses them into LogEvent objects, and writes to BigQuery. The pipeline fails with the above errors. What is the most likely cause?

Accepted Answer

The LogEvent class does not have a no-argument constructor.. Apache Beam's SDK requires that custom types used as PCollection elements (like LogEvent) have a no-argument constructor so that the framework can deserialize objects during distributed processing, especially when using the Dataflow runner. Without it, the pipeline fails at runtime with a serialization error because Beam's default coder (e.g., SerializableCoder) cannot reconstruct the object.

Answer

The pipeline is missing required import statements for LogEvent.

Answer

The BigQuery table schema does not match the LogEvent fields.

Answer

The log files are not in the expected format, causing parsing failures.

Question 5

A financial services company runs a batch Dataflow pipeline daily to process transaction data. The pipeline reads from Cloud Storage, performs complex transformations, and writes to BigQuery. Recently, the pipeline has been failing intermittently with the error: 'Workflow failed. Causes: (9c3f7a2b1d4e): The worker missed 2000 data samples in the last 30 seconds. This can be caused by a variety of factors, including slow work items, network issues, or resource contention.' The team has already increased the number of workers and tried using e2-standard-8 machine types, but the issue persists. The pipeline processes approximately 500 GB of data per run and uses approximately 200 workers. The team suspects that the issue might be related to shuffle operations. What should the team do next to resolve the issue?

Accepted Answer

Increase the persistent disk size per worker to 100 GB.. The error indicates that workers are missing data samples due to slow shuffle operations, often caused by insufficient disk I/O. Increasing the persistent disk size per worker to 100 GB provides more local scratch space for Dataflow's shuffle, reducing disk contention and allowing the shuffle to complete within the 30-second window. This directly addresses the root cause without changing the worker count or machine type.

Answer

Enable Streaming Engine for the pipeline.

Answer

Reduce the number of workers to 100 to decrease shuffle overhead.

Answer

Use Cloud Storage as a shuffle sink.

Question 6

Drag and drop the steps to set up a BigQuery dataset with a scheduled query into the correct order.

Question 7

Drag and drop the steps to set up Cloud IAP (Identity-Aware Proxy) for an App Engine app into the correct order.

Question 8

Drag and drop the steps to create a Cloud Function triggered by Cloud Storage events into the correct order.

Question 9

Match each data pipeline term to its definition.

Question 10

Match each data storage term to its characteristic.

Question 11

Match each data lifecycle stage to its description.

Question 12

A company uses Cloud Dataflow to process streaming data from Pub/Sub into BigQuery. The pipeline uses a side input from a Cloud Bigtable table containing user profile information to enrich the events. The side input is updated every hour. Which approach should the company use to ensure that the pipeline uses the latest profile data without causing high memory usage?

Accepted Answer

Use a side input that is periodically refreshed by reading the Cloud Bigtable table at a regular interval.. Option A is correct because Cloud Dataflow supports periodically refreshing side inputs by reading from an external source like Cloud Bigtable at a specified interval. This approach keeps the profile data up-to-date without storing the entire side input in memory for the lifetime of the pipeline; instead, the side input is rebuilt and cached only when refreshed, controlling memory usage.

Answer

For each incoming event, read the corresponding profile from Cloud Bigtable using a synchronous call.

Answer

Use a CoGroupByKey transform to join the stream with a bounded PCollection created from the Cloud Bigtable table.

Answer

Stream the profile updates into a separate BigQuery table and use a BigQuery streaming query to join in real-time.

Question 13

A data engineering team uses Cloud Pub/Sub to ingest clickstream events and Cloud Dataflow to process them. They need to maintain strict event ordering per user session, and the processing output must be written to a BigQuery table with exactly-once semantics. Which configuration should the team implement?

Accepted Answer

Enable message ordering in Pub/Sub with a session ID as the ordering key, and in Dataflow use a global window with a custom trigger that fires on watermark and uses a BigQuery sink with 'exactly-once' mode enabled.. Option A is correct because it combines Pub/Sub message ordering (using a session ID as the ordering key) with Dataflow's exactly-once sink to BigQuery. The global window with a watermark-based trigger ensures all events for a session are processed in order before writing, while the BigQuery 'exactly-once' mode prevents duplicate rows even if the pipeline retries. This satisfies both strict per-session ordering and exactly-once semantics.

Answer

Use a Pub/Sub pull subscription with a subscriber that acknowledges messages immediately after processing, and a Dataflow pipeline with a sliding window.

Answer

Assign a unique session ID as the message ordering key in Pub/Sub, use a Dataflow pipeline with session windows and .withAllowedLateness(0), and write to BigQuery using a batch load.

Answer

Use a Pub/Sub push subscription with an acknowledgment deadline of 600 seconds and enable exactly-once delivery on the subscription.

Question 14

A company runs a Dataproc cluster for nightly batch jobs. The cluster uses preemptible workers for cost savings. Recently, the jobs have been failing intermittently with 'Disk quota exceeded' errors on the persistent disks attached to the preemptible workers. The cluster is configured with a master node and 10 worker nodes, each with a 100 GB persistent disk. The preemptible workers are dynamically added and removed. What is the most likely cause and the best long-term solution?

Accepted Answer

The preemptible workers are exceeding the project's persistent disk quota in the region because every time a preempted worker restarts, it tries to attach a new disk. Increase the disk quota.. Option C is correct because the intermittent 'Disk quota exceeded' errors on preemptible workers are caused by the project's regional persistent disk quota being exhausted. When a preemptible worker is reclaimed, the cluster attempts to attach a new persistent disk to the replacement worker, but the old disk is not immediately deleted, leading to a buildup of unattached disks that consume quota. The best long-term solution is to increase the persistent disk quota in the region to accommodate the temporary disks from preempted workers.

Answer

The persistent disks of the preemptible workers are too small. Resize the persistent disks to 200 GB each.

Answer

The preemptible workers are using local SSDs that are not recreated on reclaim. Use non-preemptible workers with local SSDs instead.

Answer

The preemptible workers do not have enough persistent disk space to store intermediate shuffle data. Switch to standard workers to avoid this issue.

Question 15

A company uses BigQuery for real-time analytics. They stream data from IoT devices into a BigQuery table. After a few hours, some of the recent data becomes visible in the table although it was streamed less than 10 minutes ago. The data team confirms that no one ran any manual queries. What is the most likely reason for the data visibility?

Accepted Answer

The streaming buffer reached its maximum capacity (default 90 minutes) and automatically flushed the data to the table.. Option D is correct because BigQuery's streaming buffer has a maximum capacity limit, typically around 90 minutes. When the buffer reaches this capacity, BigQuery automatically flushes the buffered data to the table, making it visible. This explains why data streamed less than 10 minutes ago became visible after a few hours.

Answer

The data was stored in the streaming buffer for more than 24 hours, and BigQuery automatically flushes it to the table.

Answer

BigQuery time travel allows querying data from the past, including data still in the streaming buffer.

Answer

The table has an expiration set, and the data is made visible as soon as the table is about to expire.