PDE Designing data processing systems Practice Test 8 — 15 Questions

Question 1

An organization wants to automate their batch data processing pipeline using Cloud Composer. The pipeline consists of multiple tasks: extract from Cloud Storage, transform with Dataflow, and load into BigQuery. Which Airflow operator should be used to run Dataflow jobs?

Accepted Answer

DataflowCreatePythonJobOperator. B is correct because the DataflowCreatePythonJobOperator is specifically designed to submit and manage Apache Beam pipelines written in Python as Dataflow jobs in Google Cloud. This operator handles the creation of a Dataflow job from a Python file, which aligns with the requirement to run Dataflow transformations within a Cloud Composer DAG.

Answer

BigQueryInsertJobOperator

Answer

GCSToBigQueryOperator

Answer

DataprocSubmitJobOperator

Question 2

A Dataflow streaming pipeline reads from Pub/Sub, applies a ParDo that uses a side input from a BigQuery table (refreshed hourly), and writes to BigQuery. The side input is large and causes increased latency and worker OOM errors. Which design change solves this?

Accepted Answer

Use a stateful ParDo and store the lookup data in an external cache like Cloud Bigtable, performing lookups per element.. Option A is correct because moving the large lookup data to an external cache like Cloud Bigtable offloads memory pressure from workers, eliminating OOM errors. The side input broadcast approach keeps the entire dataset in each worker's memory, which causes OOM when the data is large. Using an external cache allows per-element lookups without storing the entire dataset in memory, reducing latency by avoiding broadcast overhead.

Answer

Increase the side input broadcast frequency to update more often.

Answer

Split the pipeline into two: one to load the side input, the other to process main input.

Answer

Use smaller worker machine types to distribute memory across more workers.

Question 3

A data engineer is monitoring a Dataflow streaming pipeline and notices that the 'System Lag' metric is increasing. Which TWO actions should be taken to diagnose the issue?

Accepted Answer

Check the Dataflow monitoring UI for each stage's throughput and backlog.. Option A is correct because the Dataflow monitoring UI provides per-stage metrics such as throughput and backlog, which directly indicate where data is accumulating. By examining these metrics, you can identify the specific stage causing the increasing system lag, enabling targeted troubleshooting without unnecessary pipeline changes.

Answer

Cancel the pipeline and restart with a larger initial worker count.

Answer

Increase the maximum number of workers to handle backlog.

Answer

Increase the BigQuery quota for streaming inserts.

Question 4

A company is designing a data lake on Cloud Storage for analytics. They need to store data in various formats (Avro, Parquet, CSV) and enable efficient querying with BigQuery and Dataproc. Which THREE practices should they follow?

Accepted Answer

Use BigLake to create BigQuery tables that reference Cloud Storage data.. BigLake allows you to create BigQuery tables that reference data stored in Cloud Storage, enabling unified governance and fine-grained access control without moving data. This is essential for a data lake architecture where BigQuery and Dataproc need to query the same underlying data in various formats like Avro, Parquet, and CSV.

Answer

Disable encryption on the bucket to improve read performance.

Answer

Store all data in CSV format for simplicity.

Question 5

A company uses Pub/Sub to decouple services. They have a topic with two subscriptions: Subscription A is a push subscription that sends messages to a Cloud Function; Subscription B is a pull subscription used by a Dataflow job. They need to ensure that messages are processed in order for a specific device_id. Which TWO configurations should they apply?

Accepted Answer

Enable message ordering on the topic and set an ordering key for each message.. Option A is correct because enabling message ordering on the topic and setting an ordering key (e.g., device_id) ensures that messages with the same key are delivered to subscribers in the order they were published. This is a fundamental Pub/Sub feature that guarantees FIFO (first-in, first-out) delivery per ordering key, which directly addresses the requirement for processing messages in order for a specific device_id.

Answer

Disable duplicate filtering on the topic.

Answer

Configure the Cloud Function to retry on failure with exponential backoff.

Answer

Use a single subscription with multiple concurrent consumers.

Question 6

A streaming Dataflow job is processing messages from Cloud Pub/Sub. The job is underutilizing resources and the throughput is lower than expected. Which parameter should be adjusted to increase parallelism?

Accepted Answer

Increase the number of workers via maxNumWorkers. The job is underutilizing resources, meaning the existing workers are not fully loaded. Increasing the number of workers via maxNumWorkers directly increases parallelism by allowing Dataflow to distribute work across more VMs, which can increase throughput without changing the per-worker resource profile. This parameter controls the upper bound on the number of workers, enabling the autoscaler to scale out when there is backlog.

Answer

Change the workerMachineType to a higher CPU machine

Answer

Set the streaming engine to Dataflow Streaming Engine

Answer

Set autoscalingAlgorithm to THROUGHPUT_BASED

Question 7

A company stores IoT sensor data in BigQuery. Queries that filter on a timestamp column and a device_id column are slow even though the table is partitioned by day. What should the data engineer do to improve query performance?

Accepted Answer

Cluster the table on device_id. Clustering on device_id organizes the data within each day partition by device_id, allowing BigQuery to prune blocks during queries that filter on that column. This reduces the amount of data scanned and improves query performance without changing the partitioning scheme. Partitioning alone only limits scans by time range; clustering adds intra-partition sorting for non-time-based filters.

Answer

Increase the partition size to monthly

Answer

Switch to ingestion-time partitioning instead of column-based

Answer

Enable automatic query rewriting with BI Engine

Question 8

A financial services company uses Cloud Pub/Sub with ordering keys to process transactions in order. Some messages are failing processing and getting stuck. The team wants to ensure that if a message fails, it can be reprocessed later without blocking subsequent messages. What should they implement?

Accepted Answer

Configure a dead letter topic and handle the failed message separately. Option C is correct because a dead letter topic (DLT) allows failed messages to be moved aside after exhausting retry attempts, so they do not block the processing of subsequent ordered messages. In Cloud Pub/Sub, ordering keys require messages with the same key to be delivered in order; if a message fails and is not acknowledged, it blocks all later messages with the same key. By configuring a dead letter topic, the failed message is automatically forwarded to the DLT after a maximum of 5 delivery attempts (default), and the original subscription can continue processing the next messages in order. The team can then reprocess the failed message from the DLT separately, without affecting the order of other messages.

Answer

Create multiple subscriptions for the same topic

Answer

Use a pull subscription with flow control settings

Answer

Increase the acknowledgment deadline to 600 seconds

Question 9

A data engineer is running a Dataproc cluster for a batch ETL job that needs to process 10 TB of data. The job is memory-intensive. The cluster currently uses n1-standard-4 workers. Performance is poor. What is the most cost-effective change to improve performance?

Accepted Answer

Use high-memory machine types (n1-highmem-4). The job is memory-intensive, and n1-standard-4 workers have 15 GB of RAM, which may be insufficient for the workload, causing excessive disk spill or OOM errors. Switching to n1-highmem-4 provides 26 GB of RAM per worker (a 73% increase) without increasing vCPU count, directly addressing the memory bottleneck at a lower cost than adding more workers. This is the most cost-effective change because it improves performance without incurring the overhead of additional vCPUs or licensing costs.

Answer

Use preemptible workers to reduce cost

Answer

Switch to n2-standard-4 machine types

Answer

Add more n1-standard-4 workers

Question 10

A team is designing an event-driven data pipeline. They need to process messages from Cloud Pub/Sub, transform them, and write to BigQuery. The messages have variable volume and spikes. What is the best serverless compute option for this workload?

Accepted Answer

Cloud Dataflow with a streaming pipeline. Cloud Dataflow with a streaming pipeline is the best serverless compute option because it is purpose-built for unbounded, variable-volume data streams from Pub/Sub and provides exactly-once processing semantics, auto-scaling, and built-in BigQuery sink integration via the Beam SDK. Unlike simpler compute options, Dataflow handles backpressure, windowing, and state management natively, making it ideal for spikes and high-throughput transformations without manual scaling or idempotency concerns.

Answer

Cloud Functions triggered by Pub/Sub

Answer

Compute Engine with a Pub/Sub client library

Answer

Cloud Run invoked via Eventarc

Question 11

A Dataflow pipeline reads from Cloud Pub/Sub and writes to Cloud Storage. The pipeline needs to guarantee exactly-once processing despite worker failures. Which configuration ensures exactly-once semantics?

Accepted Answer

Enable exactly-once delivery on the Pub/Sub subscription and use an idempotent sink. Option D is correct because Pub/Sub subscriptions can be configured with exactly-once delivery (using the `enableExactlyOnceDelivery` flag), which ensures that each message is delivered to the subscriber exactly once. Combining this with an idempotent sink (e.g., Cloud Storage with unique filenames or deduplication logic) guarantees that even if a worker fails and the pipeline retries, the output will not contain duplicates. This is the only option that directly addresses both the source and sink to achieve end-to-end exactly-once semantics.

Answer

Use a side input from a deduplication dataset

Answer

Set the pipeline to use a global window with no early triggers

Answer

Insert a Reshuffle transform after reading

Question 12

A data engineer needs to automatically delete objects from a Cloud Storage bucket after 30 days and archive them to nearline storage after 7 days. Which configuration should they use?

Accepted Answer

Set a lifecycle rule to SetStorageClass to nearline after 7 days and delete after 30 days. Option C is correct because it implements a lifecycle rule that first transitions objects to Nearline storage after 7 days (reducing costs for infrequently accessed data) and then deletes them after 30 days. This matches the requirement to archive after 7 days and delete after 30 days, using the `SetStorageClass` and `Delete` actions in the correct chronological order.

Answer

Set a lifecycle rule to SetStorageClass to nearline after 30 days only

Answer

Set a lifecycle rule to delete objects after 7 days only

Answer

Set a lifecycle rule to delete objects after 7 days and SetStorageClass to nearline after 30 days

Question 13

A BigQuery table contains streaming data from Cloud Pub/Sub. The table is partitioned by ingestion time. A user runs a query that accesses data from the last 5 minutes and gets correct results. After 90 minutes, the user runs the same query again but notices that some rows are missing. What is the most likely cause?

Accepted Answer

The query is using time travel to a snapshot before the streaming buffer was committed. Option A is correct because BigQuery's streaming buffer provides low-latency access to recently ingested data, but this data is not immediately committed to managed storage. After the streaming buffer is flushed (typically within 90 minutes), the data becomes available in the table's base storage. If the user runs a query using time travel (e.g., `FOR SYSTEM_TIME AS OF`) to a snapshot taken before the buffer was committed, the query will only see data that was in managed storage at that snapshot time, missing rows that were still in the streaming buffer at that point.

Answer

The query is using cached results that exclude recent data

Answer

The schema of the table was modified after the initial query

Answer

The table has a partition expiration of 30 days

Question 14

In Cloud Composer, a DAG has two tasks: task_A (runs an Apache Spark job on Dataproc) and task_B (loads data from Cloud Storage to BigQuery). task_B must start after task_A completes. The DAG is scheduled to run hourly. Sometimes task_B starts before task_A finishes because task_A's Dataproc job appears to complete in the Airflow metadata but the data is not yet available. What is the best way to ensure task_B only runs after the data is fully written?

Accepted Answer

Use DataprocJobOperator with a job_poll_interval and add a sensor to verify output. Option C is correct because it addresses the root cause: the Dataproc job may report completion in Airflow metadata before the output data is fully written to Cloud Storage. By using DataprocJobOperator with a job_poll_interval, you ensure Airflow waits for the actual job completion on Dataproc, and adding a sensor to verify the output (e.g., checking for a success marker file or expected data in Cloud Storage) guarantees that task_B only starts after the data is fully available. This two-step approach prevents race conditions between job completion and data consistency.

Answer

Increase the number of retries for task_B

Answer

Use a sensor after task_A that checks for a specific file in Cloud Storage

Answer

Change the DAG schedule to run every 30 minutes

Question 15

Which TWO roles are required to allow a service account to run a Dataflow job and write results to BigQuery? (Choose two.)

Accepted Answer

roles/dataflow.worker. Option B is correct because the roles/dataflow.worker role grants the service account the necessary permissions to execute Dataflow worker tasks, such as reading from sources and writing to sinks. Option C is correct because roles/bigquery.dataEditor allows the service account to insert rows into BigQuery tables, which is required for the Dataflow job to write results.

Answer

roles/pubsub.subscriber

Answer

roles/storage.objectAdmin

Answer

roles/dataflow.admin