PDE Practice Test 27 — 15 Questions

Question 1

You are a data engineer at a financial services company. You manage a batch pipeline that processes daily trade settlement reports. The pipeline runs on Cloud Dataproc using PySpark jobs triggered by Cloud Composer (Airflow). Recent trades have increased by 3x, and the pipeline now frequently fails with 'OutOfMemoryError' in the executor logs. You have already increased the executor memory from 4g to 8g, but the problem persists. The cluster uses standard worker nodes (n1-standard-4) with 15 GB RAM per node. You need to make the pipeline stable and cost-efficient. What should you do?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Increase the number of executors and reduce the executor memory to 4g, then add preemptible secondary workers to lower cost.. The OutOfMemoryError persists even after increasing executor memory to 8g, indicating that the issue is not simply insufficient memory per executor but rather that the total memory across all executors is insufficient for the 3x data volume. By increasing the number of executors (parallelism) and reducing executor memory back to 4g, you distribute the data processing load across more JVMs, reducing the memory pressure per executor. Adding preemptible secondary workers lowers cost while providing the additional compute capacity needed to handle the increased data volume efficiently.

Answer

Use n1-highmem-4 instances for the cluster to get 26 GB RAM per node and increase executor memory to 12g.

Answer

Migrate the PySpark jobs to Cloud Dataflow with the Apache Beam SDK to benefit from auto-scaling.

Answer

Enable cluster autoscaling and set minimum to 5 workers, maximum to 20 workers.

Question 2

A data engineer is creating a Dataflow Flex Template for a batch pipeline that reads from BigQuery and writes to Cloud Storage. They need to pass a runtime parameter for the output bucket. How should they define this parameter?

Accepted Answer

Define the parameter in the pipeline's code and use ValueProvider. Dataflow Flex Templates require runtime parameters to be defined as `ValueProvider` objects in the pipeline code. This allows the parameter value to be supplied at job submission time via the `--parameters` flag, enabling the same template to be reused with different output buckets without recompilation.

Answer

Set an environment variable in Cloud Shell

Answer

Use the --parameters flag with the pipeline options

Answer

Hardcode the bucket name in the template

Question 3

Your Dataflow streaming pipeline is reading from Cloud Pub/Sub and writing to BigQuery. Users report occasional data duplication in the BigQuery table. You verify the pipeline uses exactly-once processing and idempotent writes. The Dataflow monitoring shows no errors, but the pipeline has occasional worker restarts. What is the most likely cause of the duplicates?

Accepted Answer

The pipeline does not set the insertId parameter in the BigQuery streaming output.. BigQuery's streaming API uses the `insertId` parameter to deduplicate records within the streaming buffer. Without a unique `insertId`, BigQuery cannot detect and discard duplicate inserts that may occur when Dataflow retries a write after a worker restart. Even with exactly-once processing in the pipeline, the BigQuery streaming endpoint itself is at-least-once, so the `insertId` is essential for deduplication.

Answer

The pipeline is using a global window with an early trigger, causing late data to be reprocessed.

Answer

The Pub/Sub subscription is configured with at-least-once delivery, causing duplicate messages.

Answer

The BigQuery table has a time-based partitioning column that is not aligned with the event timestamp.

Question 4

Your team uses Vertex AI Feature Store to serve features for online predictions. A feature value changes frequently (e.g., user session clicks). Which type of feature should you use to ensure low-latency writes and reads?

Accepted Answer

Streaming feature. A is correct because streaming features in Vertex AI Feature Store are designed for low-latency writes and reads, making them ideal for frequently changing values like user session clicks. They use an online serving infrastructure (typically backed by Bigtable) that supports real-time updates and sub-millisecond retrieval, ensuring predictions are based on the latest data without batch delays.

Answer

Batch feature

Answer

Feature view

Answer

Bigtable-backed feature

Question 5

A financial company processes transactions in real-time and requires exactly-once processing semantics. They also need to reprocess historical data for backtesting. Which Google Cloud service should they use?

Accepted Answer

Cloud Dataflow. Cloud Dataflow (D) is correct because it provides exactly-once processing semantics via its distributed snapshot mechanism (based on the MillWheel paper) and supports both real-time streaming and batch processing for historical backtesting under a unified programming model. This allows the company to reprocess historical data using the same pipeline code, ensuring consistency across real-time and batch modes.

Answer

Cloud Pub/Sub

Answer

Cloud Functions

Answer

Cloud Dataproc

Question 6

You are monitoring a Dataflow streaming job and need to track the freshness of data being processed. What metric should you alert on?

Accepted Answer

Data freshness (seconds). Data freshness (seconds) is the correct metric to alert on because it directly measures the lag between when an event occurs and when it is processed by the Dataflow pipeline. This metric, exposed as the 'system_lag' in Dataflow monitoring, indicates how up-to-date the output is relative to the input watermark. Alerting on data freshness ensures that the pipeline is meeting service-level agreements (SLAs) for real-time or near-real-time processing.

Answer

Output throughput (elements/sec)

Answer

Error count

Answer

CPU utilization

Question 7

A company processes large volumes of GPS sensor data stored in Cloud Storage. Each hour, they run an Apache Spark job that aggregates the data by geohash region. The job must be cost-effective and scale automatically. Currently, they are using a Dataproc cluster with preemptible workers. Which improvement would best reduce costs while maintaining performance?

Accepted Answer

Use Dataproc Serverless Spark. Dataproc Serverless Spark (Option D) eliminates the need to manage a cluster, automatically scaling resources to match job demand and charging only for the resources consumed during execution. This removes the overhead of preemptible worker management and idle cluster costs, directly reducing expenses while maintaining performance for the hourly aggregation job.

Answer

Use a larger Dataproc cluster with standard workers

Answer

Migrate the job to BigQuery scheduled queries

Answer

Switch to Dataflow batch pipeline with Apache Beam

Question 8

A data engineer runs this Dataflow template to load CSV files from Cloud Storage into BigQuery. The job fails with a 'File pattern not matching any files' error. What is the most likely cause?

Accepted Answer

The CSV files are stored in a subdirectory that is not matched by the pattern. The error 'File pattern not matching any files' indicates that the file pattern specified in the Dataflow template does not resolve to any existing objects in Cloud Storage. If the CSV files are stored in a subdirectory (e.g., gs://bucket/subdir/*.csv) but the pattern only references the root (e.g., gs://bucket/*.csv), no files will be matched. This is the most likely cause because the pattern must explicitly include the subdirectory path.

Answer

The bucket name is incorrectly spelled

Answer

The template has a bug

Answer

The output table does not exist

Question 9

A team configured a garbage collection rule on a Cloud Bigtable column family with max_age of 100 seconds. After 2 minutes, they notice that data older than 100 seconds is still present. What is the most likely reason?

Accepted Answer

Garbage collection runs only periodically (e.g., once per day). Cloud Bigtable garbage collection (GC) is not applied in real time; it runs as a background process that typically executes once per day. Even though the max_age rule is set to 100 seconds, the actual deletion of expired data occurs only during the next scheduled GC cycle, which may not happen for up to 24 hours. Therefore, observing data older than 100 seconds after only 2 minutes is expected behavior.

Answer

They need to apply the rule using a different API

Answer

The max_age must be at least 1 hour

Answer

The rule is applied only to new data, not existing data

Question 10

A gaming company uses Pub/Sub to ingest player events and Dataflow for real-time analytics. They notice that the Pub/Sub subscription backlog is growing despite the Dataflow pipeline running continuously. The pipeline has a 1-hour window for aggregations. What is the most effective way to reduce the backlog?

Accepted Answer

Increase the Dataflow pipeline's worker count via autoscaling.. Increasing the Dataflow pipeline's worker count via autoscaling directly addresses the backlog by adding more parallel processing capacity to consume messages from the Pub/Sub subscription faster. Since the pipeline is continuously running but the backlog grows, the bottleneck is processing throughput, not pipeline availability. Autoscaling allows Dataflow to dynamically allocate more workers based on the backlog size, matching consumption rate to the incoming message rate.

Answer

Use a push subscription instead of pull.

Answer

Decrease the window duration to 10 minutes.

Answer

Enable Pub/Sub topic retention.

Question 11

A data pipeline uses Cloud Composer (Airflow) to orchestrate Dataproc jobs. Each job submits a Spark application that reads from BigQuery and writes to Cloud Storage. The pipeline runs nightly and takes 6 hours. Management wants to reduce costs. Which approach is most effective?

Accepted Answer

Use preemptible VMs for the Dataproc cluster. Preemptible VMs are significantly cheaper (up to 80% discount) than standard VMs and are ideal for fault-tolerant, batch workloads like nightly Dataproc jobs. Since the pipeline runs nightly and takes 6 hours, it can tolerate the occasional preemption of worker nodes by using Spark's built-in resilience (e.g., task retries). This directly reduces compute cost without sacrificing completion, assuming the cluster is configured with enough preemptible workers to handle the workload.

Answer

Switch to Cloud Dataproc billing per second instead of per minute

Answer

Increase the memory of the driver node to improve performance

Answer

Upgrade the Cloud Storage class from Standard to Nearline

Question 12

A company is designing a real-time clickstream analytics pipeline using Pub/Sub and Dataflow. The pipeline must handle late-arriving data (up to 1 hour) and ensure exactly-once processing. Which Dataflow feature should be configured to handle late data correctly?

Accepted Answer

Configure the trigger with allowed lateness of 1 hour.. Dataflow's allowed lateness feature explicitly controls how long the pipeline waits for late-arriving data before closing a window. By setting allowed lateness to 1 hour, the watermark is held back, and late data within that period is still processed with exactly-once semantics. This directly addresses the requirement for handling late data up to 1 hour while ensuring no duplicates or data loss.

Answer

Use fixed windows with a 1-hour period and enable data discarding.

Answer

Use session windows with a gap duration of 1 hour.

Answer

Set the watermark estimate to 1 hour.

Question 13

A streaming Dataflow pipeline ingests events from Cloud Pub/Sub and writes to BigQuery. The event schema evolves occasionally (new columns added). The pipeline fails when new columns appear. What is the best long-term solution?

Accepted Answer

Configure the BigQuery sink to use stored 'dynamic' schema by setting create_disposition to CREATE_NEVER and writing to a temporary table with schema auto-detection. It leverages BigQuery's schema auto-detection with a temporary table to handle schema evolution dynamically. By setting create_disposition to CREATE_NEVER, the pipeline writes to a table that already exists, while the temporary table with auto-detection allows the pipeline to infer new columns from the incoming data. This approach avoids pipeline failures when new columns appear, as the sink can adapt without manual intervention or pipeline restarts.

Answer

Stop the pipeline and update the BigQuery schema manually whenever a new column appears

Answer

Switch to Dataproc to process the data with Spark and write to BigQuery using the Avro format

Answer

Use a Cloud Function to transform the data and add null columns for missing fields

Question 14

A Dataflow streaming pipeline that uses global windows and triggers every 5 seconds is experiencing increasing lag and high system latency. The pipeline reads from Pub/Sub, transforms data with a ParDo, and writes to BigQuery. Which action is most likely to reduce lag?

Accepted Answer

Replace the global window with a sliding window of 1 minute.. B is correct because sliding windows of 1 minute allow the pipeline to process data in overlapping fixed-size windows, which can reduce the buildup of data in memory compared to global windows. Global windows with frequent triggers (every 5 seconds) can cause unbounded state growth and high latency as the pipeline must maintain state for all elements until the trigger fires, whereas sliding windows naturally bound the data per window and enable more efficient watermark and trigger management in Dataflow.

Answer

Use a session window to group related events.

Answer

Change the trigger to processing time instead of event time.

Answer

Increase the number of workers manually.

Question 15

A company uses Cloud Dataproc to run Spark ML jobs. The jobs are memory-intensive and often fail with OutOfMemory errors. Which action would most effectively reduce memory pressure without changing the Spark code?

Accepted Answer

Increase the number of worker nodes and reduce the number of cores per worker.. Increasing the number of worker nodes while reducing the number of cores per worker reduces memory pressure by distributing the workload across more JVMs, each with a smaller heap. This lowers the per-executor memory requirement and reduces the risk of OutOfMemory errors without modifying Spark code. In Cloud Dataproc, this approach directly addresses memory contention by giving each executor fewer tasks to process concurrently.

Answer

Increase the master node's memory.

Answer

Increase the number of Spark partitions.

Answer

Use preemptible VMs for workers.