PDE Practice Test 21 — 15 Questions

Question 1

Your Dataflow streaming pipeline is processing financial transactions and writing results to BigQuery. You need to monitor the pipeline for data freshness (end-to-end latency) and alert if it exceeds 5 minutes. The pipeline uses fixed windows of 1 minute. Which metrics should you use for alerting?

Accepted Answer

System Lag metric from Dataflow monitoring.. Dataflow's 'System Lag' metric measures the difference between event time and processing time, indicating how far behind the pipeline is. For windowed pipelines, this reflects overall latency. Option A (Element Count) shows throughput, not latency. Option C (Data Freshness) is a BigQuery-specific metric for table currency. Option D (Worker Threads Utilization) relates to parallelism.

Answer

Data Freshness metric from BigQuery monitoring.

Answer

Element Count metric from Dataflow monitoring.

Answer

Worker Threads Utilization metric from Dataflow monitoring.

Question 2

Your company uses Cloud Dataflow to process streaming data from Pub/Sub. The pipeline occasionally fails with a 'worker terminated unexpectedly' error. What is the most likely cause of this error?

Accepted Answer

Insufficient memory per worker causing OOM errors. The 'worker terminated unexpectedly' error in Cloud Dataflow typically indicates that a worker process ran out of memory (OOM) and was killed by the operating system. This occurs when the pipeline's memory requirements exceed the configured worker machine type's memory capacity, often due to large windowing accumulations, skewed data, or inefficient state handling.

Answer

Incorrect VPC firewall rules blocking internal communication

Answer

Staging location bucket lacks write permissions

Answer

Pub/Sub subscription throughput quota exceeded

Question 3

A team is setting up a Dataflow pipeline for a time-sensitive ETL job that must complete within a specific time window. Which monitoring metric should they use to determine if the pipeline is on track to finish on time?

Accepted Answer

The job's estimated time to completion shown in the Dataflow monitoring interface.. Option D is correct because the Dataflow monitoring interface provides an estimated time to completion for the pipeline, which is the most direct metric for determining if the job will finish within the required time window. This estimate is calculated based on current throughput, backlog, and resource utilization, making it the appropriate choice for time-sensitive ETL jobs. Other metrics like system lag or element counts do not directly predict job completion time.

Answer

The number of failed elements and retries.

Answer

The system lag metric, which measures the time between event occurrence and processing.

Answer

The number of elements processed in the current window.

Question 4

You are optimizing a BigQuery query that runs on a large table (hundreds of TB). The table is partitioned by date and frequently queried with filters on a specific customer_id column and date range. Queries are slow even after partitioning. Which optimization should you apply?

Accepted Answer

Columnar clustering on customer_id. Clustering on customer_id within the partition improves query performance because BigQuery can prune blocks based on clustered columns. Partitioning alone doesn't help with non-date filters. Materialized views may help pre-aggregated queries but not ad-hoc customer_id filters. Denormalization is not an optimization. Increasing slots is expensive and doesn't address data structure.

Answer

Increase the number of BigQuery slots

Answer

Create materialized views for each customer

Answer

Denormalize the table to reduce joins

Question 5

Your Cloud Dataflow pipeline is failing due to a 'Permission denied' error when writing to a BigQuery table. The error persists even though the service account has bigquery.dataEditor role. What is the most likely missing permission?

Accepted Answer

bigquery.tables.create on the dataset. Option A is correct because Dataflow needs bigquery.tables.create if the table doesn't exist. Option B is wrong because read permissions are not needed for writing. Option C is wrong because bucket permissions are for staging, not writing to BigQuery. Option D is wrong because pub/sub roles are not needed.

Answer

pubsub.topics.publish on a notification topic

Answer

storage.objects.create on the staging bucket

Answer

bigquery.tables.get on the table

Question 6

Your team is running a Dataflow streaming pipeline that reads from Pub/Sub, transforms data, and writes to BigQuery. You notice that the pipeline's backlog is growing and the processing latency has increased from seconds to minutes. You need to diagnose and resolve the issue. Which TWO actions should you take? (Choose two.)

Accepted Answer

Increase the maximum number of workers in the pipeline's autoscaling configuration to allow more compute resources.. Option D is correct because increasing the maximum number of workers in the autoscaling configuration allows Dataflow to scale out horizontally, adding more compute resources to handle the increased backlog and reduce processing latency. Dataflow's autoscaling algorithm uses metrics like backlog bytes and CPU utilization to decide when to add workers, but it is capped by the max workers setting. Raising this cap enables the pipeline to allocate more VMs, thus processing more messages per second and reducing the backlog.

Answer

Stop the pipeline, increase the number of workers in the streaming engine configuration, and restart it.

Answer

Increase the batch size in the WriteToBigQuery transform to reduce I/O operations.

Answer

Configure a dead-letter queue in Cloud Storage for failed messages to reduce reprocessing load.

Question 7

An organization uses Cloud Dataproc to run Spark jobs that process sensitive data. They need to ensure data is encrypted at rest and that only specific service accounts can access the data on cluster disks. What should they do?

Accepted Answer

Use customer-managed encryption keys (CMEK) for the cluster's persistent disks and assign a dedicated service account to the cluster with minimal IAM roles.. Option D is correct because using customer-managed encryption keys (CMEK) allows the organization to control and manage the encryption keys for persistent disks attached to the Dataproc cluster, ensuring data at rest is encrypted. Assigning a dedicated service account with minimal IAM roles ensures that only that service account can access the data on the cluster disks, following the principle of least privilege.

Answer

Rely on the default encryption at rest and use VPC Service Controls to limit data exfiltration.

Answer

Use customer-supplied encryption keys (CSEK) and write a startup script to mount encrypted disks.

Answer

Enable encryption at rest using Google-managed encryption keys and grant all users the Dataproc Editor role.

Question 8

A financial services firm uses Cloud Pub/Sub to ingest real-time market data. The data is processed by a Cloud Dataflow streaming pipeline that aggregates trades per symbol and writes to BigQuery. The pipeline currently uses a single global window with a trigger that fires every minute. The firm now needs to support late data up to 5 minutes and also wants to reduce the number of writes to BigQuery to avoid hitting the table limit of 1,500 inserts per second. The current pipeline writes every minute, which is acceptable for inserts per second, but after adding late data handling, the number of writes doubles. How can you redesign the pipeline to handle late data while keeping write volume low?

Accepted Answer

Use session windows with a gap duration of 5 minutes and a count-based trigger that fires after accumulating 1000 elements. Option D is correct because session windows naturally group events into bursts of activity separated by a gap duration (5 minutes), which reduces the number of writes by accumulating many trades per symbol before emitting a pane. Adding a count-based trigger that fires after 1000 elements further limits write frequency, keeping the insert rate well below BigQuery's 1,500 per second limit while still allowing late data up to the gap duration. This design handles late data implicitly within the session gap and avoids the write amplification seen with fixed windows and frequent triggers.

Answer

Use fixed windows of 5 minutes with allowed lateness 5 minutes and trigger every 30 seconds

Answer

Increase the global window duration to 10 minutes and keep the same trigger

Answer

Discard all late data and keep the current windowing

Question 9

Your company is evaluating managed messaging services for a new event-driven application. The application requires pub/sub semantics, high throughput (millions of messages per second), and integration with Google Cloud services like Cloud Functions and Dataflow. Which TWO services should you consider? (Choose two.)

Accepted Answer

Cloud Pub/Sub. Cloud Pub/Sub (A) is the correct choice because it provides fully managed, highly scalable pub/sub messaging with exactly-once delivery semantics and support for millions of messages per second. It integrates natively with Cloud Functions and Dataflow, making it ideal for event-driven architectures requiring high throughput and decoupled communication.

Answer

Cloud Scheduler

Answer

Cloud Functions

Answer

Cloud Tasks

Question 10

A company runs a Dataflow pipeline that reads from Pub/Sub, aggregates events in a 10-minute fixed window, and writes to BigQuery. Recently, the pipeline has been failing with 'high uncommitted bytes' errors during periods of high traffic. What is the most likely cause and recommended action?

Accepted Answer

Use a global window with a trigger that fires early based on element count to reduce the number of open windows.. The 'high uncommitted bytes' error in Dataflow occurs when the system holds too much data in memory across many open windows, exceeding the default 200 MB limit. Using a global window with an early trigger based on element count reduces the number of simultaneous open windows and allows data to be committed more frequently, preventing memory pressure. This approach is recommended over reducing window size or scaling workers because the root cause is window fan-out, not throughput or parallelism.

Answer

Reduce the window size from 10 minutes to 1 minute to decrease the amount of data per window.

Answer

Increase the number of worker machines to handle higher throughput.

Answer

Set a maximum number of workers and use a Pub/Sub flow control setting to limit incoming messages.

Question 11

You are implementing a data pipeline that reads from Cloud Storage (parquet files), transforms data with Cloud Dataflow, and writes to BigQuery. The pipeline runs on a batch schedule every hour. You notice that the Dataflow job takes 10 minutes, but the overall pipeline latency is 15 minutes due to file availability and scheduling. The business requires latency under 5 minutes. Which change should you make?

Accepted Answer

Switch to streaming pipeline with .watchForNewFiles() and process files as they arrive. The root cause of the latency is file availability and scheduling delay, not the processing time. Switching to a streaming pipeline with `.watchForNewFiles()` (or the equivalent `FileIO.match().continuously()`) allows Dataflow to process files as soon as they arrive in Cloud Storage, eliminating the batch scheduling wait and reducing overall latency to near the processing time.

Answer

Batch the hourly data into a single larger hourly run

Answer

Use a larger machine type for the Dataflow workers

Answer

Increase the number of workers and use smaller input files

Question 12

A Dataflow batch job fails consistently with the error shown. The job uses a custom container image and runs in a VPC with a private IP. What should the engineer do to resolve the issue?

Accepted Answer

Verify that the VPC has Private Google Access enabled and that Cloud NAT is configured for outbound internet access if needed.. The error indicates that the Dataflow batch job cannot access required resources (e.g., container image, dependencies) because the VPC with private IPs lacks outbound internet connectivity. Option B is correct because enabling Private Google Access allows the VMs to reach Google APIs (like Container Registry) via the Google network, and Cloud NAT provides outbound internet access for non-Google APIs or external dependencies. Without these, the job fails to pull the custom container image or download necessary artifacts.

Answer

Request a CPU quota increase in the region.

Answer

Rebuild the custom container image and upload it to Container Registry.

Answer

Check that the custom image is based on the latest Dataflow SDK version.

Question 13

You are a data engineer at a financial services company. You manage a batch pipeline that processes daily trade settlement reports. The pipeline runs on Cloud Dataproc using PySpark jobs triggered by Cloud Composer (Airflow). Recent trades have increased by 3x, and the pipeline now frequently fails with 'OutOfMemoryError' in the executor logs. You have already increased the executor memory from 4g to 8g, but the problem persists. The cluster uses standard worker nodes (n1-standard-4) with 15 GB RAM per node. You need to make the pipeline stable and cost-efficient. What should you do?

Accepted Answer

Increase the number of executors and reduce the executor memory to 4g, then add preemptible secondary workers to lower cost.. Option C is correct because the OutOfMemoryError persists even after increasing executor memory to 8g, indicating that the issue is not simply insufficient memory per executor but rather that the total memory across all executors is insufficient for the 3x data volume. By increasing the number of executors (parallelism) and reducing executor memory back to 4g, you distribute the data processing load across more JVMs, reducing the memory pressure per executor. Adding preemptible secondary workers lowers cost while providing the additional compute capacity needed to handle the increased data volume efficiently.

Answer

Use n1-highmem-4 instances for the cluster to get 26 GB RAM per node and increase executor memory to 12g.

Answer

Migrate the PySpark jobs to Cloud Dataflow with the Apache Beam SDK to benefit from auto-scaling.

Answer

Enable cluster autoscaling and set minimum to 5 workers, maximum to 20 workers.

Question 14

A data pipeline uses Cloud Composer (Airflow) to orchestrate Dataproc jobs. Each job submits a Spark application that reads from BigQuery and writes to Cloud Storage. The pipeline runs nightly and takes 6 hours. Management wants to reduce costs. Which approach is most effective?

Accepted Answer

Use preemptible VMs for the Dataproc cluster. Preemptible VMs are significantly cheaper (up to 80% discount) than standard VMs and are ideal for fault-tolerant, batch workloads like nightly Dataproc jobs. Since the pipeline runs nightly and takes 6 hours, it can tolerate the occasional preemption of worker nodes by using Spark's built-in resilience (e.g., task retries). This directly reduces compute cost without sacrificing completion, assuming the cluster is configured with enough preemptible workers to handle the workload.

Answer

Switch to Cloud Dataproc billing per second instead of per minute

Answer

Increase the memory of the driver node to improve performance

Answer

Upgrade the Cloud Storage class from Standard to Nearline

Question 15

A company uses Dataproc for transient clusters. Which TWO actions can reduce costs?

Accepted Answer

Set cluster autoscaling to minimize idle resources. Option B is correct because Dataproc cluster autoscaling automatically adjusts the number of worker nodes based on the YARN memory and CPU utilization metrics. By scaling down during idle periods, you avoid paying for unused compute capacity, directly reducing costs for transient clusters that have variable workloads.

Answer

Increase master node size

Answer

Use standard VMs for all nodes

Answer

Use persistent clusters to avoid creation overhead