PDE Building and operationalizing data processing systems Practice Test 6 — 15 Questions

Question 1

A Dataflow batch job fails consistently with the error shown. The job uses a custom container image and runs in a VPC with a private IP. What should the engineer do to resolve the issue?

Accepted Answer

Verify that the VPC has Private Google Access enabled and that Cloud NAT is configured for outbound internet access if needed.. The error indicates that the Dataflow batch job cannot access required resources (e.g., container image, dependencies) because the VPC with private IPs lacks outbound internet connectivity. Option B is correct because enabling Private Google Access allows the VMs to reach Google APIs (like Container Registry) via the Google network, and Cloud NAT provides outbound internet access for non-Google APIs or external dependencies. Without these, the job fails to pull the custom container image or download necessary artifacts.

Answer

Request a CPU quota increase in the region.

Answer

Rebuild the custom container image and upload it to Container Registry.

Answer

Check that the custom image is based on the latest Dataflow SDK version.

Question 2

Based on the exhibit, what is the most likely cause of duplicate rows despite using the same event_id as insertId?

Accepted Answer

BigQuery's streaming buffer deduplication is best-effort and may not catch duplicates within a short time window.. BigQuery's streaming buffer uses best-effort deduplication based on the `insertId` field. When multiple rows are inserted with the same `event_id` mapped to `insertId` within a short time window (typically up to a few minutes), the deduplication mechanism may fail to remove all duplicates, especially under high throughput or network retries. This is a documented limitation of BigQuery streaming, not a guarantee of exactly-once semantics.

Answer

The Dataflow pipeline is retrying inserts due to network errors, and the same event_id is not being used in retries.

Answer

The pipeline is writing more than 100,000 rows per second, exceeding BigQuery's streaming quota.

Answer

The table is partitioned by timestamp, so BigQuery cannot deduplicate across partitions.

Question 3

A data engineer notices that Spark jobs on the Dataproc cluster shown often fail with executor lost errors. What is the most likely reason?

Accepted Answer

All 10 workers are preemptible and can be reclaimed by Compute Engine at any time.. Preemptible VMs in Google Compute Engine can be terminated at any time due to resource contention or other factors, with only 30 seconds notice. If all 10 worker nodes are preemptible, Spark executors running on them will be frequently lost, causing job failures. This is the most direct cause of 'executor lost' errors in a Dataproc cluster.

Answer

The master node has only 4 vCPUs, which may be insufficient for job coordination.

Answer

The cluster is in a single zone, so a zone failure could cause all workers to shut down.

Answer

Autoscaling is enabled and scaling down is causing workers to be removed during job execution.

Question 4

Your company uses Cloud Dataflow to process streaming data from Pub/Sub. The pipeline occasionally fails with a 'worker terminated unexpectedly' error. What is the most likely cause of this error?

Accepted Answer

Insufficient memory per worker causing OOM errors. The 'worker terminated unexpectedly' error in Cloud Dataflow typically indicates that a worker process ran out of memory (OOM) and was killed by the operating system. This occurs when the pipeline's memory requirements exceed the configured worker machine type's memory capacity, often due to large windowing accumulations, skewed data, or inefficient state handling.

Answer

Incorrect VPC firewall rules blocking internal communication

Answer

Staging location bucket lacks write permissions

Answer

Pub/Sub subscription throughput quota exceeded

Question 5

A data pipeline uses Cloud Composer (Airflow) to orchestrate Dataproc jobs. Each job submits a Spark application that reads from BigQuery and writes to Cloud Storage. The pipeline runs nightly and takes 6 hours. Management wants to reduce costs. Which approach is most effective?

Accepted Answer

Use preemptible VMs for the Dataproc cluster. Preemptible VMs are significantly cheaper (up to 80% discount) than standard VMs and are ideal for fault-tolerant, batch workloads like nightly Dataproc jobs. Since the pipeline runs nightly and takes 6 hours, it can tolerate the occasional preemption of worker nodes by using Spark's built-in resilience (e.g., task retries). This directly reduces compute cost without sacrificing completion, assuming the cluster is configured with enough preemptible workers to handle the workload.

Answer

Switch to Cloud Dataproc billing per second instead of per minute

Answer

Increase the memory of the driver node to improve performance

Answer

Upgrade the Cloud Storage class from Standard to Nearline

Question 6

You are designing a streaming pipeline using Cloud Dataflow with exactly-once semantics. The source is Pub/Sub and the sink is Cloud Bigtable. The pipeline must handle late data up to 10 minutes. You need to minimize cost while maintaining correctness. Which configuration should you use?

Accepted Answer

Global window with allowed lateness 10 minutes and trigger=afterWatermark with early firings. Option C is correct because a global window with an after-watermark trigger and early firings is the most cost-effective way to handle unbounded data from Pub/Sub with exactly-once semantics, while allowing up to 10 minutes of lateness. Fixed or sliding windows would create many small window states, increasing Bigtable write costs and shuffle overhead. The global window minimizes state and processing, and the trigger ensures results are emitted promptly without accumulating panes.

Answer

Fixed windows of 1 minute with allowed lateness 10 minutes and accumulating fired panes

Answer

Sliding windows of 1 minute with allowed lateness 10 minutes and accumulating fired panes

Answer

Session windows of 5 minutes with gap duration 1 minute and discarding fired panes

Question 7

Your team uses Cloud Dataproc to run a Spark ML training job. The job is failing with an error: 'Container killed by YARN for exceeding memory limits.' What should you do to fix this?

Accepted Answer

Increase the spark.executor.memory property. The error 'Container killed by YARN for exceeding memory limits' indicates that the Spark executor process is using more memory than the YARN container allows. Increasing `spark.executor.memory` allocates a larger YARN container for each executor, providing the necessary headroom for the Spark application's memory demands, including overhead for off-heap memory and JVM internals.

Answer

Use preemptible VMs for faster execution

Answer

Increase the number of worker nodes

Answer

Enable the external shuffle service

Question 8

A company has a Cloud Functions function that triggers on new files in Cloud Storage and writes a message to Pub/Sub for downstream processing. Recently, the function has been timing out after 60 seconds. The downstream processing is critical. What is the best solution?

Accepted Answer

Increase function timeout to 540 seconds and delegate heavy processing to Cloud Dataflow. Option D is correct because Cloud Functions has a maximum timeout of 540 seconds (9 minutes) for HTTP-triggered functions, and by increasing the timeout you allow the function to complete its work. Delegating heavy processing to Cloud Dataflow offloads the computationally intensive tasks, preventing future timeouts and ensuring scalable, reliable downstream processing for critical workloads.

Answer

Replace Cloud Functions with a Cloud Run job that has longer timeout

Answer

Increase the function memory to 2 GB to speed up execution

Answer

Reduce the function timeout to 30 seconds to force faster execution

Question 9

You are implementing a data pipeline that reads from Cloud Storage (parquet files), transforms data with Cloud Dataflow, and writes to BigQuery. The pipeline runs on a batch schedule every hour. You notice that the Dataflow job takes 10 minutes, but the overall pipeline latency is 15 minutes due to file availability and scheduling. The business requires latency under 5 minutes. Which change should you make?

Accepted Answer

Switch to streaming pipeline with .watchForNewFiles() and process files as they arrive. The root cause of the latency is file availability and scheduling delay, not the processing time. Switching to a streaming pipeline with `.watchForNewFiles()` (or the equivalent `FileIO.match().continuously()`) allows Dataflow to process files as soon as they arrive in Cloud Storage, eliminating the batch scheduling wait and reducing overall latency to near the processing time.

Answer

Batch the hourly data into a single larger hourly run

Answer

Use a larger machine type for the Dataflow workers

Answer

Increase the number of workers and use smaller input files

Question 10

Your Cloud Dataflow pipeline is failing due to a 'Permission denied' error when writing to a BigQuery table. The error persists even though the service account has bigquery.dataEditor role. What is the most likely missing permission?

Accepted Answer

bigquery.tables.create on the dataset. Option A is correct because Dataflow needs bigquery.tables.create if the table doesn't exist. Option B is wrong because read permissions are not needed for writing. Option C is wrong because bucket permissions are for staging, not writing to BigQuery. Option D is wrong because pub/sub roles are not needed.

Answer

pubsub.topics.publish on a notification topic

Answer

storage.objects.create on the staging bucket

Answer

bigquery.tables.get on the table

Question 11

A company uses Cloud Composer (Airflow) to orchestrate a daily batch job that runs a custom Python script on a Compute Engine instance. The process is slow because the instance takes 2 minutes to boot. How can you reduce the total runtime?

Accepted Answer

Create a custom image with the script and dependencies pre-installed. Option C is correct because creating a custom image with the script and dependencies pre-installed eliminates the need to install packages or configure the environment at boot time. In Cloud Composer, when a Compute Engine instance is provisioned via a BashOperator or SSHOperator, the boot process includes OS initialization and package installation. A custom image bypasses these steps, reducing boot time from minutes to seconds, directly addressing the 2-minute boot delay.

Answer

Switch to Dataproc Serverless to avoid VM boot time

Answer

Use a larger machine type for faster provisioning

Answer

Use a GPU-accelerated instance to speed up the script

Question 12

You are designing a data pipeline that must process sensitive customer data with strict access controls. The data is ingested via Cloud Pub/Sub, processed by Cloud Dataflow, and stored in BigQuery. The security team requires that data is encrypted at rest and in transit, and that access is limited to specific service accounts. Which implementation strategy meets all requirements?

Accepted Answer

Use CMEK for Pub/Sub, Dataflow, and BigQuery, and VPC-SC with per-service service accounts. Option D is correct because it combines Customer-Managed Encryption Keys (CMEK) for all three services (Pub/Sub, Dataflow, BigQuery) to ensure data is encrypted at rest with keys controlled by the customer, and uses VPC Service Controls (VPC-SC) with per-service service accounts to enforce network perimeter security and least-privilege access. This meets the requirements for encryption at rest and in transit (CMEK also covers in-transit encryption via TLS) and strict access controls via service accounts and VPC-SC.

Answer

Use Cloud KMS for BigQuery only; leave Dataflow with default encryption

Answer

Use VPC Service Controls and Cloud Armor for network security

Answer

Use default Google-managed encryption keys and IAM roles only

Question 13

Which TWO are best practices for managing a Cloud Dataflow pipeline in production?

Accepted Answer

Set up Cloud Monitoring alerts based on Dataflow job metrics. Option C is correct because Cloud Monitoring alerts on Dataflow job metrics (e.g., system lag, watermark delay, or element count) enable proactive detection of pipeline health issues such as backpressure or stuck workers. This is a best practice for production pipelines to ensure reliability and timely intervention.

Answer

Always use batch mode for streaming data to reduce cost

Answer

Disable autoscaling to keep compute costs predictable

Answer

Restart the pipeline when code changes are needed

Question 14

Which THREE actions reduce the cost of a Cloud Composer environment?

Accepted Answer

Delete old and unused DAG files to reduce scheduler load. Option A is correct because deleting old and unused DAG files reduces the number of DAGs the scheduler must parse and evaluate. The Cloud Composer scheduler scans the DAG folder every 30 seconds by default; fewer DAG files mean lower CPU and memory consumption, directly reducing the cost of the environment's compute resources.

Answer

Use standard network tier instead of premium

Answer

Increase the number of schedulers for higher throughput

Question 15

Which TWO are valid approaches to handle late-arriving data in a Cloud Dataflow streaming pipeline?

Accepted Answer

Set allowed lateness on the window. Option B is correct because setting allowed lateness on a window in Cloud Dataflow allows the pipeline to wait for late-arriving data within a specified duration after the watermark passes the window end. This is a standard mechanism to handle out-of-order or delayed events without discarding them, ensuring completeness of windowed aggregations.

Answer

Change to processing time windows instead of event time windows

Answer

Discard any events that arrive after the window closes

Answer

Use a trigger that fires every second

PDE Building and operationalizing data processing systems Practice Test 6 — 15 Questions

Exhibit

PDE Building and operationalizing data processing systems Practice Test 6 — 15 Questions

Exhibit