PDE Practice Exam 59 — 20 Questions

Question 1

You are designing a Cloud Composer workflow that loads data from Cloud Storage into BigQuery, runs a Dataflow job to transform the data, and then triggers a Dataproc Spark job. After each step, you need to conditionally branch based on success or failure. Which Airflow feature allows you to pass messages between tasks to enable dynamic branching?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

XComs. XComs (cross-communications) in Airflow allow tasks to exchange small amounts of data, such as status or metadata. This data can be used by BranchPythonOperator to conditionally choose downstream tasks.

Answer

Sensors

Answer

TaskFlow API

Answer

DAG dependencies

Question 2

Match each data storage term to its characteristic.

Accepted Answer

Cloud Storage: Object storage for unstructured data. Cloud Storage is object storage for unstructured data; Cloud SQL is a managed relational database; Bigtable is a NoSQL wide-column database. Common confusions involve swapping these definitions.

Answer

Cloud Storage: Managed relational database for MySQL, PostgreSQL, and SQL Server

Answer

Bigtable: Object storage for unstructured data

Question 3

A data engineer is designing a Cloud Bigtable schema for high-volume time-series data. Which TWO practices should they follow to avoid performance issues?

Accepted Answer

Use a hashed prefix in the row key to distribute writes. Using a hashed prefix to avoid hot-spotting and grouping related columns into column families are recommended. Timestamp-first keys cause hot-spotting. Single column family for all data is inefficient. Large number of column families also adds overhead.

Answer

Place the timestamp as the first component of the row key

Answer

Create as many column families as possible

Answer

Store all columns in a single column family

Question 4

Your company is building a real-time anomaly detection system for financial transactions. The system must process streams of transactions and flag anomalies within seconds. The volume is moderate (5000 transactions per second). You want a fully managed solution that integrates with BigQuery for historical analysis. Which service should you use for stream processing?

Accepted Answer

Cloud Dataflow. Cloud Dataflow is a fully managed service ideal for real-time stream processing with low latency. It can read from Pub/Sub, perform transformations (e.g., anomaly detection), and write to BigQuery for historical analysis. Dataproc requires cluster management; Data Fusion is batch-oriented; Pub/Sub alone does not process data.

Answer

Cloud Pub/Sub with push subscriptions

Answer

Cloud Dataproc with Spark Streaming

Answer

Cloud Data Fusion

Question 5

A financial services company deploys a regression model to predict loan default risk. The model is served using Vertex AI Endpoints with autoscaling. After deployment, latency increases significantly during peak hours, causing timeouts. The model uses scikit-learn and has a large feature set. Which action should the team take to reduce latency while maintaining prediction accuracy?

Accepted Answer

Apply feature selection to reduce the number of input features.. The latency spike is caused by the large feature set, which increases the time for preprocessing and inference in the scikit-learn model. Reducing the number of input features via feature selection directly decreases the computational load per request, lowering latency without sacrificing accuracy if the selected features retain predictive power. This addresses the root cause, unlike scaling or resource changes that only mask the symptom.

Answer

Switch to batch prediction for all requests.

Answer

Increase the minimum number of replicas in the endpoint to handle peak load.

Answer

Increase the memory allocation for the serving container.

Question 6

A developer wants to create a BigQuery table that automatically expires data older than 30 days to reduce storage costs. Which table design feature should be used?

Accepted Answer

Partitioned table with partition expiration. Partitioned tables with a partition expiration allow automatic deletion of partitions. Clustering does not affect data expiration. Materialized views are for pre-computed aggregates, not data lifecycle. Authorized views control access.

Answer

Authorized view

Answer

Clustered table

Answer

Materialized view

Question 7

A company trains a custom model using TensorFlow and wants to deploy it to Vertex AI for low-latency predictions. The model is large (2 GB). Which deployment option should they choose?

Accepted Answer

Deploy to Vertex AI Endpoint with a custom container. Deploying a large (2 GB) model to Vertex AI Endpoint with a custom container allows you to package the model, its dependencies, and a serving framework (e.g., TensorFlow Serving) into a Docker image. This approach supports low-latency predictions by keeping the model loaded in memory across requests, and it can scale to handle real-time inference traffic, unlike batch or serverless options that have cold-start or size limitations.

Answer

Use Vertex AI Batch Prediction job

Answer

Deploy as a Cloud Function

Answer

Deploy to Cloud Run with minimum instances

Question 8

Your company uses Cloud Dataflow to process streaming data from Pub/Sub. The pipeline occasionally fails with a 'worker terminated unexpectedly' error. What is the most likely cause of this error?

Accepted Answer

Insufficient memory per worker causing OOM errors. The 'worker terminated unexpectedly' error in Cloud Dataflow typically indicates that a worker process ran out of memory (OOM) and was killed by the operating system. This occurs when the pipeline's memory requirements exceed the configured worker machine type's memory capacity, often due to large windowing accumulations, skewed data, or inefficient state handling.

Answer

Incorrect VPC firewall rules blocking internal communication

Answer

Staging location bucket lacks write permissions

Answer

Pub/Sub subscription throughput quota exceeded

Question 9

A company is developing a streaming Dataflow pipeline to process real-time sensor data. To ensure data quality, the team wants to detect malformed records and late data. Which two practices should they implement? (Choose two.)

Accepted Answer

Configure a dead letter queue for unprocessable records.. A dead letter queue (DLQ) is a standard pattern in streaming pipelines for isolating malformed or unprocessable records without blocking the main data flow. In Dataflow, this is typically implemented by writing bad records to a separate output (e.g., a Pub/Sub topic or Cloud Storage bucket) for later analysis or reprocessing. Option D is correct because the Dataflow system lag metric in Cloud Monitoring measures the time between when data enters the pipeline and when it is processed, making it an effective way to detect late data and trigger alerts for SLA violations.

Answer

Use Beam’s PAssert to validate each element in the pipeline.

Answer

Enable Dataflow’s built-in schema validation on the PCollection.

Answer

Run a separate batch pipeline to re-process data for validation.

Question 10

A data pipeline uses Dataflow to read from Pub/Sub, window messages into 1-minute fixed windows, and write to BigQuery. The pipeline occasionally has late-arriving data. How should they configure the pipeline to allow late data up to 5 minutes and then trigger a final pane?

Accepted Answer

withAllowedLateness(Duration.standardMinutes(5)).triggering(AfterWatermark.pastEndOfWindow().withLateFirings(AfterPane.elementCountAtLeast(1))). In Beam, allowed lateness and triggering combine to handle late data.

Answer

withAllowedLateness(Duration.standardMinutes(5)).triggering(Repeatedly.forever(AfterPane.elementCountAtLeast(1)))

Answer

triggering(AfterWatermark.pastEndOfWindow()).withAllowedLateness(Duration.standardMinutes(5))

Answer

withAllowedLateness(Duration.standardMinutes(5)).accumulatingFiredPanes()

Question 11

A company processes financial transactions using Cloud Dataflow. They need to ensure that late-arriving data is handled correctly for fraud detection. The pipeline uses event time processing. Which approach should they use to handle late data?

Accepted Answer

Fixed windows with allowed lateness. Fixed windows with allowed lateness are the standard approach in Cloud Dataflow (Apache Beam) for handling late-arriving data in event-time processing. By specifying an allowed lateness duration, the pipeline retains the window state for that period, allowing late events to be correctly assigned to their original window and triggering recomputation of results. This ensures fraud detection pipelines can account for delayed transactions without missing or misordering data.

Answer

Sliding windows with early firing

Answer

Session windows with gap duration

Answer

Global windows with triggers

Question 12

A Dataflow batch job frequently fails with 'OutOfMemoryError'. Which THREE are common causes? (Choose 3)

Accepted Answer

Inefficient GroupByKey with hot keys. A hot key in a GroupByKey operation causes all values for that key to be processed by a single worker, leading to memory exhaustion when the key's associated data exceeds the worker's memory capacity. This is a common cause of OutOfMemoryError in Dataflow batch jobs, as the SDK buffers all values for a key before emitting the result.

Answer

Too many parallel workers

Answer

Using Dataflow Shuffle

Question 13

A logistics company uses Cloud Functions to process incoming tracking events from IoT devices. Events are sent via HTTP triggers. During peak hours, some events fail with 500 errors. What is the best strategy to handle this reliably?

Accepted Answer

Use Cloud Pub/Sub as an intermediary: send events to Pub/Sub and trigger Cloud Functions via Pub/Sub subscription.. Cloud Pub/Sub decouples event ingestion from processing, providing at-least-once delivery and built-in retry with exponential backoff. This ensures that HTTP 500 errors from Cloud Functions are automatically retried without data loss, even during peak loads, and the Pub/Sub subscription can be configured with a dead-letter queue for persistent failures.

Answer

Implement client-side retry with exponential backoff.

Answer

Increase the Cloud Functions timeout to 9 minutes and memory to 2GB.

Answer

Switch to Cloud Tasks and configure retry parameters.

Question 14

A company uses Cloud Dataproc to run nightly Spark ETL jobs that process about 500 GB of data each night. The jobs currently take 4 hours to complete. The company wants to reduce the runtime to under 2 hours to meet a new SLA. The cluster is configured with 10 worker nodes (n1-standard-4) and 1 master node (n1-standard-4). The jobs are CPU-bound and use only default settings. The cluster is deleted after each job and recreated. The data is stored in Cloud Storage. The company is open to increasing cost but wants the most cost-effective solution to meet the SLA. Which approach should they take?

Accepted Answer

Increase the number of worker nodes to 20 and use preemptible VMs for half of them.. Adding more worker nodes (from 10 to 20) directly increases parallelism for CPU-bound Spark jobs, and using preemptible VMs for half of them reduces cost while still meeting the SLA. Since the job is CPU-bound and uses default settings, scaling horizontally with a mix of standard and preemptible VMs is the most cost-effective way to halve runtime, as Spark can efficiently distribute the workload across more cores.

Answer

Use a regional Cloud Storage bucket to improve read throughput.

Answer

Replace worker nodes with n1-highmem-16 instances to increase memory.

Answer

Change machine type to n2-standard-8 for all nodes.

Question 15

A streaming Dataflow job is processing messages from Cloud Pub/Sub. The job is underutilizing resources and the throughput is lower than expected. Which parameter should be adjusted to increase parallelism?

Accepted Answer

Increase the number of workers via maxNumWorkers. The job is underutilizing resources, meaning the existing workers are not fully loaded. Increasing the number of workers via maxNumWorkers directly increases parallelism by allowing Dataflow to distribute work across more VMs, which can increase throughput without changing the per-worker resource profile. This parameter controls the upper bound on the number of workers, enabling the autoscaler to scale out when there is backlog.

Answer

Change the workerMachineType to a higher CPU machine

Answer

Set the streaming engine to Dataflow Streaming Engine

Answer

Set autoscalingAlgorithm to THROUGHPUT_BASED

Question 16

A data engineering team needs to process a large volume of CSV files stored in Cloud Storage using Dataproc. The files are generated hourly and each contains millions of rows. They want to minimize the number of Dataproc cluster nodes to reduce cost while processing within an hour. Which configuration should they recommend?

Accepted Answer

Use a cluster with many small worker nodes and use Spark dynamic allocation.. Using many small worker nodes with Spark dynamic allocation allows the cluster to scale resources precisely to the workload, minimizing idle capacity and cost. Dynamic allocation enables executors to be added or removed based on the processing demands of the hourly CSV files, ensuring the job completes within the hour without over-provisioning nodes.

Answer

Use a cluster with preemptible worker nodes only.

Answer

Use a cluster with local SSDs for temporary storage.

Answer

Use a cluster with a few large worker nodes and use Spark static allocation.

Question 17

You are designing a streaming pipeline that must guarantee exactly-once processing. Which three services or features can help achieve this? (Choose THREE.)

Accepted Answer

Cloud Spanner for deduplication state across the pipeline. Cloud Spanner is correct because it provides globally distributed, strongly consistent transactions that can be used to maintain deduplication state across the entire streaming pipeline. By storing a unique key for each processed event in Spanner, the pipeline can atomically check and record whether an event has already been handled, ensuring exactly-once semantics even in the face of retries or failures.

Answer

Cloud Functions for post-processing

Answer

BigQuery streaming inserts with a unique key for deduplication

Question 18

A data pipeline ingests real-time events from Cloud Pub/Sub into BigQuery using Dataflow. The pipeline uses a sliding window of 5 minutes with a 1-minute period to aggregate event counts. Recently, the pipeline started failing with 'The worker failed to provide a heartbeat.' The Dataflow logs show high CPU usage on the workers. What is the best course of action to resolve the issue?

Accepted Answer

Increase the number of workers and enable autoscaling to distribute the load.. The 'worker failed to provide a heartbeat' error combined with high CPU usage indicates that workers are overloaded and cannot process data fast enough to maintain their heartbeat to the Dataflow service. Increasing the number of workers and enabling autoscaling distributes the computational load across more machines, reducing per-worker CPU pressure and allowing heartbeats to be sent on time. This directly addresses the root cause of resource exhaustion.

Answer

Reduce the number of workers to minimize coordination overhead.

Answer

Use a global window with a trigger to reduce state size.

Answer

Change the windowing to a fixed 5-minute window to reduce computations.

Question 19

A company uses Cloud Composer to orchestrate data pipelines. One DAG fails intermittently with the error: 'Task received SIGTERM signal.' The task runs a long-running Dataproc job. What is the most likely cause?

Accepted Answer

The Airflow task timed out due to the default execution timeout.. The default Airflow task execution timeout is 28 days in Cloud Composer, but individual tasks can have a shorter `execution_timeout` set in the DAG definition. When a long-running Dataproc job exceeds this timeout, Airflow sends a SIGTERM signal to the task to kill it, resulting in the observed error. This is the most likely cause because the error message directly indicates a forced termination by the Airflow scheduler, not an infrastructure or code failure.

Answer

The Dataproc cluster was preempted by Google Cloud.

Answer

The Dataproc job failed due to an error in the code.

Answer

The Cloud Composer environment ran out of disk space.

Question 20

What is the root cause of this error and the correct solution?

Accepted Answer

The Dataflow worker service account needs the BigQuery Data Viewer role.. Dataflow workers execute under a specific service account (compute engine default or custom), and that service account must have the BigQuery Data Viewer role to read data from BigQuery tables. Without this permission, the workers cannot access the source data, causing the job to fail with access errors. The BigQuery User role is insufficient for reading table data, and the BigQuery Admin role is overly permissive and not required for this task.

Answer

The BigQuery table requires authorized view access.

Answer

The user running the job needs the BigQuery Admin role.

Answer

The Dataflow service account needs the BigQuery User role.