PDE Practice Test 22 — 15 Questions

Question 1

A data engineer is using Spark on Dataproc to process a large dataset. They notice the job is slow due to excessive shuffling. They want to optimize the job by using a more efficient data structure that reduces serialization overhead and provides better memory management. Which Spark API should they use?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

DataFrames or Datasets. Spark DataFrames/Datasets use Tungsten execution engine, which provides optimized serialization and memory management. RDDs lack these optimizations. Spark SQL is a module, not an API. Spark Streaming is for streaming.

Answer

Spark SQL

Answer

Spark Streaming

Answer

RDDs

Question 2

A company is using Pub/Sub to ingest clickstream events and Dataflow to write to BigQuery. They observe that some events are malformed and cause the pipeline to fail. They need a solution that captures malformed events without blocking the pipeline and allows reprocessing later. Which Dataflow pattern should they implement?

Accepted Answer

Write malformed events to a dead letter sink (e.g., another Pub/Sub topic or GCS bucket) and continue processing healthy events. Dead letter sinks (DLQ) are the standard pattern for handling bad records in Dataflow. The pipeline writes malformed records to a separate sink (e.g., Pub/Sub topic or GCS) for later analysis. Side inputs are for enriching data, not error handling. Reshuffle doesn't apply. Output tags (side outputs) can also be used, but explicit dead letter pattern is more standard.

Answer

Use a side input to filter malformed events before the main pipeline

Answer

Use the Reshuffle transform to reattempt failures

Answer

Use logging alerts to notify the team and stop the pipeline on error

Question 3

Your company has a Dataproc cluster that runs Spark jobs. You need to choose between RDDs, DataFrames, and Datasets for a new job that performs complex aggregations on structured data. Which TWO statements are correct regarding performance and ease of use?

Accepted Answer

DataFrames store data in a columnar format, allowing better compression.. DataFrames are optimized with Catalyst optimizer and Tungsten execution, providing better performance than RDDs for structured data. Datasets combine type safety with optimized execution, but for most analytics workloads, DataFrames are sufficient and simpler.

Answer

DataFrames and Datasets are both available in PySpark.

Answer

RDDs are easier to use than DataFrames for complex aggregations.

Answer

Datasets provide compile-time type safety and are always faster than DataFrames.

Question 4

A company uses BigQuery to store event data. They need to load data from multiple sources with different schemas and expect frequent schema changes. Which approach provides the most flexibility for schema evolution while minimizing load failures and performance impact?

Accepted Answer

Use the Storage Write API in buffered mode with schema auto-detection. Using the Storage Write API with buffered mode allows schema auto-detection and flexible schema updates without failing loads, and provides better performance than legacy streaming inserts.

Answer

Load data as JSON files in Cloud Storage and use external tables

Answer

Use legacy streaming inserts with schema auto-detect enabled

Answer

Use Dataflow to preprocess and write to BigQuery using Storage Write API in committed mode

Question 5

A company uses Kafka on Dataproc to ingest streaming data. They want to process the data with Spark Structured Streaming and write results to BigQuery. The team is using Dataproc clusters. Which approach minimizes cost while maintaining performance?

Accepted Answer

Use a Dataproc cluster with standard master nodes and preemptible worker nodes. Preemptible VMs are cost-effective for worker nodes; master nodes should be standard for reliability.

Answer

Use a Dataproc cluster with all preemptible VMs

Answer

Use a single-node Dataproc cluster

Answer

Use a Dataproc cluster with standard nodes and enable autoscaling

Question 6

A data engineer is using Apache Spark on Dataproc to process a large dataset. They need to perform complex aggregation and transformation with high performance. The dataset has a known schema and they want to take advantage of Catalyst optimizer. Which Spark API should they use?

Accepted Answer

DataFrames. DataFrames provide high-level API with Catalyst optimizer for performance, making them ideal for complex aggregations and transformations on structured data.

Answer

Spark SQL only

Answer

Datasets

Answer

RDDs

Question 7

You need to schedule a simple workflow that fetches data from an API every hour, transforms it using Cloud Functions, and writes the result to Cloud Storage. The workflow has no complex branching or retry logic beyond basic retries. Which orchestration service is the MOST cost-effective and simplest to implement?

Accepted Answer

Workflows. Workflows is serverless, pay-per-execution, and defined in YAML/JSON. It integrates natively with Cloud Functions and Cloud Storage. Cloud Composer is overkill for simple linear workflows and incurs cluster costs. Cloud Scheduler alone cannot orchestrate multiple steps. Dataflow is for data processing, not orchestration.

Answer

Cloud Scheduler

Answer

Cloud Composer

Answer

Dataflow

Question 8

Which BigQuery SQL function returns the rank of a row within a window, with gaps in the ranking for ties?

Accepted Answer

RANK(). RANK() assigns the same rank to ties and leaves gaps (e.g., 1,1,3). ROW_NUMBER() assigns unique consecutive numbers. DENSE_RANK() does not leave gaps.

Answer

NTILE()

Answer

DENSE_RANK()

Answer

ROW_NUMBER()

Question 9

A company runs a Dataflow streaming pipeline that processes financial transactions. They need to apply a new transformation that enriches the data with a lookup from Cloud Bigtable without stopping the pipeline. The pipeline must be updated in a way that minimises data loss and preserves exactly-once semantics. What is the recommended approach?

Accepted Answer

Use the Dataflow update option with the same pipeline name and new version, ensuring the transform is backward compatible.. Dataflow supports updating a streaming pipeline without draining by replacing the pipeline version. Using the --update flag with the same pipeline name and a new version allows the pipeline to be upgraded while preserving the state exactly-once.

Answer

Drain the pipeline first, then start a new pipeline with the updated code.

Answer

Create a new pipeline in parallel and switch the Pub/Sub subscription to the new pipeline.

Answer

Stop the pipeline, update the code, and restart with a new pipeline name.

Question 10

You are troubleshooting a Dataproc cluster that runs nightly Spark jobs. The jobs are failing with out-of-memory errors. You want to reduce costs while fixing the issue. Which combination of actions should you take? (Select the BEST answer.)

Accepted Answer

Increase the number of preemptible worker nodes and use standard machine types for the master.. Preemptible workers are cheaper and can be used for worker nodes, but not for master nodes. Adding more preemptible workers can increase parallelism and reduce memory pressure per worker, but may cause more preemptions. Using high-memory master nodes is not necessary for worker memory issues. Using SSDs for scratch storage can improve performance but does not directly address OOM. Reducing worker count would exacerbate the problem.

Answer

Use SSD persistent disks for all nodes to improve I/O performance.

Answer

Decrease the number of worker nodes and increase the size of each worker.

Answer

Switch to a high-memory machine type for the master node only.

Question 11

A company runs a critical real-time data pipeline using Dataflow that ingests events from Cloud Pub/Sub, performs aggregations using sliding windows, and writes results to BigQuery. The pipeline is deployed in us-central1. The pipeline's latency has increased recently, and the Dataflow monitoring shows that the 'system lag' metric is consistently above 5 minutes. The pipeline is using Streaming Engine and has 10 workers with 4 vCPUs each. The pipeline processes approximately 100,000 events per second. The team has verified that the source Pub/Sub topic has sufficient publish throughput and the BigQuery table has no quota issues. The pipeline logs show that some workers are experiencing GC overhead limit exceeded errors. The pipeline code uses stateful processing with a custom keyed state for deduplication. What is the most likely cause of the increased latency?

Accepted Answer

The stateful processing is causing large state sizes that lead to GC overhead; use a more efficient state backend or increase worker memory.. The GC overhead limit exceeded errors indicate that workers are spending too much time garbage collecting, which is a classic symptom of excessive heap memory usage. Stateful processing with custom keyed state for deduplication can cause large per-key state sizes, especially with sliding windows that maintain overlapping state for each key. This forces the JVM to constantly garbage collect, increasing system lag beyond 5 minutes. Using a more efficient state backend (e.g., reducing state size or using Dataflow's built-in deduplication) or increasing worker memory directly addresses the root cause.

Answer

The number of workers is insufficient; increasing to 20 workers will reduce latency.

Answer

The sliding window duration is too long; reducing it to 1 minute will improve performance.

Answer

The deduplication logic is causing a bottleneck; removing it will reduce latency.

Question 12

A company runs a Dataproc cluster for nightly batch jobs. The cluster uses preemptible workers for cost savings. Recently, the jobs have been failing intermittently with 'Disk quota exceeded' errors on the persistent disks attached to the preemptible workers. The cluster is configured with a master node and 10 worker nodes, each with a 100 GB persistent disk. The preemptible workers are dynamically added and removed. What is the most likely cause and the best long-term solution?

Accepted Answer

The preemptible workers are exceeding the project's persistent disk quota in the region because every time a preempted worker restarts, it tries to attach a new disk. Increase the disk quota.. The intermittent 'Disk quota exceeded' errors on preemptible workers are caused by the project's regional persistent disk quota being exhausted. When a preemptible worker is reclaimed, the cluster attempts to attach a new persistent disk to the replacement worker, but the old disk is not immediately deleted, leading to a buildup of unattached disks that consume quota. The best long-term solution is to increase the persistent disk quota in the region to accommodate the temporary disks from preempted workers.

Answer

The persistent disks of the preemptible workers are too small. Resize the persistent disks to 200 GB each.

Answer

The preemptible workers are using local SSDs that are not recreated on reclaim. Use non-preemptible workers with local SSDs instead.

Answer

The preemptible workers do not have enough persistent disk space to store intermediate shuffle data. Switch to standard workers to avoid this issue.

Question 13

A data engineer wants to train a linear regression model in BigQuery ML to predict sales. The training data includes a categorical feature with 1000+ unique values. Which method is most appropriate to handle this feature in the CREATE MODEL statement?

Accepted Answer

Use TRANSFORM clause with ML.FEATURE_CROSS or manual hashing.. BigQuery ML automatically one-hot encodes categorical features with fewer than a threshold of unique values. For high-cardinality features, you can use TRANSFORM to apply feature engineering like hashing or bucketizing.

Answer

Set max_categorical_features=100 in the model options.

Answer

Use the OPTIONS(ENCODE='ONE_HOT_ENCODING') parameter in the model options.

Answer

The model automatically handles high-cardinality features without any additional steps.

Question 14

A company uses AutoML Tables to train a classification model. They want to improve model performance by engineering new features from existing timestamp columns. Which three techniques can they apply within AutoML Tables? (Choose 3)

Accepted Answer

Create a new column with the difference between two timestamps.. AutoML Tables automatically extracts features from timestamp columns, such as day of week, month, hour, etc. Users can also manually create new columns via the UI or by preprocessing data before import. However, manual SQL functions cannot be used directly within AutoML Tables.

Answer

Manually add a column with a boolean indicating if the timestamp falls on a weekend.

Answer

Apply a SQL UDF in the AutoML Tables training configuration.

Question 15

You need to split a time-series dataset into training and evaluation sets for a forecasting model. The data is ordered by timestamp. Which splitting technique should you use?

Accepted Answer

Sequential split where training data precedes evaluation data in time.. For time-series data, a random split would leak future information into training. A sequential split (earlier data for training, later for evaluation) is required.

Answer

Use k-fold cross-validation with random folds.

Answer

Stratified split based on the target variable.

Answer

Random split with 80% training, 20% evaluation.