mediumMultiple Choice
PDE Practice Question: Your team runs a weekly batch ETL pipeline using…
Your team runs a weekly batch ETL pipeline using Cloud Dataproc. The pipeline reads raw data from Cloud Storage, transforms it with Apache Spark, and writes results to BigQuery. Recently, the pipeline has been failing with the error 'Out of Memory' during the shuffle phase. The cluster uses standard worker nodes (n1-standard-4). What is the most effective way to resolve this without increasing total cost?
⚠ Common exam trap
Many candidates assume memory errors must be solved by adding more memory (Option D) or more nodes (Option B), ignoring the cost constraint and the fact that repartitioning can resolve the issue without additional resources.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
Increase the number of Spark partitions by setting spark.sql.shuffle.partitions to a higher value.
The 'Out of Memory' error during the shuffle phase indicates that individual executor tasks are processing too much data per partition. Increasing `spark.sql.shuffle.partitions` reduces the amount of data each task handles, lowering memory pressure per executor without adding more nodes or upgrading hardware. This directly addresses the shuffle memory bottleneck while keeping the total cluster cost unchanged.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✓
Increase the number of Spark partitions by setting spark.sql.shuffle.partitions to a higher value.
Why this is correct
Raising `spark.sql.shuffle.partitions` splits shuffle data into more, smaller partitions, reducing per-task memory pressure during the shuffle phase. This directly addresses the Out of Memory failures on the existing n1-standard-4 workers without adding nodes, so total cost stays unchanged.
- ✗
Increase the number of worker nodes by adding more n1-standard-4 instances.
Why it's wrong here
Adding n1-standard-4 workers increases shuffle parallelism but each node keeps the same 4 vCPU and 15 GB memory, so per-executor memory pressure during shuffle persists while cost rises. It is tempting because horizontal scaling commonly fixes throughput, and would help if the bottleneck were insufficient parallelism rather than memory.
- ✗
Enable dynamic allocation and use preemptible VMs for some workers.
Why it's wrong here
Preemptible workers and dynamic allocation address cost and elasticity, not the shuffle-phase memory shortfall on n1-standard-4 nodes; preemption can even worsen failures by removing executors mid-shuffle. Preemptible VMs suit fault-tolerant, non-shuffle-bound batch workloads where cost reduction outweighs interruption risk.
- ✗
Switch worker nodes to n1-highmem-4 instances to provide more memory.
Why it's wrong here
n1-highmem-4 offers the same four vCPUs as n1-standard-4 with more RAM per node, so the per-node price rises and total cost increases, breaching the constraint. Highmem shapes suit memory-intensive workloads where the budget permits paying a premium for additional RAM.
Visual reference
Go deeper
Related to this question
About these practice questions
This PDE question is part of Courseiva's 747-question bank — original exam-style content with full explanations and wrong-answer analysis, never real exam questions or exam dumps. Learn why practice questions differ from exam dumps →
JA
Written by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
This PDE practice question is part of Courseiva's free Google Cloud certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the PDE exam.