The answer is to increase the spark.executor.memory property. This is correct because when a Spark container is killed by YARN for exceeding memory limits, it means the executor process has consumed more memory than the YARN container’s allocated boundary, including off-heap and JVM overhead. By raising spark.executor.memory, you request a larger YARN container, giving the executor sufficient headroom to handle the job’s memory demands without being preemptively terminated. On the Google Professional Data Engineer exam, this scenario tests your understanding of Spark resource tuning on Cloud Dataproc, often appearing as a troubleshooting question where candidates mistakenly adjust spark.memory.offHeap or shuffle partitions instead. A common trap is forgetting that YARN enforces a hard limit based on the container size, not just Spark’s internal memory regions. Remember the mnemonic: “Executor memory is the container’s ceiling—raise it to stop the killing.”
PDE Practice Question: Building and operationalizing data processing systems
This PDE practice question tests your understanding of building and operationalizing data processing systems. The scenario asks you to isolate a root cause — eliminate options that address a different problem before choosing. After answering, compare your reasoning against the explanation and wrong-answer breakdown below. Once you have made your selection, read the full explanation to reinforce the concept and understand why each distractor is designed to mislead on exam day.
Your team uses Cloud Dataproc to run a Spark ML training job. The job is failing with an error: 'Container killed by YARN for exceeding memory limits.' What should you do to fix this?
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
Increase the spark.executor.memory property
The error 'Container killed by YARN for exceeding memory limits' indicates that the Spark executor process is using more memory than the YARN container allows. Increasing `spark.executor.memory` allocates a larger YARN container for each executor, providing the necessary headroom for the Spark application's memory demands, including overhead for off-heap memory and JVM internals.
Key principle: Answer the scenario, not the keyword: identify the specific constraint before choosing the most familiar-sounding option.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
✓
Increase the spark.executor.memory property
Why this is correct
This directly addresses the memory limit for each executor.
Related concept
Read the scenario before looking for a memorised answer.
✗
Use preemptible VMs for faster execution
Why it's wrong here
Preemptible VMs are cheaper but don't fix memory limits.
✗
Increase the number of worker nodes
Why it's wrong here
More nodes distribute load but don't increase per-container memory.
✗
Enable the external shuffle service
Why it's wrong here
External shuffle service reduces shuffle memory but not executor memory.
Common exam traps
Common exam trap: answer the scenario, not the keyword
The trap here is that candidates often confuse scaling horizontally (adding nodes) with scaling vertically (increasing per-node resources), and assume more nodes will fix memory limits when the issue is per-container allocation.
Detailed technical explanation
How to think about this question
YARN enforces memory limits via cgroups, and Spark's executor memory includes both `spark.executor.memory` (heap) and `spark.executor.memoryOverhead` (off-heap). The total container memory is the sum of these two, and if the executor exceeds this total, YARN kills the container. A common subtlety is that Spark's default memory overhead is 10% of executor memory (minimum 384 MB), which can be insufficient for memory-intensive operations like large broadcasts or Python UDFs in PySpark.
KKey Concepts to Remember
Read the scenario before looking for a memorised answer.
Find the constraint that changes the correct option.
Eliminate answers that are true in general but not in this case.
TExam Day Tips
→Watch for words such as best, first, most likely and least administrative effort.
→Review why wrong options are wrong, not only why the correct option is correct.
Key takeaway
Answer the scenario, not the keyword: identify the specific constraint before choosing the most familiar-sounding option.
Real-world example
How this comes up in practice
An e-commerce site experiences heavy traffic on Black Friday and near-zero traffic during off-peak weeks. Rather than provisioning permanent large VMs, the team uses auto-scaling groups that add capacity automatically under load and reduce it overnight. Questions like this test whether you understand elasticity, availability zones, and cloud compute scaling patterns.
What to study next
Got this wrong? Here's your next step.
Identify which exam domain this question belongs to, review the core concept, then practise similar questions from the same domain.
Building and operationalizing data processing systems — This question tests Building and operationalizing data processing systems — Read the scenario before looking for a memorised answer..
What is the correct answer to this question?
The correct answer is: Increase the spark.executor.memory property — The error 'Container killed by YARN for exceeding memory limits' indicates that the Spark executor process is using more memory than the YARN container allows. Increasing `spark.executor.memory` allocates a larger YARN container for each executor, providing the necessary headroom for the Spark application's memory demands, including overhead for off-heap memory and JVM internals.
What should I do if I get this PDE question wrong?
Identify which exam domain this question belongs to, review the core concept, then practise similar questions from the same domain.
What is the key concept behind this question?
Read the scenario before looking for a memorised answer.
About these practice questions
Courseiva creates original exam-style practice questions with explanations and wrong-answer analysis. It does not publish real exam questions, exam dumps, or protected exam content. Learn why practice questions differ from exam dumps →
Share a tip, memory trick, or ask about the reasoning behind this question. Do not post real exam questions, leaked content, braindumps, or copyrighted exam material. Comments are moderated and may be removed without notice.
This PDE practice question is part of Courseiva's free Google Cloud certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the PDE exam.
Question Discussion
Share a tip, memory trick, or ask about the reasoning behind this question. Do not post real exam questions, leaked content, braindumps, or copyrighted exam material. Comments are moderated and may be removed without notice.
Sign in to join the discussion.