Quick Answer
The answer is to increase `spark.sql.shuffle.partitions` to a higher value, such as 500. This configuration change directly resolves the OutOfMemoryError during shuffles by reducing the amount of data each partition must hold, thereby lowering memory pressure on each executor. When a Spark job processes 500 GB of data with only 10 workers of 14 GB each, the default shuffle partitions (often 200) create partitions too large for the available executor memory, causing the failure. On the DP-203 exam, this scenario tests your understanding of Spark shuffle memory tuning and resource allocation in Azure Databricks—a common trap is to over-provision by adding more workers, which wastes cost, when a simple partition adjustment suffices. Remember the mnemonic: “Shuffle partitions shrink strain” to recall that increasing partition count reduces per-partition memory load.
This DP-203 practice question tests your understanding of design and develop data processing. The scenario asks you to isolate a root cause — eliminate options that address a different problem before choosing. After answering, compare your reasoning against the explanation and wrong-answer breakdown below. Once you have made your selection, read the full explanation to reinforce the concept and understand why each distractor is designed to mislead on exam day.
A company is running a Spark job on Azure Databricks that processes 500 GB of data daily. The job frequently fails with 'OutOfMemoryError' during shuffles. The cluster uses 10 workers of type Standard_DS3_v2 (14 GB memory each). Which configuration change should you make to improve stability without over-provisioning?
Set spark.sql.shuffle.partitions to a higher value, e.g., 500.
Reduces data per partition, easing memory.
Increase the driver memory to 28 GB.
Why wrong: Driver memory not the bottleneck.
Increase the number of workers to 20.
Why wrong: Increases parallelism but may not address memory per task.
Reduce spark.sql.shuffle.partitions to 100.
Why wrong: Increases partition size, worsening OOM.
Answer choices
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
Set spark.sql.shuffle.partitions to a higher value, e.g., 500.
The 'OutOfMemoryError' during shuffles indicates that individual partitions are too large for the executor memory. Increasing `spark.sql.shuffle.partitions` to 500 reduces the amount of data per partition, lowering memory pressure during shuffle operations. This directly addresses the error without adding more hardware.
Key principle: Answer the scenario, not the keyword: identify the specific constraint before choosing the most familiar-sounding option.
Answer analysis
For each option: why learners choose it and why it is or isn't the right answer here.
Set spark.sql.shuffle.partitions to a higher value, e.g., 500.
Why this is correct
Reduces data per partition, easing memory.
Related concept
Read the scenario before looking for a memorised answer.
Increase the driver memory to 28 GB.
Why it's wrong here
Driver memory not the bottleneck.
Increase the number of workers to 20.
Why it's wrong here
Increases parallelism but may not address memory per task.
Reduce spark.sql.shuffle.partitions to 100.
Why it's wrong here
Increases partition size, worsening OOM.
Common exam traps
The trap here is that candidates often assume adding more workers (Option C) is the only way to fix memory errors, but the question tests understanding that partition size, not just cluster size, is the root cause of shuffle OOM errors.
Detailed technical explanation
Spark's shuffle operation uses a hash-based partitioning scheme where each partition is written to a separate file. The default `spark.sql.shuffle.partitions` is 200, which for 500 GB of data results in ~2.5 GB per partition, exceeding the ~4 GB executor memory (after overhead) on a Standard_DS3_v2 node. Increasing partitions to 500 reduces each partition to ~1 GB, fitting comfortably within memory. Real-world tuning often involves balancing partition size to avoid both memory errors and excessive small file overhead.
Key takeaway
Answer the scenario, not the keyword: identify the specific constraint before choosing the most familiar-sounding option.
Real-world example
A media company stores terabytes of video archives that are accessed once a year for audit purposes. Moving these objects to a cold storage tier (Azure Archive, S3 Glacier, or Google Nearline) costs a fraction of hot storage. Questions like this test whether you understand storage tiers, access frequency tradeoffs, and retrieval latency requirements.
What to study next
Identify which exam domain this question belongs to, review the core concept, then practise similar questions from the same domain.
Design and develop data processing — study guide chapter
Learn the concepts, then practise the questions
Design and develop data processing practice questions
Targeted practice on this topic area only
All DP-203 questions
846 questions across all exam domains
Microsoft Azure Data Engineer Associate DP-203 study guide
Full concept coverage aligned to exam objectives
DP-203 practice test guide
How to use practice tests most effectively before exam day
Related practice questions
Use these pages to review the topic behind this question. This is how one missed question becomes focused revision.
Practise DP-203 questions linked to Secure, monitor, and optimize data storage and data processing.
Practise DP-203 questions linked to Design and develop data processing.
Practise DP-203 questions linked to Design and implement data security.
Practise DP-203 questions linked to Monitor and optimize data storage and processing.
Practise DP-203 questions linked to Design and implement data storage.
Practise DP-203 questions linked to Develop data processing.
Practise DP-203 questions linked to DP-203 fundamentals.
Practise DP-203 questions linked to DP-203 scenario.
Practise DP-203 questions linked to DP-203 troubleshooting.
Practice this exam
Short sessions build daily habit. Longer sessions build exam-day stamina. Try a timed session to simulate real conditions.
FAQ
Design and develop data processing — This question tests Design and develop data processing — Read the scenario before looking for a memorised answer..
The correct answer is: Set spark.sql.shuffle.partitions to a higher value, e.g., 500. — The 'OutOfMemoryError' during shuffles indicates that individual partitions are too large for the executor memory. Increasing `spark.sql.shuffle.partitions` to 500 reduces the amount of data per partition, lowering memory pressure during shuffle operations. This directly addresses the error without adding more hardware.
Identify which exam domain this question belongs to, review the core concept, then practise similar questions from the same domain.
Read the scenario before looking for a memorised answer.
About these practice questions
Courseiva creates original exam-style practice questions with explanations and wrong-answer analysis. It does not publish real exam questions, exam dumps, or protected exam content. Learn why practice questions differ from exam dumps →
Last reviewed: Jun 11, 2026
This DP-203 practice question is part of Courseiva's free Microsoft certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the DP-203 exam.
Question Discussion
Share a tip, memory trick, or ask about the reasoning behind this question. Do not post real exam questions, leaked content, braindumps, or copyrighted exam material. Comments are moderated and may be removed without notice.
Sign in to join the discussion.