DEA-C01 Data Ingestion and Transformation Practice Question

Exhibit

Error Log:
[ERROR] org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1.0 failed 4 times, most recent failure: Lost task 0.3 in stage 1.0 (TID 6, ip-10-0-0-12.ec2.internal, executor 1): java.lang.OutOfMemoryError: Java heap space
	at org.apache.spark.sql.catalyst.expressions.UnsafeRow.<init>(UnsafeRow.java:42)

Refer to the exhibit. An AWS Glue ETL job is failing with an OutOfMemoryError. The job reads from Amazon S3 and performs a GROUP BY on a large dataset. Which change should the data engineer make to resolve this error?

⚠ Common exam trap

Candidates often confuse partition tuning (coalesce/repartition) with resource allocation, mistakenly thinking that adjusting partitions alone can fix memory errors without increasing the underlying compute and memory capacity.

Answer choices

Why each option matters

Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.

Correct answer & explanation

✓

Increase the number of DPUs allocated to the Glue job.

The OutOfMemoryError in an AWS Glue ETL job performing a GROUP BY on a large dataset indicates that the executors do not have enough memory to handle the shuffle operations required for aggregation. Increasing the number of DPUs (Data Processing Units) allocated to the Glue job increases the total memory and compute resources available, allowing the job to process larger partitions without running out of memory.

Answer analysis

Option-by-option breakdown

For each option: why learners choose it and why it is or isn't the right answer here.

✗
Use coalesce to reduce the number of partitions.
Why it's wrong here
Coalesce merges partitions into fewer, larger ones, concentrating each GROUP BY key's rows onto fewer executors and worsening the memory pressure causing the OutOfMemoryError. It suits reducing small-file write overhead before output, not relieving aggregation memory; repartitioning by key spreads the load instead.
✓
Increase the number of DPUs allocated to the Glue job.
Why this is correct
OutOfMemoryError during a GROUP BY on a large dataset occurs because each executor has insufficient memory for the shuffle and aggregation. Increasing DPUs adds more executors and memory, distributing the GROUP BY workload and preventing the driver or executor from exhausting heap during the shuffle phase.
✗
Increase the number of partitions in the DataFrame.
Why it's wrong here
Adding partitions scatters rows arbitrarily, so a single GROUP BY key's values still land on one executor and overflow its memory. More partitions help parallelism for narrow transformations, but aggregation needs repartitioning by the grouping key so each key's data is processed within executor memory limits.
✗
Use repartition to increase the number of partitions.
Why it's wrong here
Repartitioning increases parallelism but shuffles the entire dataset, and each task still materialises its partition's GROUP BY state in memory, so the driver or executor can still exhaust heap. Repartition suits skew correction; OutOfMemoryError on aggregation needs a broadcast or map-side combine strategy.

Visual reference

Quick reference

AWS S3 Storage Class Comparison

Storage Class	Min Duration	Retrieval	Use Case
S3 Standard	None	Immediate	Frequently accessed data
S3 Standard-IA	30 days	Immediate	Infrequent access, rapid retrieval
S3 One Zone-IA	30 days	Immediate	Non-critical infrequent data
S3 Intelligent-Tiering	None	Immediate–hours	Unknown or changing access patterns
S3 Glacier Instant	90 days	Milliseconds	Archive with instant retrieval
S3 Glacier Flexible	90 days	Minutes–hours	Archive, flexible retrieval
S3 Glacier Deep Archive	180 days	Hours	Long-term compliance archive

Go deeper

DEA-C01 Data Ingestion and Transformation Practice Question

Exhibit

Why each option matters

Option-by-option breakdown

AWS S3 Storage Class Comparison

Related to this question