DEA-C01 Data Ingestion and Transformation Practice Question

Exhibit

Error Log:
[ERROR] org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1.0 failed 4 times, most recent failure: Lost task 0.3 in stage 1.0 (TID 6, ip-10-0-0-12.ec2.internal, executor 1): java.lang.OutOfMemoryError: Java heap space
	at org.apache.spark.sql.catalyst.expressions.UnsafeRow.<init>(UnsafeRow.java:42)

Refer to the exhibit. An AWS Glue ETL job is failing with an OutOfMemoryError. The job reads from Amazon S3 and performs a GROUP BY on a large dataset. Which change should the data engineer make to resolve this error?

⚠ Common exam trap

Candidates often confuse partition tuning (coalesce/repartition) with resource allocation, mistakenly thinking that adjusting partitions alone can fix memory errors without increasing the underlying compute and memory capacity.

Answer choices

Why each option matters

Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.

Correct answer & explanation

✓

Increase the number of DPUs allocated to the Glue job.

The OutOfMemoryError in an AWS Glue ETL job performing a GROUP BY on a large dataset indicates that the executors do not have enough memory to handle the shuffle operations required for aggregation. Increasing the number of DPUs (Data Processing Units) allocated to the Glue job increases the total memory and compute resources available, allowing the job to process larger partitions without running out of memory.

Answer analysis

Option-by-option breakdown

For each option: why learners choose it and why it is or isn't the right answer here.

✗
Use coalesce to reduce the number of partitions.
Why it's wrong here
Coalesce reduces partitions but may cause data skew.
✓
Increase the number of DPUs allocated to the Glue job.
Why this is correct
More DPUs increase total memory available.
✗
Increase the number of partitions in the DataFrame.
Why it's wrong here
More partitions may increase overhead, not reduce memory per task.
✗
Use repartition to increase the number of partitions.
Why it's wrong here
Repartition increases partitions but may increase memory usage.

Visual reference

Quick reference

AWS S3 Storage Class Comparison

Storage Class	Min Duration	Retrieval	Use Case
S3 Standard	None	Immediate	Frequently accessed data
S3 Standard-IA	30 days	Immediate	Infrequent access, rapid retrieval
S3 One Zone-IA	30 days	Immediate	Non-critical infrequent data
S3 Intelligent-Tiering	None	Immediate–hours	Unknown or changing access patterns
S3 Glacier Instant	90 days	Milliseconds	Archive with instant retrieval
S3 Glacier Flexible	90 days	Minutes–hours	Archive, flexible retrieval
S3 Glacier Deep Archive	180 days	Hours	Long-term compliance archive

Go deeper

DEA-C01 Data Ingestion and Transformation Practice Question

Exhibit

Why each option matters

Option-by-option breakdown

AWS S3 Storage Class Comparison

Related to this question