DEA-C01 Data Operations and Support Practice Question

A data engineer is monitoring an AWS Glue job that reads from an Amazon S3 bucket and writes to Amazon Redshift. The job has been running for 2 hours, which is longer than usual. The engineer checks the Glue job's metrics and sees that the number of active executors is high, but the job is not making progress. The engineer suspects a data skew issue. Which action should the engineer take to diagnose and mitigate the skew?

⚠ Common exam trap

The trap here is assuming that adding more resources (DPUs) will solve skew, when in fact skew requires redistributing data, not just adding compute.

Answer choices

Why each option matters

Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.

Correct answer & explanation

✓

Use the Spark UI to inspect the stage details and identify tasks that are taking significantly longer than others.

Data skew in Spark jobs manifests as a few tasks taking much longer than others. The Spark UI is the primary tool to diagnose skew by showing task-level metrics. Once identified, mitigation strategies include salting keys, using broadcast joins, or repartitioning. The correct action here is to use the Spark UI to inspect stage details.

Answer analysis

Option-by-option breakdown

For each option: why learners choose it and why it is or isn't the right answer here.

✓
Use the Spark UI to inspect the stage details and identify tasks that are taking significantly longer than others.
Why this is correct
The Spark UI provides detailed information about stages and tasks, including task duration and input size. By examining the stage details, you can identify tasks that are processing much more data than others, indicating data skew. This is the most direct way to diagnose skew in a Glue job.
✗
Increase the number of DPUs for the Glue job to add more executors and reduce the impact of skew.
Why it's wrong here
Adding more DPUs increases the number of executors, but it does not address the root cause of skew. Skewed partitions will still be processed by a single task, which can become a bottleneck. More executors may not help if the data is not evenly distributed, and it increases cost.
✗
Enable AWS Glue job metrics and examine the 'glue.driver.aggregate.bytesRead' metric to identify skewed partitions.
Why it's wrong here
The 'glue.driver.aggregate.bytesRead' metric shows the total bytes read by the job, but it does not provide per-partition or per-executor details needed to identify skew. To diagnose skew, you need metrics like 'glue.executor.aggregate.bytesRead' or Spark UI details that show task duration and data distribution.
✗
Modify the Glue job to use the 'groupFiles' option to combine small files and improve parallelism.
Why it's wrong here
The 'groupFiles' option is used to combine multiple small files into larger chunks for processing, which can improve performance when there are many small files. However, it does not address data skew within a single large file or partition. Skew occurs when a few keys have disproportionately more data, and grouping files does not redistribute that data.

Quick reference

AWS S3 Storage Class Comparison

Storage Class	Min Duration	Retrieval	Use Case
S3 Standard	None	Immediate	Frequently accessed data
S3 Standard-IA	30 days	Immediate	Infrequent access, rapid retrieval
S3 One Zone-IA	30 days	Immediate	Non-critical infrequent data
S3 Intelligent-Tiering	None	Immediate–hours	Unknown or changing access patterns
S3 Glacier Instant	90 days	Milliseconds	Archive with instant retrieval
S3 Glacier Flexible	90 days	Minutes–hours	Archive, flexible retrieval
S3 Glacier Deep Archive	180 days	Hours	Long-term compliance archive

Go deeper

DEA-C01 Data Operations and Support Practice Question

Why each option matters

Option-by-option breakdown

AWS S3 Storage Class Comparison

Related to this question