DEA-C01 Data Ingestion and Transformation Practice Question

A data engineer is using AWS Glue to run an ETL job that reads from Amazon S3, performs a join between two large datasets, and writes the result to Amazon Redshift. The job is taking longer than expected, and the engineer suspects data skew. Which technique can help mitigate data skew in the join?

⚠ Common exam trap

The trap here is assuming that adding more resources or using broadcast join will solve skew, when in fact skew requires specific runtime optimization techniques like AQE.

Answer choices

Why each option matters

Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.

Correct answer & explanation

✓

Enable the 'spark.sql.adaptive.enabled' configuration and set 'spark.sql.adaptive.skewJoin.enabled' to true.

Enabling Adaptive Query Execution with skew join handling allows Spark to dynamically detect and split skewed partitions during the join. This redistributes the workload more evenly across executors, mitigating the performance bottleneck caused by data skew.

Answer analysis

Option-by-option breakdown

For each option: why learners choose it and why it is or isn't the right answer here.

✓
Enable the 'spark.sql.adaptive.enabled' configuration and set 'spark.sql.adaptive.skewJoin.enabled' to true.
Why this is correct
Adaptive Query Execution (AQE) in Spark dynamically optimizes query plans at runtime. Enabling skew join handling allows AQE to detect skewed partitions and split them into smaller sub-partitions, balancing the workload. This significantly reduces the impact of data skew during joins, improving job performance.
✗
Use AWS Glue's 'groupFiles' option to combine small files before the join.
Why it's wrong here
Grouping small files can improve read performance, but it does not address data skew during a join. Data skew occurs when certain keys have disproportionately more data, causing some partitions to be much larger. Grouping files does not redistribute the skewed keys.
✗
Increase the number of DPUs to provide more memory for the join operation.
Why it's wrong here
Adding more DPUs increases overall resources but does not directly address skew. Skewed partitions will still be processed by a single task, potentially causing out-of-memory errors or long runtimes. More DPUs can help with overall throughput but not with the imbalance caused by skew.
✗
Use a broadcast join to replicate the smaller dataset across all nodes.
Why it's wrong here
Broadcast join is effective when one dataset is small enough to fit in memory, but here both datasets are large. Broadcasting a large dataset would cause memory issues and is not feasible. It does not address skew in the larger datasets.

Quick reference

AWS S3 Storage Class Comparison

Storage Class	Min Duration	Retrieval	Use Case
S3 Standard	None	Immediate	Frequently accessed data
S3 Standard-IA	30 days	Immediate	Infrequent access, rapid retrieval
S3 One Zone-IA	30 days	Immediate	Non-critical infrequent data
S3 Intelligent-Tiering	None	Immediate–hours	Unknown or changing access patterns
S3 Glacier Instant	90 days	Milliseconds	Archive with instant retrieval
S3 Glacier Flexible	90 days	Minutes–hours	Archive, flexible retrieval
S3 Glacier Deep Archive	180 days	Hours	Long-term compliance archive

Go deeper

DEA-C01 Data Ingestion and Transformation Practice Question

Why each option matters

Option-by-option breakdown

AWS S3 Storage Class Comparison

Related to this question