DEA-C01 Data Ingestion and Transformation Practice Question

A data engineer is using AWS Glue to process a large dataset in Amazon S3. The dataset consists of many small JSON files (average 100 KB each) stored in a single prefix. The Glue job reads these files, performs transformations, and writes the output to Parquet in another S3 location. The job is running slowly and consuming many DPUs. Which action should the data engineer take to improve performance?

⚠ Common exam trap

The trap here is assuming that adding more DPUs or using bookmarks will solve performance issues with many small files, when the real solution is to group files to reduce overhead.

Answer choices

Why each option matters

Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.

Correct answer & explanation

✓

Use AWS Glue's `groupFiles` option to group multiple small files into larger chunks for processing.

The `groupFiles` option in AWS Glue allows the job to group multiple small files into a single partition, reducing the number of tasks and the overhead of reading many small files. This directly addresses the performance bottleneck caused by many small JSON files, improving job speed and reducing DPU consumption. Other options do not solve the small file issue.

Answer analysis

Option-by-option breakdown

For each option: why learners choose it and why it is or isn't the right answer here.

✓
Use AWS Glue's `groupFiles` option to group multiple small files into larger chunks for processing.
Why this is correct
AWS Glue provides the `groupFiles` option (e.g., 'inPartition') to group small files into larger groups, reducing the number of tasks and improving read throughput. This is specifically designed to handle many small files by coalescing them, which reduces overhead and improves job performance without excessive DPUs.
✗
Convert the JSON files to Parquet using an AWS Glue crawler before processing.
Why it's wrong here
An AWS Glue crawler catalogs data; it does not convert file formats. Converting JSON to Parquet would require an ETL job, which is what the engineer is already doing. The crawler would not solve the small file problem and would add an extra step without performance benefit.
✗
Enable AWS Glue job bookmarks to skip already processed files.
Why it's wrong here
Job bookmarks track processed data to avoid reprocessing, but they do not address the performance issue of reading many small files. The job would still read all new small files each run, and the overhead of opening many files remains. Bookmarks help with incremental processing, not with small file compaction.
✗
Increase the number of DPUs allocated to the job to enable more parallel tasks.
Why it's wrong here
Adding more DPUs increases parallelism, but with many small files, the overhead of task scheduling and file opening may not scale linearly. It could lead to diminishing returns and higher cost. The root cause is the small file size, not insufficient compute resources.

Quick reference

AWS S3 Storage Class Comparison

Storage Class	Min Duration	Retrieval	Use Case
S3 Standard	None	Immediate	Frequently accessed data
S3 Standard-IA	30 days	Immediate	Infrequent access, rapid retrieval
S3 One Zone-IA	30 days	Immediate	Non-critical infrequent data
S3 Intelligent-Tiering	None	Immediate–hours	Unknown or changing access patterns
S3 Glacier Instant	90 days	Milliseconds	Archive with instant retrieval
S3 Glacier Flexible	90 days	Minutes–hours	Archive, flexible retrieval
S3 Glacier Deep Archive	180 days	Hours	Long-term compliance archive

Go deeper

DEA-C01 Data Ingestion and Transformation Practice Question

Why each option matters

Option-by-option breakdown

AWS S3 Storage Class Comparison

Related to this question