DEA-C01 Data Ingestion and Transformation Practice Question
A company runs an AWS Glue ETL job that reads data from Amazon S3, transforms it, and writes back to S3 in a different partition structure. The job uses the 'spark.sql.shuffle.partitions' option set to 200. After the job completes, the output has many small files. The data engineer wants to minimize the number of output files while maintaining job performance. Which action should the engineer take?
⚠ Common exam trap
It's easy for candidates to confuse 'coalesce' with 'repartition' or assume that adjusting 'spark.sql.shuffle.partitions' directly controls output file count, when in fact it only controls the number of partitions during shuffle operations, not the final write partition count.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
Use 'coalesce(n)' with n based on target file size (e.g., 128 MB) before writing.
'coalesce(n)' reduces the number of partitions without triggering a full shuffle, allowing you to control the number of output files based on a target file size (e.g., 128 MB). This minimizes small files while preserving job performance, as coalesce is a narrow transformation that avoids the overhead of a shuffle. In contrast, 'repartition(n)' would cause a full shuffle, degrading performance.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✓
Use 'coalesce(n)' with n based on target file size (e.g., 128 MB) before writing.
Why this is correct
Coalesce reduces partitions without a full shuffle, minimizing files.
- ✗
Enable S3 multipart upload for the Glue job.
Why it's wrong here
Multipart upload is for large files, not file count.
- ✗
Increase the 'spark.sql.shuffle.partitions' to 500.
Why it's wrong here
Increases parallelism, leading to more files.
- ✗
Reduce the 'spark.sql.shuffle.partitions' to 50.
Why it's wrong here
Reduces parallelism, may cause OOM, but still produces many small files.
Quick reference
AWS S3 Storage Class Comparison
| Storage Class | Min Duration | Retrieval | Use Case |
|---|---|---|---|
| S3 Standard | None | Immediate | Frequently accessed data |
| S3 Standard-IA | 30 days | Immediate | Infrequent access, rapid retrieval |
| S3 One Zone-IA | 30 days | Immediate | Non-critical infrequent data |
| S3 Intelligent-Tiering | None | Immediate–hours | Unknown or changing access patterns |
| S3 Glacier Instant | 90 days | Milliseconds | Archive with instant retrieval |
| S3 Glacier Flexible | 90 days | Minutes–hours | Archive, flexible retrieval |
| S3 Glacier Deep Archive | 180 days | Hours | Long-term compliance archive |
Go deeper
Related to this question
About these practice questions
One of 1,711 original DEA-C01 practice questions on Courseiva, each with a full explanation and wrong-answer analysis — not exam dumps or protected exam content. Learn why practice questions differ from exam dumps →
JA
Written by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
This DEA-C01 practice question is part of Courseiva's free Amazon Web Services certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the DEA-C01 exam.