DEA-C01 Data Ingestion and Transformation Practice Question

A data engineer is using AWS Glue to run an ETL job that reads from an Amazon S3 bucket and writes to another S3 bucket. The job processes data in CSV format and the engineer wants to ensure the output is partitioned by year, month, and day based on a timestamp column in the data. The engineer needs to optimize the job for performance and cost. Which approach should the engineer take?

⚠ Common exam trap

It's easy for candidates to confuse job bookmarks with partitioning, or thinking that ResolveChoice can derive new columns, when actually partitionColumns is the direct method for output partitioning.

Answer choices

Why each option matters

Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.

Correct answer & explanation

✓

Use the AWS Glue DynamicFrame partitionColumns parameter to specify year, month, and day as partition keys.

The partitionColumns parameter in AWS Glue's write operation is the correct way to partition output data by specified columns. It creates a hierarchical directory structure in S3, enabling efficient querying with services like Athena and Redshift Spectrum. Other options either misuse transformations or misunderstand the purpose of bookmarks and single-file output.

Answer analysis

Option-by-option breakdown

For each option: why learners choose it and why it is or isn't the right answer here.

✗
Write the output to a single CSV file and then use Amazon Athena to create a partitioned table.
Why it's wrong here
Writing to a single CSV file defeats the purpose of partitioning and can cause performance bottlenecks. Athena can create a partitioned table over existing data, but it does not physically partition the data. The data would still be in one file, so queries would not benefit from partition pruning. This approach does not meet the requirement to partition the output.
✗
Use the AWS Glue job's bookmark feature to automatically partition the output by timestamp.
Why it's wrong here
Job bookmarks are used to track processed data and prevent reprocessing, not to partition output. They do not create partition columns or directory structures. This option misunderstands the purpose of bookmarks and does not achieve the desired partitioning of output data.
✗
Use the AWS Glue ResolveChoice transformation to split the timestamp column into separate year, month, and day columns, then write to S3 with partitioning.
Why it's wrong here
ResolveChoice is used to handle data type conflicts in a DynamicFrame, not to derive new columns. To split a timestamp, you would use a Map transformation or SQL expression. Therefore, this approach does not correctly create the partition columns and is not appropriate for the task.
✓
Use the AWS Glue DynamicFrame partitionColumns parameter to specify year, month, and day as partition keys.
Why this is correct
The partitionColumns parameter in AWS Glue's write_dynamic_frame method allows you to specify columns to partition the output by. When writing to S3, Glue will create a directory structure based on these columns, such as year=2023/month=01/day=01. This is the standard and efficient way to partition output data in Glue, improving query performance and reducing costs for downstream analytics.

Quick reference

AWS S3 Storage Class Comparison

Storage Class	Min Duration	Retrieval	Use Case
S3 Standard	None	Immediate	Frequently accessed data
S3 Standard-IA	30 days	Immediate	Infrequent access, rapid retrieval
S3 One Zone-IA	30 days	Immediate	Non-critical infrequent data
S3 Intelligent-Tiering	None	Immediate–hours	Unknown or changing access patterns
S3 Glacier Instant	90 days	Milliseconds	Archive with instant retrieval
S3 Glacier Flexible	90 days	Minutes–hours	Archive, flexible retrieval
S3 Glacier Deep Archive	180 days	Hours	Long-term compliance archive

Go deeper

DEA-C01 Data Ingestion and Transformation Practice Question

Why each option matters

Option-by-option breakdown

AWS S3 Storage Class Comparison

Related to this question