DEA-C01 Data Ingestion and Transformation Practice Question

A data engineer maintains an AWS Glue ETL job that reads JSON files from Amazon S3, applies transformations using a DynamicFrame, and writes Parquet to another S3 bucket. The job currently reads the entire source prefix on every run, but the source is now partitioned by year/month/day and only new partitions need processing. The engineer wants to process only new partitions without changing the job's transformation logic. Which change should the engineer make?

⚠ Common exam trap

Candidates often confuse partition discovery (crawlers and catalog updates) with partition filtering at read time, when only job bookmarks give automatic incremental reads.

Answer choices

Why each option matters

Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.

Correct answer & explanation

✓

Enable job bookmarks on the Glue job so it tracks previously processed S3 objects and skips them on subsequent runs

Job bookmarks are the native Glue mechanism for incremental processing. When enabled, Glue tracks the state of previously processed S3 objects and partitions, so each run reads only new data. This meets the requirement to process only new partitions without rewriting the transformation logic or manually specifying dates.

Answer analysis

Option-by-option breakdown

For each option: why learners choose it and why it is or isn't the right answer here.

✓
Enable job bookmarks on the Glue job so it tracks previously processed S3 objects and skips them on subsequent runs
Why this is correct
Glue job bookmarks persist state about which S3 objects and partitions were processed in prior runs, so subsequent runs read only new data. This requires no change to the transformation logic and directly addresses incremental processing of newly arriving partitions.
✗
Create a Glue crawler that catalogs only the new partitions and point the job at the crawler's output table
Why it's wrong here
A crawler can discover new partitions and update the Data Catalog, but the Glue job still reads the table's full partition set unless the job is told to filter. Crawling alone does not restrict the read scope, so the job would continue processing all partitions and waste DPU time.
✗
Configure the job to use a smaller number of DPUs so each run scans less data
Why it's wrong here
Reducing DPUs limits compute capacity but does not change which S3 objects the job reads. The job would still enumerate and process the full prefix, just more slowly, so it does not achieve incremental partition processing and may even increase runtime and cost.
✗
Add a pushdown predicate to the DynamicFrame that filters on the year, month, and day partition columns using hardcoded values
Why it's wrong here
A pushdown predicate can restrict which partitions are read, but hardcoding date values means the job must be edited each run and cannot autonomously discover new partitions. This does not provide the automatic incremental behavior the scenario requires.

Quick reference

AWS S3 Storage Class Comparison

Storage Class	Min Duration	Retrieval	Use Case
S3 Standard	None	Immediate	Frequently accessed data
S3 Standard-IA	30 days	Immediate	Infrequent access, rapid retrieval
S3 One Zone-IA	30 days	Immediate	Non-critical infrequent data
S3 Intelligent-Tiering	None	Immediate–hours	Unknown or changing access patterns
S3 Glacier Instant	90 days	Milliseconds	Archive with instant retrieval
S3 Glacier Flexible	90 days	Minutes–hours	Archive, flexible retrieval
S3 Glacier Deep Archive	180 days	Hours	Long-term compliance archive

Go deeper

DEA-C01 Data Ingestion and Transformation Practice Question

Why each option matters

Option-by-option breakdown

AWS S3 Storage Class Comparison

Related to this question