Incremental ETL Processing with AWS Glue Job Bookmarks
A data engineer is designing a data pipeline that uses AWS Glue to process data from an RDS MySQL database. The pipeline must capture only incremental changes (inserts and updates) and run every hour. Which approach is most cost-effective and reliable?
Quick Answer
The answer is to use Glue job bookmarks to track and process only new and updated records. This approach is correct because Glue job bookmarks automatically persist state information about previously processed data, enabling incremental ETL processing without reprocessing entire tables. When a Glue job runs with bookmarks enabled, it reads only the data that has changed since the last run, making it both cost-effective and reliable for hourly pipelines from RDS MySQL. On the AWS Certified Data Engineer Associate DEA-C01 exam, this question tests your understanding of how to balance cost and complexity—common traps include over-engineering with DMS for CDC or relying on timestamp-based queries that miss unindexed or non-updated timestamp fields. Remember the key trade-off: Glue bookmarks handle state for you, while full scans waste compute and money. Memory tip: think “bookmark = bookmark your place in the data,” so you never re-read the whole chapter.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
Use Glue job bookmarks to track and process only new and updated records
AWS Glue job bookmarks track processed data and enable incremental processing by automatically storing state information about previously processed data, so only new and updated records are processed in subsequent runs. This approach is cost-effective because it avoids full table scans and reduces data processing. Option B is not optimal because AWS DMS with CDC adds extra cost and operational overhead for a simple hourly incremental load, and it is not directly integrated with Glue. Option C can work but is less reliable if timestamps are not updated on changes or if there are late-arriving records, and it may require additional indexing. Option D is inefficient because full table scans each hour are costly and slow, especially for large tables.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✓
Use Glue job bookmarks to track and process only new and updated records
Why this is correct
Bookmarks efficiently handle incremental loads.
- ✗
Use AWS DMS with change data capture (CDC) to replicate changes to S3
Why it's wrong here
DMS adds cost and overhead for hourly CDC.
- ✗
Add a timestamp column and query rows where timestamp > last run
Why it's wrong here
May miss updates if timestamp not updated on change.
- ✗
Perform a full table scan each hour and compare with previous snapshot
Why it's wrong here
Full scans are resource-intensive and slow.
Go deeper
Related to this question
About these practice questions
Courseiva writes every DEA-C01 question from scratch — 1,711 in total, each with an explanation and a wrong-answer breakdown. None are copied from real exams or dumps. Learn why practice questions differ from exam dumps →
Same concept, more angles
1 more way this is tested on DEA-C01
These questions test the same concept from different angles. Work through them to make sure you can recognise it however the exam phrases it.
Variation 1. A data engineer is designing a data pipeline that ingests JSON files from an S3 bucket, transforms them using AWS Glue, and loads into Amazon Redshift. The data is updated daily, and the pipeline must handle late-arriving data from the previous day. Which approach minimizes reprocessing?
hard- ✓ A.Use AWS Glue job bookmarks to process only new files based on S3 event notifications.
- B.Stream data using Amazon Kinesis Data Firehose to Redshift.
- C.Enable S3 versioning and process only the latest version of each object.
- D.Schedule a full reload of all data from S3 to Redshift each day.
Why A: AWS Glue job bookmarks track previously processed files and process only new or changed files, which handles late-arriving data without reprocessing all data. Option B uses Amazon Kinesis Data Firehose to stream data to Redshift; this is designed for real-time streaming, not a batch pipeline with daily updates, and does not inherently handle late-arriving data without custom logic. Option C (S3 versioning) can manage multiple versions but does not provide incremental processing for late-arriving data; it would require custom logic to determine which version to process. Option D (scheduling a full reload) would reprocess all data daily, which is inefficient and does not handle late-arriving data efficiently.
JA
Written by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
This DEA-C01 practice question is part of Courseiva's free Amazon Web Services certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the DEA-C01 exam.