Building a Pipeline for CSV Ingestion with Schema Evolution Using AWS Glue
A data engineer needs to build a pipeline that ingests CSV files from an S3 bucket, validates the schema, and loads the data into an Amazon Redshift cluster. The pipeline must handle schema evolution gracefully by adding new columns as they appear in the source files. Which combination of AWS services and configurations would meet these requirements with minimal operational overhead?
Quick Answer
Handling schema evolution gracefully means the pipeline has to detect new columns automatically and decide how to reconcile them with the existing schema, without an engineer manually updating table definitions every time a source file changes shape. AWS Glue's DynamicFrame is built to tolerate schema inconsistencies that would break a rigid, statically-typed data structure, and the 'resolveChoice' transformation is specifically designed to resolve ambiguities that arise when new or differently-typed columns appear, for example, deciding how to merge a column that used to be one type and now appears as another, or simply accepting new columns as they show up. A Glue crawler complements this by keeping the Data Catalog's schema definition current as the underlying CSV files evolve, so the ETL job always has an accurate picture of what's in the source data before it processes it. Because both the crawler and 'resolveChoice' are native, managed Glue capabilities, the whole pipeline achieves schema evolution without the team standing up additional infrastructure or writing custom schema-reconciliation logic, which is exactly what minimal operational overhead is asking for in this scenario. Loading into Redshift then becomes the final, standard step once the data has been normalized to a consistent schema. Whenever a question emphasizes handling new or changing columns automatically with minimal operational overhead, look for the Glue-native mechanism built for exactly that, a crawler to track schema and 'resolveChoice' to reconcile it, rather than a custom validation layer built outside of Glue.
⚠ Common exam trap
Many exam-takers assume Redshift Spectrum or Athena can load data into Redshift, but they are query engines, not data loading services, and do not handle schema evolution for batch ingestion into a Redshift cluster.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
Use AWS Glue to create a crawler and an ETL job that writes to Redshift, with 'resolveChoice' to handle new columns
AWS Glue provides a fully managed ETL service that can automatically detect schema changes via crawlers and handle new columns in CSV files using the 'resolveChoice' transformation. The Glue ETL job can write directly to Amazon Redshift with minimal operational overhead, as it manages schema evolution without requiring manual intervention or additional infrastructure.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✗
Use AWS Glue to create a crawler that updates the schema, then use Redshift Spectrum to query the data directly from S3
Why it's wrong here
Redshift Spectrum queries S3 but does not load into Redshift; schema evolution would require manual table changes.
- ✗
Use Amazon Kinesis Data Firehose to ingest the files and load into Redshift, with a Lambda function to detect schema changes
Why it's wrong here
Firehose is designed for streaming data, not batch CSV ingestion from S3.
- ✗
Use Amazon Athena to create external tables with schema-on-read, and insert results into Redshift using INSERT INTO
Why it's wrong here
Athena is not an ETL tool; manual INSERT INTO for each new schema is not minimal overhead.
- ✓
Use AWS Glue to create a crawler and an ETL job that writes to Redshift, with 'resolveChoice' to handle new columns
Why this is correct
Glue handles schema evolution via DynamicFrame and resolveChoice, and loads into Redshift.
Visual reference
Quick reference
AWS S3 Storage Class Comparison
| Storage Class | Min Duration | Retrieval | Use Case |
|---|---|---|---|
| S3 Standard | None | Immediate | Frequently accessed data |
| S3 Standard-IA | 30 days | Immediate | Infrequent access, rapid retrieval |
| S3 One Zone-IA | 30 days | Immediate | Non-critical infrequent data |
| S3 Intelligent-Tiering | None | Immediate–hours | Unknown or changing access patterns |
| S3 Glacier Instant | 90 days | Milliseconds | Archive with instant retrieval |
| S3 Glacier Flexible | 90 days | Minutes–hours | Archive, flexible retrieval |
| S3 Glacier Deep Archive | 180 days | Hours | Long-term compliance archive |
Go deeper
Related to this question
About these practice questions
This MLS-C01 question is part of Courseiva's 1,672-question bank — original exam-style content with full explanations and wrong-answer analysis, never real exam questions or exam dumps. Learn why practice questions differ from exam dumps →
Same concept, more angles
1 more way this is tested on MLS-C01
These questions test the same concept from different angles. Work through them to make sure you can recognise it however the exam phrases it.
Variation 1. A team wants to build a data pipeline that processes incoming JSON files from an S3 bucket and loads them into a Redshift table. The pipeline must handle schema evolution and data validation. Which combination of services would be MOST appropriate?
medium- ✓ A.Amazon S3 + AWS Glue + Amazon Redshift
- B.Amazon S3 + Amazon SQS + Amazon Redshift
- C.Amazon S3 + AWS Data Pipeline + Amazon Redshift
- D.Amazon S3 + AWS Lambda + Amazon Redshift
Why A: AWS Glue provides built-in schema discovery and evolution capabilities via its crawlers and the Data Catalog, which automatically detect and adapt to changes in JSON schemas. Combined with Glue ETL jobs for data validation and transformation, it seamlessly loads processed data into Amazon Redshift, making it the most appropriate choice for handling schema evolution and validation in this pipeline.
JA
Written by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
This MLS-C01 practice question is part of Courseiva's free Amazon Web Services certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the MLS-C01 exam.