AWS Glue Crawler Configuration: Multiple Formats and Partition Updates
A company is building a data lake on Amazon S3 and wants to use AWS Glue to catalog the data. The data includes CSV, Parquet, and JSON files. The team wants to ensure that the Glue crawler can infer the schema correctly and update the Data Catalog when new partitions are added. Which crawler configuration should be used?
Quick Answer
AWS Glue crawlers are designed to inspect the actual files in a location and infer schema from them, and a single crawler can handle multiple file formats, CSV, Parquet, and JSON here, within the same crawl because it classifies each file individually rather than assuming the whole dataset shares one format. That means there's no need to run separate crawlers per format just to get correct schema inference; one crawler configured against the data lake location is enough to catalog all three formats correctly. The second part of the requirement, keeping the Data Catalog current as new partitions arrive, is handled by the 'Update all new and existing partitions' setting, which tells the crawler to both add newly discovered partitions and refresh the schema of partitions that already exist in the catalog if their structure has changed. Without that setting, a crawler might only register new partitions while leaving stale schema information in place for existing ones, which would cause downstream analytics and ML jobs to see an outdated or incomplete view of the data. Combining a single multi-format crawler with this partition-update behavior keeps the catalog accurate and current with the least manual maintenance, which is exactly what the scenario's requirement for automatic schema inference and catalog freshness calls for. When a Glue-related question asks how to keep a Data Catalog accurate as a data lake evolves with mixed formats and growing partitions, look for the crawler setting that explicitly refreshes both new and existing partition metadata.
⚠ Common exam trap
The trap here is that candidates mistakenly think disabling partition updates prevents schema conflicts, but in reality, it causes stale metadata for existing partitions, while a single crawler with updates enabled correctly handles schema evolution across all file formats.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
Create a single crawler that includes all file extensions and set the 'Update all new and existing partitions' option.
A single AWS Glue crawler can handle multiple file formats (CSV, Parquet, JSON) in a data lake on Amazon S3, and enabling 'Update all new and existing partitions' ensures the Data Catalog is refreshed with both new partitions and any schema changes in existing partitions. This configuration maintains a consistent and up-to-date catalog without manual intervention, which is essential for downstream analytics and machine learning workloads.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✗
Create separate crawlers for each file format and schedule them at different times.
Why it's wrong here
Incorrect: Not necessary; one crawler can handle multiple formats.
- ✗
Use a crawler that only catalogs Parquet files because they are more efficient.
Why it's wrong here
Incorrect: All formats need to be cataloged.
- ✗
Use a crawler with 'Update all new and existing partitions' disabled to avoid schema conflicts.
Why it's wrong here
Incorrect: Disabling update would cause new partitions to be missed.
- ✓
Create a single crawler that includes all file extensions and set the 'Update all new and existing partitions' option.
Why this is correct
Correct: Single crawler with partition updates ensures comprehensive cataloging.
Quick reference
AWS S3 Storage Class Comparison
| Storage Class | Min Duration | Retrieval | Use Case |
|---|---|---|---|
| S3 Standard | None | Immediate | Frequently accessed data |
| S3 Standard-IA | 30 days | Immediate | Infrequent access, rapid retrieval |
| S3 One Zone-IA | 30 days | Immediate | Non-critical infrequent data |
| S3 Intelligent-Tiering | None | Immediate–hours | Unknown or changing access patterns |
| S3 Glacier Instant | 90 days | Milliseconds | Archive with instant retrieval |
| S3 Glacier Flexible | 90 days | Minutes–hours | Archive, flexible retrieval |
| S3 Glacier Deep Archive | 180 days | Hours | Long-term compliance archive |
Go deeper
Related to this question
About these practice questions
Courseiva writes every MLS-C01 question from scratch — 1,672 in total, each with an explanation and a wrong-answer breakdown. None are copied from real exams or dumps. Learn why practice questions differ from exam dumps →
Same concept, more angles
2 more ways this is tested on MLS-C01
These questions test the same concept from different angles. Work through them to make sure you can recognise it however the exam phrases it.
Variation 1. A company wants to build a data lake on Amazon S3. The data lake will store raw data in its original format and also transformed data in Parquet. The data is generated by various sources and must be cataloged for discovery. Which service should the company use to automatically discover, catalog, and make the data searchable?
easy- ✓ A.AWS Glue Data Catalog
- B.Amazon S3
- C.Amazon Athena
- D.Amazon EMR
Why A: AWS Glue Data Catalog is a central metadata repository that can automatically crawl S3 data sources to discover, catalog, and make data searchable. It is the correct choice for building a data lake with raw and transformed data in Parquet. Option B (Amazon S3) is storage, not a catalog. Option C (Amazon Athena) is a query engine, not a catalog. Option D (Amazon EMR) is a big data processing framework, not a catalog.
Variation 2. A team is building a data lake on Amazon S3 and using AWS Glue to catalog data. They notice that Glue crawlers are taking too long to update the catalog for a large dataset with millions of small files. Which approach will MOST improve crawler performance?
hard- A.Increase the frequency of the crawler runs.
- ✓ B.Consolidate the small files into larger files (e.g., 100 MB each).
- C.Partition the data by date in S3.
- D.Use a custom classifier to parse the data.
Why B: AWS Glue crawlers incur significant overhead when processing millions of small files because each file requires a separate read, schema inference, and metadata write operation. Consolidating small files into larger files (e.g., 100 MB each) reduces the total number of objects that the crawler must scan, dramatically decreasing the time spent on file-level operations and improving overall throughput.
JA
Written by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
This MLS-C01 practice question is part of Courseiva's free Amazon Web Services certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the MLS-C01 exam.