Fixing Glue Crawler Missing Partitions with Partition Indexing
A company uses AWS Glue crawlers to populate the AWS Glue Data Catalog from Amazon S3. The data is partitioned by year/month/day/hour. The crawler runs every hour and adds new partitions. However, the data engineer notices that the crawler is taking longer to run as the number of partitions grows, and sometimes it misses new partitions. What is the most cost-effective and reliable way to address this?
Quick Answer
The answer is enabling the crawler’s partition index feature, because it allows AWS Glue to efficiently locate new partitions in Amazon S3 without re-scanning the entire table metadata. When a Glue crawler runs hourly against deeply partitioned data by year/month/day/hour, the growing number of partitions causes scanning overhead that leads to missed partitions; partition indexing creates a lightweight index that the crawler queries first, drastically reducing runtime and ensuring all new partitions are captured reliably. On the AWS Certified Machine Learning Specialty MLS-C01 exam, this scenario tests your understanding of Glue Data Catalog optimization for large-scale ML pipelines, where cost-effective automation is critical—a common trap is choosing to increase crawler frequency, which only worsens the scanning bottleneck, or relying on manual commands like MSCK REPAIR TABLE that break automation. Memory tip: think of partition indexing as a “table of contents” for your S3 folders—the crawler reads the index instead of flipping through every page.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
Enable the crawler's partition index feature.
Enabling the crawler's partition index feature allows AWS Glue to quickly find new partitions without re-scanning the entire table, reducing runtime and improving reliability. Option B is incorrect because manually adding partitions is error-prone and does not scale. Option C is incorrect because the MSCK REPAIR TABLE command is a manual step that does not automate the process. Option D is incorrect because increasing the crawler's frequency does not address the underlying issue of scanning overhead and may lead to resource contention.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✓
Enable the crawler's partition index feature.
Why this is correct
Partition indexes allow the crawler to efficiently discover new partitions without scanning the entire dataset.
- ✗
Manually add new partitions using ALTER TABLE ADD PARTITION in Athena.
Why it's wrong here
Manual partition addition does not scale and requires constant monitoring.
- ✗
Use the Athena MSCK REPAIR TABLE command after the crawler runs.
Why it's wrong here
MSCK REPAIR TABLE is a manual operation and does not solve the underlying issue of the crawler missing partitions.
- ✗
Increase the crawler's schedule to run every 30 minutes.
Why it's wrong here
More frequent runs will increase cost and may still miss partitions if the crawler cannot keep up.
Quick reference
AWS S3 Storage Class Comparison
| Storage Class | Min Duration | Retrieval | Use Case |
|---|---|---|---|
| S3 Standard | None | Immediate | Frequently accessed data |
| S3 Standard-IA | 30 days | Immediate | Infrequent access, rapid retrieval |
| S3 One Zone-IA | 30 days | Immediate | Non-critical infrequent data |
| S3 Intelligent-Tiering | None | Immediate–hours | Unknown or changing access patterns |
| S3 Glacier Instant | 90 days | Milliseconds | Archive with instant retrieval |
| S3 Glacier Flexible | 90 days | Minutes–hours | Archive, flexible retrieval |
| S3 Glacier Deep Archive | 180 days | Hours | Long-term compliance archive |
Go deeper
Related to this question
About these practice questions
Courseiva writes every MLS-C01 question from scratch — 1,672 in total, each with an explanation and a wrong-answer breakdown. None are copied from real exams or dumps. Learn why practice questions differ from exam dumps →
Same concept, more angles
2 more ways this is tested on MLS-C01
These questions test the same concept from different angles. Work through them to make sure you can recognise it however the exam phrases it.
Variation 1. A company uses AWS Glue to catalog data in S3. Data is partitioned by year, month, day. The Glue crawler runs daily but sometimes misses new partitions. What should be done to ensure all partitions are cataloged?
medium- A.Use a custom classifier to detect partition patterns.
- B.Increase the crawler schedule to run every hour.
- ✓ C.Configure the crawler to update all partitions on each run.
- D.Enable partition indexing in the Glue table properties.
Why C: Configuring the Glue crawler to 'update all partitions on each run' forces the crawler to scan the entire S3 path and register any new partitions it finds, even if the partition structure hasn't changed. This ensures all missed partitions are cataloged. Partition indexing (D) improves query performance by creating an index over existing partitions, but does not automatically discover new partitions.
Variation 2. A company is using AWS Glue to catalog metadata from various data sources. The crawler is configured to run daily. However, the catalog is not reflecting new partitions added to an S3 bucket during the day. What is the MOST likely cause?
medium- A.The S3 bucket has insufficient permissions for the Glue crawler
- B.The table schema has changed and the crawler does not update it
- ✓ C.The crawler is not scheduled frequently enough to capture changes
- D.The data format is not supported by AWS Glue
Why C: The crawler is configured to run daily, but new partitions are being added to the S3 bucket throughout the day. Since the crawler only runs once per day, it will not detect and catalog those new partitions until its next scheduled run. To capture changes more frequently, the crawler schedule should be increased or an event-driven trigger (e.g., using Amazon S3 Events and AWS Lambda) should be implemented.
JA
Written by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
This MLS-C01 practice question is part of Courseiva's free Amazon Web Services certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the MLS-C01 exam.