MLS-C01 Modeling Practice Question
A company uses Amazon SageMaker to deploy a model that predicts customer churn. The model is retrained weekly. The data scientist notices that the model's accuracy remains high, but the business reports that the model is not capturing new churn patterns. What is the most likely cause?
⚠ Common exam trap
The trap here is that candidates see 'accuracy remains high' and assume the model is overfitting or underfitting, but the key clue is 'not capturing new churn patterns'—which points to a shift in the underlying data distribution (concept drift), not a static model fit issue.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
The model is experiencing concept drift
Concept drift occurs when the statistical properties of the target variable change over time, causing the model's predictions to become less relevant even if accuracy metrics remain high. In this scenario, the model is retrained weekly but still fails to capture new churn patterns because the underlying customer behavior has shifted—a classic sign of concept drift rather than a data or overfitting issue. Amazon SageMaker's built-in Model Monitor can detect such drift by comparing inference data distributions against a baseline.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✗
The model is underfitting the data
Why it's wrong here
Underfitting would show low accuracy on both training and test sets.
- ✗
The model has data leakage from future data
Why it's wrong here
Data leakage would give overly optimistic accuracy, but the model is not capturing new patterns.
- ✗
The model is overfitting to the training data
Why it's wrong here
Overfitting would cause poor performance on new data, but the accuracy remains high, suggesting the test set may be outdated.
- ✓
The model is experiencing concept drift
Why this is correct
Concept drift means the underlying data distribution changes, so the model's accuracy on old patterns remains high but it misses new patterns.
Go deeper
Related to this question
About these practice questions
Courseiva writes every MLS-C01 question from scratch — 1,672 in total, each with an explanation and a wrong-answer breakdown. None are copied from real exams or dumps. Learn why practice questions differ from exam dumps →
JA
Written by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
This MLS-C01 practice question is part of Courseiva's free Amazon Web Services certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the MLS-C01 exam.