MLS-C01 Modeling Practice Question
A data scientist is using Amazon SageMaker Autopilot to automatically build a model for a regression problem. The dataset has 100 features and 50,000 rows. Autopilot recommends a model with an R² of 0.85 on the validation set. However, when deployed to production, the model performs poorly (R² of 0.2). What is the most likely cause?
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
The production data distribution has shifted from the training data distribution
A large discrepancy between validation and production performance often indicates data drift. Option A (overfitting) is possible but less likely given validation performance. Option C (feature importance) is not the direct cause. Option D (Autopilot bug) is rare.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✗
The model is overfitting to the training data
Why it's wrong here
Validation R² is 0.85, which suggests good generalization to validation set.
- ✓
The production data distribution has shifted from the training data distribution
Why this is correct
Data drift causes model performance to degrade in production.
- ✗
The model is underfitting due to insufficient training
Why it's wrong here
Underfitting would show low R² on validation too.
- ✗
Autopilot selected the wrong features
Why it's wrong here
Autopilot performs feature engineering; wrong features would likely affect validation as well.
Go deeper
Related to this question
About these practice questions
One of 1,672 original MLS-C01 practice questions on Courseiva, each with a full explanation and wrong-answer analysis — not exam dumps or protected exam content. Learn why practice questions differ from exam dumps →
JA
Written by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
This MLS-C01 practice question is part of Courseiva's free Amazon Web Services certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the MLS-C01 exam.