AI-900 Practice Question: Describe fundamental principles of machine learning on Azure
A company builds a machine learning model to predict whether a customer will purchase a product. They use a training dataset with 50% purchasers and 50% non-purchasers. The model achieves 90% accuracy on the test set. However, when deployed, the model performs poorly because the actual customer base has only 5% purchasers. What is the most likely cause of this poor performance?
⚠ Common exam trap
It's easy for candidates to confuse high accuracy on a balanced test set with real-world readiness, failing to recognize that accuracy is misleading when class distributions shift dramatically between training and production.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
The training and deployment data have different distributions.
The model was trained on a balanced dataset (50% purchasers, 50% non-purchasers) but deployed on a real-world dataset with only 5% purchasers. This mismatch in class distribution between training and deployment data causes the model to fail, as it learned decision boundaries optimized for balanced classes. This is a classic case of distribution shift, specifically prior probability shift, which invalidates the model's assumptions about the target variable's base rate.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✗
The model is overfitted to the training data.
Why it's wrong here
Overfitting means the model performs well on training but poorly on unseen data from the same distribution. In this scenario, the test set likely mirrored the training distribution (balanced), and accuracy was high, so overfitting is not the primary issue.
When this WOULD be correct
A model achieves 99% accuracy on training data but only 70% on a test set drawn from the same distribution. The model memorized noise or specific patterns in the training data, failing to generalize to unseen data from the same source.
- ✗
The model is underfitted and fails to capture key patterns.
Why it's wrong here
Underfitting would manifest as low accuracy on both the training and test sets, not just in deployment. The model's 90% test accuracy indicates it captured the patterns in the balanced training distribution well, so it is not underfitting. Underfitting cannot explain why the model performs well on the test set but poorly on deployment data; the mismatch stems from the fact that the deployment data has a different distribution (5% purchasers) than the training/test data (50% purchasers), which is a dataset shift problem, not a model capacity problem.
When this WOULD be correct
A model trained on a dataset with complex patterns achieves low accuracy on both training and test sets (e.g., 60% on a binary classification task). This indicates the model is too simple to capture underlying relationships, making underfitting the likely cause.
- ✗
Data leakage caused inflated accuracy during testing.
Why it's wrong here
Data leakage is not the culprit here because there is no evidence that target information or future data contaminated the training features. Leakage would inflate both training and test accuracy by giving the model information it shouldn't have, but the problem described is a drop in real-world performance relative to test performance, which suggests the test set was not representative of deployment. The balanced test set accurately reflected the training distribution, and the model performed well on that distribution, but the deployment distribution is different—so the issue is distribution shift, not leakage.
When this WOULD be correct
A model achieves 99% accuracy on the test set but performs poorly in production. Investigation reveals that the test set contained future data (e.g., time-based leakage) or features that indirectly reveal the target (e.g., customer ID). In that scenario, data leakage is the cause.
- ✓
The training and deployment data have different distributions.
Why this is correct
This is correct. The training set was artificially balanced at 50% purchasers and 50% non-purchasers, while the production deployment has only a 5% purchase rate. This represents a prior probability shift, a form of dataset shift, which changes the optimal decision threshold and the model's predicted probabilities become miscalibrated for the real-world base rate. As a result, the model's high 90% test accuracy, measured on the balanced distribution, does not transfer to the deployment distribution where the class balance is drastically different.
Option-by-option analysis
Why each answer is right or wrong
Understanding why wrong answers are wrong — and when they would be correct — is what separates a 750 score from a 900. The AI-900 exam frequently reuses these exact scenarios with slightly different constraints.
✓The training and deployment data have different distributions.Correct answer▾
Why this is correct
This is correct. The training set was artificially balanced at 50% purchasers and 50% non-purchasers, while the production deployment has only a 5% purchase rate. This represents a prior probability shift, a form of dataset shift, which changes the optimal decision threshold and the model's predicted probabilities become miscalibrated for the real-world base rate. As a result, the model's high 90% test accuracy, measured on the balanced distribution, does not transfer to the deployment distribution where the class balance is drastically different.
✗The model is overfitted to the training data.Wrong answer — click to see why▾
Why this is wrong here
Overfitting would cause high accuracy on training data but poor generalization to new data from the same distribution. Here, the poor performance is due to a shift in class distribution (50% purchasers in training vs 5% in deployment), not overfitting.
★ When this WOULD be the correct answer
A model achieves 99% accuracy on training data but only 70% on a test set drawn from the same distribution. The model memorized noise or specific patterns in the training data, failing to generalize to unseen data from the same source.
Why candidates choose this
Candidates often attribute any performance drop after deployment to overfitting, without considering that the data distribution itself has changed (covariate shift or class imbalance shift).
✗The model is underfitted and fails to capture key patterns.Wrong answer — click to see why▾
Why this is wrong here
Underfitting would cause poor performance on both training and test sets, but here the model achieved 90% accuracy on the test set, indicating it captured patterns well. The issue is a mismatch between training and deployment data distributions, not insufficient model complexity.
★ When this WOULD be the correct answer
A model trained on a dataset with complex patterns achieves low accuracy on both training and test sets (e.g., 60% on a binary classification task). This indicates the model is too simple to capture underlying relationships, making underfitting the likely cause.
Why candidates choose this
Candidates may assume poor deployment performance is due to the model not learning enough, especially when the training data is balanced but real-world data is imbalanced, confusing underfitting with distribution shift.
✗Data leakage caused inflated accuracy during testing.Wrong answer — click to see why▾
Why this is wrong here
Data leakage would cause inflated accuracy on both training and test sets, but here the test set accuracy (90%) is consistent with the training distribution (50% purchasers), not the deployment distribution (5% purchasers). The poor performance is due to distribution shift, not leakage.
★ When this WOULD be the correct answer
A model achieves 99% accuracy on the test set but performs poorly in production. Investigation reveals that the test set contained future data (e.g., time-based leakage) or features that indirectly reveal the target (e.g., customer ID). In that scenario, data leakage is the cause.
Why candidates choose this
Candidates may confuse high test accuracy with overfitting or leakage, especially when the model fails in production. They might think that any gap between test and real-world performance is due to leakage, without considering distribution shift.
Analysis generated from the official AI-900blueprint and verified against question context. The “when correct” sections are what AI assistants cite when candidates ask “what’s the difference between these options?”
Go deeper
Related to this question
Learn chapter
Machine Learning Core Concepts
Key term
Model
In IT and AI, a model is a trained mathematical representation that learns patterns from data to make predictions or decisions.
Key term
Machine learning
Machine learning is a branch of artificial intelligence where computers learn patterns from data to make decisions or predictions without being explicitly programmed for every task.
About these practice questions
This AI-900 question is part of Courseiva's 985-question bank — original exam-style content with full explanations and wrong-answer analysis, never real exam questions or exam dumps. Learn why practice questions differ from exam dumps →
JA
Written by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
This AI-900 practice question is part of Courseiva's free Microsoft certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the AI-900 exam.