Describe fundamental principles of machine learning on Azure →mediumMultiple ChoiceObjective-mapped

AI-900 Practice Question: Describe fundamental principles of machine learning on Azure

A data scientist trains a model on historical data and achieves high accuracy on both the training set and a held-out test set. However, when the model is deployed in production, it performs poorly on new, unseen data. Which issue is most likely the cause?

⚠ Common exam trap

A common mix-up: candidates confuse high accuracy on both training and test sets with overfitting, but the key differentiator is that overfitting would show a significant gap between training and test accuracy, whereas data leakage produces deceptively high accuracy on both sets.

Answer choices

Why each option matters

Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.

Correct answer & explanation

✓

Data leakage

Data leakage occurs when information from outside the training dataset is inadvertently used to train the model, causing it to learn patterns that do not generalize to new data. In this scenario, the high accuracy on both training and test sets but poor production performance indicates that the test set was contaminated with information from the future or from the target variable, making the model appear accurate during validation but fail in real-world deployment.

Answer analysis

Option-by-option breakdown

For each option: why learners choose it and why it is or isn't the right answer here.

✗
Overfitting
Why it's wrong here
Overfitting is characterized by a large gap between training and validation/test performance: the model memorizes noise and idiosyncrasies of the training set, so its in-sample accuracy is very high while out-of-sample accuracy drops sharply. The scenario reports high accuracy on both training and test sets, which is the opposite of the expected generalization gap, so overfitting cannot explain the observed metrics.
✗
Underfitting
Why it's wrong here
Underfitting occurs when the model is too simple or has insufficient training to capture the underlying relationship, resulting in poor predictive performance on both the training set and the test set. Because the question states that the model achieves high accuracy on both historical and evaluation data, underfitting is directly contradicted by the evidence, since underfit models struggle even to fit the data they were trained on.
✓
Data leakage
Why this is correct
Data leakage is the correct diagnosis: it happens when the training features contain information that would not be available at inference time, such as the target itself, a future value, or a post-outcome field. The model exploits this hidden shortcut, achieving very high accuracy on both training and test splits during offline evaluation, but in real-world deployment the leaked signal is absent, causing immediate and drastic performance collapse. This pattern of artificially perfect historical evaluation followed by severe production failure is the classic signature of data leakage.
✗
Concept drift
Why it's wrong here
Concept drift describes a change in the underlying data distribution or the relationship between features and labels over time, which typically causes model performance to degrade gradually as the world evolves. In the given scenario, the model fails drastically and suddenly upon deployment, and the historical evaluation was suspiciously high; concept drift would not create that instantly perfect-then-broken pattern, especially without a time-dependent decline, making data leakage a far more accurate explanation.

Go deeper

AI-900 Practice Question: Describe fundamental principles of machine learning on Azure

Why each option matters

Option-by-option breakdown

Related to this question