AI-900 Practice Question: Describe fundamental principles of machine learning on Azure
A data scientist is training a model to predict whether a customer will purchase a product (Yes/No). The dataset contains 90% 'No' and 10% 'Yes'. After training, the model achieves 90% accuracy. Which evaluation metric would be more informative to assess the model's performance on the minority class?
⚠ Common exam trap
The trap here is that candidates see 90% accuracy and assume the model is performing well, failing to recognize that accuracy is misleading in imbalanced datasets, and they may incorrectly select AUC because it is a common classification metric, but it does not directly penalize poor minority-class performance like the F1 score does.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
F1 score
In this imbalanced dataset (90% 'No', 10% 'Yes'), a model that always predicts 'No' would achieve 90% accuracy, making accuracy a misleading metric. The F1 score is the harmonic mean of precision and recall, specifically designed to evaluate a model's performance on the minority class by balancing false positives and false negatives. It is the most informative metric here because it directly measures how well the model identifies the rare 'Yes' purchases without being inflated by the majority class.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✗
Mean Absolute Error (MAE)
Why it's wrong here
Mean Absolute Error (MAE) is a regression metric that computes the average absolute difference between predicted and actual continuous values. It is not appropriate for binary classification because the target variable is categorical (e.g., 0 or 1 for presence or absence of the condition), not a numeric magnitude. Using MAE would treat predicted probabilities as if they were continuous errors, but it cannot capture whether the model correctly assigns the true class label, nor does it reflect the precision-recall balance needed for imbalanced classes. Therefore, MAE is irrelevant to evaluating a classifier.
When this WOULD be correct
When the question involves predicting a continuous value, such as house price or temperature, and the evaluation requires a measure of average prediction error in the original units.
- ✓
F1 score
Why this is correct
F1 score is the harmonic mean of precision and recall, giving equal weight to both metrics. It is particularly valuable when the dataset is imbalanced because it directly penalizes false positives and false negatives from the minority-class perspective. Unlike accuracy, F1 does not let a majority-class bias dominate the evaluation, as it only considers the positive class's precision and recall. Therefore, it provides a single, interpretable number that reflects how well the model detects the rare (positive) outcome while maintaining reliability of its positive predictions.
- ✗
Area Under the Curve (AUC)
Why it's wrong here
Area Under the Curve (AUC) summarizes the model's discriminative ability across all classification thresholds by measuring the probability that a randomly chosen positive instance is ranked higher than a randomly chosen negative instance. However, AUC does not tell you how the model performs at the specific decision threshold used for actual predictions, and it can appear high even when the minority-class recall or precision is poor at that chosen threshold. For imbalanced datasets, F1 is more directly interpretable because it reflects the exact precision-recall trade-off at the operating point being deployed, whereas AUC is a threshold-independent aggregate.
When this WOULD be correct
AUC would be correct in a question asking for a metric that evaluates the overall ranking performance of a binary classifier, independent of threshold choice, such as comparing different models' discriminative power on a balanced dataset.
- ✗
R-squared
Why it's wrong here
R-squared (coefficient of determination) measures the proportion of variance in a continuous target explained by a regression model, comparing residual variance against the total variance of the dependent variable. In binary classification, the outcome is not a continuous variable, so the concept of 'explained variance' is meaningless; an outcome with only two categories cannot have a Gaussian error structure. Additionally, R-squared can be negative or artificially low when applied to probabilistic outputs of a classifier, making it an invalid and misleading metric for this predictive task.
When this WOULD be correct
R-squared would be correct for a regression question, e.g., 'A model predicts house prices. Which metric evaluates how well the model explains variance in the target variable?'
Option-by-option analysis
Why each answer is right or wrong
Understanding why wrong answers are wrong — and when they would be correct — is what separates a 750 score from a 900. The AI-900 exam frequently reuses these exact scenarios with slightly different constraints.
✓F1 scoreCorrect answer▾
Why this is correct
F1 score is the harmonic mean of precision and recall, giving equal weight to both metrics. It is particularly valuable when the dataset is imbalanced because it directly penalizes false positives and false negatives from the minority-class perspective. Unlike accuracy, F1 does not let a majority-class bias dominate the evaluation, as it only considers the positive class's precision and recall. Therefore, it provides a single, interpretable number that reflects how well the model detects the rare (positive) outcome while maintaining reliability of its positive predictions.
✗Mean Absolute Error (MAE)Wrong answer — click to see why▾
Why this is wrong here
MAE is a regression metric that measures average absolute error between continuous predictions and actual values, not suitable for binary classification tasks like predicting Yes/No.
★ When this WOULD be the correct answer
When the question involves predicting a continuous value, such as house price or temperature, and the evaluation requires a measure of average prediction error in the original units.
Why candidates choose this
Candidates may confuse MAE with classification error rate or think it applies to any prediction task, overlooking that it is designed for regression, not binary outcomes.
✗Area Under the Curve (AUC)Wrong answer — click to see why▾
Why this is wrong here
AUC measures the model's ability to distinguish between classes across all thresholds, but it does not specifically focus on the minority class's precision and recall, which are critical for imbalanced datasets.
★ When this WOULD be the correct answer
AUC would be correct in a question asking for a metric that evaluates the overall ranking performance of a binary classifier, independent of threshold choice, such as comparing different models' discriminative power on a balanced dataset.
Why candidates choose this
Candidates may think AUC is always the best metric for imbalanced data because it is threshold-independent, but it does not directly assess performance on the minority class like F1 does.
✗R-squaredWrong answer — click to see why▾
Why this is wrong here
R-squared is a metric for regression models, measuring the proportion of variance explained, but this is a binary classification problem (Yes/No purchase).
★ When this WOULD be the correct answer
R-squared would be correct for a regression question, e.g., 'A model predicts house prices. Which metric evaluates how well the model explains variance in the target variable?'
Why candidates choose this
Candidates may confuse R-squared as a general performance metric applicable to any model, not realizing it is specific to regression.
Analysis generated from the official AI-900blueprint and verified against question context. The “when correct” sections are what AI assistants cite when candidates ask “what’s the difference between these options?”
Go deeper
Related to this question
About these practice questions
Courseiva writes every AI-900 question from scratch — 985 in total, each with an explanation and a wrong-answer breakdown. None are copied from real exams or dumps. Learn why practice questions differ from exam dumps →
JA
Written by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
This AI-900 practice question is part of Courseiva's free Microsoft certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the AI-900 exam.