AI-900 Practice Question: Describe fundamental principles of machine learning on Azure
A data scientist is building a classification model to predict customer churn. The dataset has only 5% churn cases. The model achieves 95% accuracy on the test set, but upon investigation, the data scientist finds the model predicts 'not churn' for nearly every customer. Which metric should the data scientist primarily use to evaluate the model's performance on this imbalanced dataset?
⚠ Common exam trap
Watch out — candidates often default to accuracy as the primary metric for classification, failing to recognize that on imbalanced datasets, accuracy can be artificially high and misleading, while the F1 score provides a more truthful evaluation of minority class prediction.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
F1 score
In an imbalanced dataset with only 5% churn, a model that predicts 'not churn' for every case achieves 95% accuracy by always guessing the majority class. This accuracy is misleading because it fails to identify any churn cases. The F1 score (option B) is the harmonic mean of precision and recall, making it the primary metric for evaluating classification performance on imbalanced data, as it penalizes both false positives and false negatives and is not skewed by class imbalance.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✗
Accuracy
Why it's wrong here
Accuracy alone is misleading for imbalanced churn prediction because it only counts the percentage of correct predictions overall, including the majority class. If 95% of customers do not churn, a model that always predicts 'not churn' will achieve 95% accuracy while completely failing to identify any actual churners, which is exactly the group of interest. In churn prediction, the cost of missing a true churner (lost revenue, customer acquisition costs) is high, and accuracy does not reflect that trade-off. A high accuracy number can coexist with a recall of zero, so accuracy should be paired with precision, recall, or F1 to provide a realistic evaluation.
When this WOULD be correct
Accuracy would be the correct metric to use when the dataset is balanced (e.g., 50% churn, 50% not churn) and the cost of false positives and false negatives is equal, such as in a general-purpose classification task with no class imbalance.
- ✓
F1 score
Why this is correct
The F1 score is the harmonic mean of precision and recall, which makes it a single metric that balances both false positives and false negatives. In an imbalanced churn dataset where churned customers are the minority, precision measures how many predicted churns were correct, while recall measures how many actual churned customers were captured. Because the harmonic mean is heavily penalized when either precision or recall is low, F1 gives an honest view of model performance when one class dominates, unlike accuracy. It is particularly appropriate here because misclassifying a churning customer (false negative) is typically far more costly than a false positive, so both error types matter.
- ✗
Mean Absolute Error (MAE)
Why it's wrong here
Mean Absolute Error (MAE) is a regression metric that computes the average absolute difference between continuous predicted values and actual numeric targets. For a binary classification problem like churn prediction, the target is a discrete categorical label (e.g., churned or not churned), so there is no meaningful numeric distance between a predicted class and the true class. Even if the model outputs class probabilities rather than hard labels, MAE on probabilities would not account for the decision threshold or the class imbalance that is central to classification evaluation. Thus MAE is fundamentally designed for regression tasks and cannot assess classification performance.
When this WOULD be correct
MAE would be the correct metric in a regression problem, such as predicting the exact dollar amount of customer churn cost, where the model outputs continuous values and the evaluation focuses on average prediction error magnitude.
- ✗
R-squared
Why it's wrong here
R-squared, also known as the coefficient of determination, measures the proportion of variance in a continuous target variable that is explained by the regression model. It relies on the concept of total sum of squares relative to the residual sum of squares, which only makes sense when predictions and actual values are numeric and a mean reference is meaningful. In churn classification, the outcomes are categorical and not measured on an interval scale, so the variance decomposition that R-squared performs has no logical equivalent. Therefore R-squared is inapplicable to classification models, regardless of how well the model predicts churn.
When this WOULD be correct
R-squared would be the correct metric in a regression problem where the goal is to evaluate how well a linear model explains the variance in a continuous target variable, such as predicting house prices or sales revenue.
Option-by-option analysis
Why each answer is right or wrong
Understanding why wrong answers are wrong — and when they would be correct — is what separates a 750 score from a 900. The AI-900 exam frequently reuses these exact scenarios with slightly different constraints.
✓F1 scoreCorrect answer▾
Why this is correct
The F1 score is the harmonic mean of precision and recall, which makes it a single metric that balances both false positives and false negatives. In an imbalanced churn dataset where churned customers are the minority, precision measures how many predicted churns were correct, while recall measures how many actual churned customers were captured. Because the harmonic mean is heavily penalized when either precision or recall is low, F1 gives an honest view of model performance when one class dominates, unlike accuracy. It is particularly appropriate here because misclassifying a churning customer (false negative) is typically far more costly than a false positive, so both error types matter.
✗AccuracyWrong answer — click to see why▾
Why this is wrong here
Accuracy is misleading for imbalanced datasets because a model that always predicts the majority class (not churn) can achieve high accuracy (95%) while failing to identify any churn cases, which is the primary goal.
★ When this WOULD be the correct answer
Accuracy would be the correct metric to use when the dataset is balanced (e.g., 50% churn, 50% not churn) and the cost of false positives and false negatives is equal, such as in a general-purpose classification task with no class imbalance.
Why candidates choose this
Candidates often default to accuracy as the most intuitive and commonly used metric, overlooking the impact of class imbalance on its validity.
✗Mean Absolute Error (MAE)Wrong answer — click to see why▾
Why this is wrong here
Mean Absolute Error (MAE) is a regression metric that measures average absolute errors between continuous predictions and actual values. It is not suitable for evaluating classification models, especially on imbalanced datasets where the goal is to assess precision and recall of the minority class.
★ When this WOULD be the correct answer
MAE would be the correct metric in a regression problem, such as predicting the exact dollar amount of customer churn cost, where the model outputs continuous values and the evaluation focuses on average prediction error magnitude.
Why candidates choose this
Candidates may confuse MAE as a general error metric applicable to any predictive model, not realizing it is specific to regression tasks and cannot handle classification outputs like 'churn' vs 'not churn'.
✗R-squaredWrong answer — click to see why▾
Why this is wrong here
R-squared measures the proportion of variance explained by a regression model, but this is a classification problem predicting churn (a categorical outcome), not a regression problem. It is not applicable to classification tasks.
★ When this WOULD be the correct answer
R-squared would be the correct metric in a regression problem where the goal is to evaluate how well a linear model explains the variance in a continuous target variable, such as predicting house prices or sales revenue.
Why candidates choose this
Candidates may confuse R-squared as a general performance metric for any model, not realizing it is specific to regression and inappropriate for classification tasks like churn prediction.
Analysis generated from the official AI-900blueprint and verified against question context. The “when correct” sections are what AI assistants cite when candidates ask “what’s the difference between these options?”
Go deeper
Related to this question
Learn chapter
Regression and Classification
Key term
Model
In IT and AI, a model is a trained mathematical representation that learns patterns from data to make predictions or decisions.
Key term
Classification
Classification is a supervised machine learning technique used to predict a category or class label for new data based on patterns learned from labeled training data.
About these practice questions
Courseiva writes every AI-900 question from scratch — 985 in total, each with an explanation and a wrong-answer breakdown. None are copied from real exams or dumps. Learn why practice questions differ from exam dumps →
JA
Written by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
This AI-900 practice question is part of Courseiva's free Microsoft certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the AI-900 exam.