F1-Score: The Go-To Metric for Imbalanced Datasets
A data scientist trains a binary classification model to distinguish between images of cats and dogs. On the test set, the model achieves 98% accuracy, but a deeper inspection reveals that the test set contains 95% cats and 5% dogs, and the model predicts 'cat' for every single image. Which metric should the data scientist prioritize to get a more realistic evaluation of the model's performance on this imbalanced dataset?
Quick Answer
The answer is the F1-score, which is the correct metric to prioritize for imbalanced classification. This metric combines precision and recall into a single harmonic mean, making it far more reliable than accuracy when one class significantly outnumbers another. In the cat and dog scenario, the model’s 98% accuracy is misleading because it simply predicts the majority class every time, achieving zero recall on the minority class (dogs). The F1-score penalizes this failure by dropping sharply, revealing the model’s true inability to distinguish between classes. On the Microsoft Azure AI Fundamentals AI-900 exam, this concept tests your understanding of evaluation metrics for imbalanced datasets, a common trap where high accuracy hides poor performance. A key memory tip: think of F1 as the “balance beam” that forces both precision and recall to be high—if either is low, the score falls.
⚠ Common exam trap
The trap here is that candidates see 98% accuracy and assume the model is performing well, failing to recognize that accuracy is meaningless on imbalanced datasets where the model can achieve high accuracy by simply predicting the majority class.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
F1-score
The F1-score is the harmonic mean of precision and recall, providing a single metric that balances both when classes are imbalanced. In this scenario, accuracy is misleadingly high (98%) because the model always predicts the majority class (cat), achieving high accuracy without actually learning to distinguish cats from dogs. The F1-score penalizes the model for its poor recall on the minority class (dogs), giving a more realistic evaluation of its performance.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✗
Precision
Why it's wrong here
Precision measures the proportion of positive identifications that were actually correct. While useful, it does not consider false negatives, so it can still be high if the model rarely predicts the minority class, as in this scenario.
When this WOULD be correct
When the cost of false positives is high, such as in spam detection where marking a legitimate email as spam is unacceptable, precision is the key metric to minimize false positives.
- ✗
Recall
Why it's wrong here
Recall measures the proportion of actual positives that were identified correctly. In this case, the model has zero recall for the dog class because it never predicts 'dog', so recall alone would be 0 for dogs but 100% for cats, giving an incomplete picture.
When this WOULD be correct
In a scenario where the cost of missing positive cases is high (e.g., detecting rare diseases), recall is prioritized. For example, if a model must identify all instances of a rare cancer, even at the expense of false positives, recall would be the key metric.
- ✓
F1-score
Why this is correct
The F1-score combines precision and recall into a single metric that penalizes extreme values. For this model, the F1-score for the minority class (dogs) would be very low, revealing the poor performance that accuracy hides.
- ✗
Accuracy
Why it's wrong here
Accuracy is the ratio of correct predictions to total predictions. In this highly imbalanced dataset, the model achieves 95% accuracy just by always guessing the majority class, which does not reflect its inability to detect dogs.
When this WOULD be correct
Accuracy is the correct metric when the dataset is balanced (e.g., equal number of cats and dogs) and the cost of false positives and false negatives is equal. For example, in a question where the test set has 50% cats and 50% dogs, accuracy would be appropriate.
Option-by-option analysis
Why each answer is right or wrong
Understanding why wrong answers are wrong — and when they would be correct — is what separates a 750 score from a 900. The AI-900 exam frequently reuses these exact scenarios with slightly different constraints.
✓F1-scoreCorrect answer▾
Why this is correct
The F1-score combines precision and recall into a single metric that penalizes extreme values. For this model, the F1-score for the minority class (dogs) would be very low, revealing the poor performance that accuracy hides.
✗PrecisionWrong answer — click to see why▾
Why this is wrong here
Precision measures the proportion of positive predictions that are correct, but in this case the model predicts 'cat' for all images, so precision for 'cat' is 95% (since 95% of images are cats), which is misleadingly high and does not reveal the model's failure to detect dogs.
★ When this WOULD be the correct answer
When the cost of false positives is high, such as in spam detection where marking a legitimate email as spam is unacceptable, precision is the key metric to minimize false positives.
Why candidates choose this
Candidates may think precision is important because it focuses on correct positive predictions, but they overlook that in this imbalanced dataset, the model's high precision for the majority class masks its poor performance on the minority class.
✗RecallWrong answer — click to see why▾
Why this is wrong here
Recall measures the proportion of actual positives correctly identified, but here the model predicts all images as 'cat' (the majority class), so recall for 'dog' (the minority class) is 0%, which is poor. However, recall alone does not account for precision, and in this imbalanced scenario, the F1-score (harmonic mean of precision and recall) provides a more balanced evaluation.
★ When this WOULD be the correct answer
In a scenario where the cost of missing positive cases is high (e.g., detecting rare diseases), recall is prioritized. For example, if a model must identify all instances of a rare cancer, even at the expense of false positives, recall would be the key metric.
Why candidates choose this
Candidates may think recall is important because the model fails to detect dogs (the minority class), but they overlook that recall for the majority class (cats) is perfect, and without considering precision, recall alone doesn't reveal the model's trivial prediction strategy.
✗AccuracyWrong answer — click to see why▾
Why this is wrong here
Accuracy is misleading on imbalanced datasets because a model can achieve high accuracy by always predicting the majority class, as in this case where the model predicts 'cat' for all images and achieves 98% accuracy despite having zero predictive power for dogs.
★ When this WOULD be the correct answer
Accuracy is the correct metric when the dataset is balanced (e.g., equal number of cats and dogs) and the cost of false positives and false negatives is equal. For example, in a question where the test set has 50% cats and 50% dogs, accuracy would be appropriate.
Why candidates choose this
Candidates often default to accuracy because it is the most intuitive and commonly used metric, and they may overlook class imbalance or fail to consider that high accuracy can be achieved by a trivial model.
Analysis generated from the official AI-900blueprint and verified against question context. The “when correct” sections are what AI assistants cite when candidates ask “what’s the difference between these options?”
Go deeper
Related to this question
Learn chapter
Machine Learning Core Concepts
Key term
Classification
Classification is a supervised machine learning technique used to predict a category or class label for new data based on patterns learned from labeled training data.
Key term
Model
In IT and AI, a model is a trained mathematical representation that learns patterns from data to make predictions or decisions.
About these practice questions
One of 985 original AI-900 practice questions on Courseiva, each with a full explanation and wrong-answer analysis — not exam dumps or protected exam content. Learn why practice questions differ from exam dumps →
Same concept, more angles
4 more ways this is tested on AI-900
These questions test the same concept from different angles. Work through them to make sure you can recognise it however the exam phrases it.
Variation 1. A data scientist is building a binary classification model to predict fraudulent credit card transactions. The dataset is highly imbalanced: only 1% of transactions are fraudulent. The cost of a false negative is very high because missing a fraudulent transaction can lead to significant financial loss. Which evaluation metric should the data scientist prioritize to minimize false negatives?
medium- A.Accuracy
- B.Precision
- ✓ C.Recall
- D.F1 Score
Why C: Recall (also known as sensitivity or true positive rate) measures the proportion of actual positive cases (fraudulent transactions) that are correctly identified. In this highly imbalanced scenario where missing a fraud (false negative) is extremely costly, maximizing recall ensures that the model catches as many fraudulent transactions as possible, even if it means some false positives occur. This directly aligns with the goal of minimizing false negatives.
Variation 2. A data scientist has trained a binary classification model to detect fraudulent credit card transactions. The dataset contains 99.9% legitimate transactions and only 0.1% fraudulent ones. The model predicts all transactions as legitimate, achieving 99.9% accuracy on the test set. However, the business requires the model to actually catch as many fraudulent transactions as possible. Which metric would best reveal the model's failure to identify fraud?
medium- A.Accuracy
- ✓ B.Recall
- C.Precision
- D.F1 score
Why B: Recall (also known as sensitivity) measures the proportion of actual positive cases (fraudulent transactions) that were correctly identified by the model. In this scenario, the model predicts all transactions as legitimate, so it correctly identifies 0 out of the 0.1% fraudulent transactions, yielding a recall of 0%. This directly reveals the model's complete failure to catch fraud, despite the high accuracy.
Variation 3. A data scientist is training a binary classification model to detect fraudulent transactions. The dataset contains only 1% fraudulent transactions. The model achieves 99% accuracy on the test set, but when deployed, it fails to detect most actual fraud cases. Which metric would best reveal this issue?
hard- A.Accuracy
- B.Precision
- ✓ C.Recall
- D.F1 score
Why C: Recall (sensitivity) measures the proportion of actual positive cases correctly identified. In this highly imbalanced dataset (1% fraud), a model can achieve 99% accuracy by simply predicting 'non-fraud' for every transaction, which yields zero true positives. Recall reveals this failure because it focuses solely on how many fraudulent transactions were caught, ignoring the vast majority of non-fraud cases.
Variation 4. A data scientist trains a binary classification model to detect fraudulent credit card transactions. The dataset contains 99.5% legitimate transactions and 0.5% fraudulent transactions. The model predicts every transaction as legitimate and achieves 99.5% accuracy on the test set. Which metric would best reveal that the model is failing to identify any fraudulent transactions?
medium- A.Precision
- ✓ B.Recall
- C.F1 score
- D.Mean Absolute Error (MAE)
Why B: Recall (also known as sensitivity) measures the proportion of actual positive cases correctly identified by the model. In this scenario, the model predicts all transactions as legitimate, so it correctly identifies zero fraudulent transactions, giving a recall of 0%. Accuracy alone is misleading because the dataset is highly imbalanced (99.5% legitimate, 0.5% fraudulent), and a 99.5% accuracy can be achieved by simply predicting the majority class. Recall directly reveals the model's failure to detect any fraud.
JA
Written by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
This AI-900 practice question is part of Courseiva's free Microsoft certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the AI-900 exam.