Evaluation Metrics for Imbalanced Binary Classification
Which TWO metrics are appropriate for evaluating a binary classification model trained on imbalanced data? (Select TWO.)
Quick Answer
The answer is the F1 score, along with the Area Under the Precision-Recall Curve (AUC-PR), as the two most appropriate evaluation metrics for imbalanced binary classification. The F1 score is the harmonic mean of precision and recall, making it robust when the positive class is rare because it penalizes models that simply predict the majority class for all instances, unlike accuracy which can be misleadingly high. On the AWS Certified Machine Learning Specialty MLS-C01 exam, this concept tests your ability to select metrics that avoid the "accuracy paradox"—where a model appears strong by always guessing the majority class. A common trap is choosing accuracy or ROC-AUC, which can overestimate performance on severely imbalanced data. For a memory tip, think "F1 for the few": when the positive class is the minority, F1 and AUC-PR focus on how well you catch those rare, critical cases.
⚠ Common exam trap
The MLS-C01 exam often tests the misconception that ROC-AUC is always the best metric for imbalanced data, but the trap here is that ROC-AUC can be misleadingly high when the positive class is rare, whereas precision-recall curve and F1 score better reflect model performance on the minority class.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
F1 score
The F1 score is appropriate for imbalanced binary classification because it balances precision and recall, making it robust when the positive class is rare. Unlike accuracy, it does not get inflated by a majority negative class, and it directly penalizes models that predict the majority class for all instances.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✗
Log loss
Why it's wrong here
Log loss is not specifically for imbalance.
- ✓
F1 score
Why this is correct
F1 balances precision and recall.
- ✗
Accuracy
Why it's wrong here
Accuracy is misleading for imbalanced data.
- ✓
Precision-recall curve
Why this is correct
PR curve is sensitive to imbalance.
- ✗
ROC-AUC
Why it's wrong here
ROC-AUC is unsuitable for imbalanced data because its axes, True Positive Rate and False Positive Rate, are insensitive to class distribution. This can lead to an overly optimistic assessment, as a high AUC might mask poor predictive performance on the minority class. It is tempting because ROC-AUC is a widely recognised metric for binary classification, providing an aggregate measure of performance across all classification thresholds. It would be an appropriate choice for evaluating models on balanced datasets or where the costs of false positives and false negatives are symmetrical.
Go deeper
Related to this question
About these practice questions
Courseiva writes every MLS-C01 question from scratch — 1,672 in total, each with an explanation and a wrong-answer breakdown. None are copied from real exams or dumps. Learn why practice questions differ from exam dumps →
Same concept, more angles
2 more ways this is tested on MLS-C01
These questions test the same concept from different angles. Work through them to make sure you can recognise it however the exam phrases it.
Variation 1. Which TWO metrics are appropriate for evaluating a binary classification model when the cost of false negatives is high?
medium- A.Accuracy
- B.AUC-ROC
- ✓ C.Recall
- ✓ D.F1 score
- E.Precision
Why C: When false negatives are costly, we want to minimize them, so recall (true positive rate) is important. Precision is also important to avoid too many false positives, but F1 score balances both. Recall directly measures false negatives, and F1 combines precision and recall. AUC-ROC is a general measure, and accuracy can be misleading. Therefore, the two appropriate metrics are Recall (option C) and F1 score (option D).
Variation 2. Which TWO metrics are appropriate for evaluating a binary classification model when the cost of false negatives is high? (Choose 2)
easy- ✓ A.Recall
- B.Specificity
- C.Accuracy
- D.F1 score
- E.Precision
Why A: When the cost of false negatives is high, the primary goal is to minimize false negatives by capturing as many actual positives as possible. Recall (Option A) directly measures this by calculating TP/(TP+FN). Specificity (Option B) measures TN/(TN+FP) and focuses on false positives, not false negatives; it is irrelevant when FN cost is high. Accuracy (C) can be misleading with imbalanced data. F1 score (D) balances precision and recall but does not specifically target FN. Precision (E) focuses on false positives, not false negatives. Therefore, only Recall is appropriate.
JA
Written by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
This MLS-C01 practice question is part of Courseiva's free Amazon Web Services certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the MLS-C01 exam.