Choosing Evaluation Metrics for Imbalanced Classification
A data scientist is examining a dataset for a binary classification problem. The target variable has a 1:1000 imbalance. Which technique should be used to assess model performance during exploratory data analysis?
Quick Answer
The answer is the Area Under the Precision-Recall curve (AUPRC). This metric is the correct choice because with a 1:1000 class imbalance, the positive class is extremely rare, and AUPRC focuses exclusively on the minority class by evaluating the trade-off between precision and recall. Unlike ROC AUC, which can appear deceptively high when the majority class dominates, AUPRC is sensitive to changes in how well the model identifies the few positive instances, making it the most robust evaluation metric for imbalanced classification. On the AWS Certified Machine Learning Specialty MLS-C01 exam, this concept tests your understanding of when to avoid ROC AUC for skewed targets; a common trap is assuming ROC AUC works universally. Remember the memory tip: "When positives are scarce, PR curves are fair—ROC lies when negatives outnumber the prize."
⚠ Common exam trap
It's easy for candidates to default to ROC AUC as the universal metric for classification, not realizing that in extreme imbalance, ROC AUC can be misleadingly high because the false positive rate is diluted by the vast number of true negatives.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
Area under the Precision-Recall curve
With a 1:1000 class imbalance, the positive class is extremely rare. The Area Under the Precision-Recall curve (AUPRC) focuses on the performance of the positive class and is sensitive to changes in precision and recall, making it a robust metric for imbalanced datasets. Unlike ROC AUC, which can be overly optimistic when negatives dominate, AUPRC provides a realistic assessment of model performance on the minority class.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✓
Area under the Precision-Recall curve
Why this is correct
PR AUC is sensitive to class imbalance and focuses on the positive class.
- ✗
F1 score
Why it's wrong here
F1 score is threshold-dependent and may not capture overall performance across thresholds.
- ✗
Area under the ROC curve
Why it's wrong here
ROC AUC can be overly optimistic with extreme imbalance.
- ✗
Cohen's kappa
Why it's wrong here
Cohen's kappa accounts for chance but is less standard for imbalance.
Go deeper
Related to this question
About these practice questions
Courseiva writes every MLS-C01 question from scratch — 1,672 in total, each with an explanation and a wrong-answer breakdown. None are copied from real exams or dumps. Learn why practice questions differ from exam dumps →
Same concept, more angles
1 more way this is tested on MLS-C01
These questions test the same concept from different angles. Work through them to make sure you can recognise it however the exam phrases it.
Variation 1. A machine learning engineer trains a binary classifier on an imbalanced dataset where the positive class represents 1% of the data. After training, the model achieves 99% accuracy but only 10% recall on the positive class. Which metric should the engineer focus on to evaluate the model's performance on the minority class?
medium- ✓ A.F1 score
- B.Accuracy
- C.AUC-ROC
- D.Precision
Why A: (F1 score) is the correct metric because it balances precision and recall, providing a single measure that is robust to class imbalance. With only 1% positive class, accuracy (Option B) is misleadingly high due to the majority class. AUC-ROC (Option C) can still be high even if recall is low, as it evaluates ranking rather than absolute performance. Precision (Option D) only considers the proportion of correct positive predictions, ignoring false negatives, which is not suitable when recall is poor. The F1 score captures both aspects, making it the best choice for evaluating minority class performance in this scenario.
JA
Written by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
This MLS-C01 practice question is part of Courseiva's free Amazon Web Services certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the MLS-C01 exam.