Quick Answer
The answer is the F1 score and AUC-ROC. This combination is critical for evaluation metrics for imbalanced classification because accuracy becomes a misleading metric when the positive class represents only 1% of the data—a model can achieve 99% accuracy by simply predicting the majority class, as seen with the 10% recall. The F1 score balances precision and recall, directly measuring the minority class performance, while AUC-ROC evaluates the model’s discriminative power across all thresholds, making it robust to severe imbalance. On the AWS Certified Machine Learning Specialty MLS-C01 exam, this scenario tests your understanding of how standard metrics fail under class skew and your ability to select threshold-independent and harmonic metrics. A common trap is to rely on accuracy or precision alone; instead, remember that when the positive class is rare, F1 and AUC-ROC are your go-to pair. Memory tip: “F1 for the few, AUC for the view”—F1 focuses on the minority class, AUC gives the full picture across thresholds.
This MLS-C01 practice question tests your understanding of modeling. The scenario asks you to isolate a root cause — eliminate options that address a different problem before choosing. After answering, compare your reasoning against the explanation and wrong-answer breakdown below. Once you have made your selection, read the full explanation to reinforce the concept and understand why each distractor is designed to mislead on exam day.
A data scientist is training a binary classifier on an imbalanced dataset where the positive class represents 1% of the data. The model currently achieves 99% accuracy but a recall of only 10% on the positive class. Which metric combination should the data scientist prioritize to evaluate model improvements?
F1 score and AUC-ROC
F1 score balances precision and recall; AUC-ROC is robust to imbalance.
Precision and recall at 90% precision
Why wrong: Focusing only on precision may ignore recall.
Accuracy and RMSE
Why wrong: Accuracy is misleading for imbalanced data; RMSE is for regression.
Precision and RMSE
Why wrong: RMSE is not suitable for classification.
Answer choices
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
F1 score and AUC-ROC
With a highly imbalanced dataset (1% positive class), 99% accuracy is misleading because the model can achieve it by simply predicting the majority class. The low recall (10%) indicates the model fails to identify most positive instances. The F1 score balances precision and recall, providing a single metric for minority class performance, while AUC-ROC evaluates the model's ability to distinguish between classes across all thresholds, making it robust to class imbalance.
Key principle: Answer the scenario, not the keyword: identify the specific constraint before choosing the most familiar-sounding option.
Answer analysis
For each option: why learners choose it and why it is or isn't the right answer here.
F1 score and AUC-ROC
Why this is correct
F1 score balances precision and recall; AUC-ROC is robust to imbalance.
Related concept
Read the scenario before looking for a memorised answer.
Precision and recall at 90% precision
Why it's wrong here
Focusing only on precision may ignore recall.
Accuracy and RMSE
Why it's wrong here
Accuracy is misleading for imbalanced data; RMSE is for regression.
Precision and RMSE
Why it's wrong here
RMSE is not suitable for classification.
Common exam traps
The trap here is that candidates see 99% accuracy and assume the model is good, failing to recognize that accuracy is a poor metric for imbalanced datasets, and that metrics like RMSE are for regression, not classification.
Detailed technical explanation
AUC-ROC measures the area under the receiver operating characteristic curve, which plots the true positive rate (recall) against the false positive rate at various thresholds. It is insensitive to class distribution because it evaluates ranking quality, not absolute probabilities. The F1 score is the harmonic mean of precision and recall, and it penalizes extreme imbalances between the two; for a model with 99% accuracy but 10% recall, the F1 score would be low (e.g., ~0.18 if precision is also low), clearly indicating poor minority class performance.
Key takeaway
Answer the scenario, not the keyword: identify the specific constraint before choosing the most familiar-sounding option.
Real-world example
A cloud solutions architect for a retail company is evaluating services for a new workload. The correct answer here reflects best practice for the specific scenario described — not a general cloud recommendation. Answer the scenario, not the keyword: identify the specific constraint before choosing the most familiar-sounding option. Cloud exam questions reward reading the constraint carefully: the same technology can be right or wrong depending on the use case.
What to study next
Identify which exam domain this question belongs to, review the core concept, then practise similar questions from the same domain.
Modeling — study guide chapter
Learn the concepts, then practise the questions
Modeling practice questions
Targeted practice on this topic area only
All MLS-C01 questions
1,755 questions across all exam domains
AWS Certified Machine Learning Specialty MLS-C01 study guide
Full concept coverage aligned to exam objectives
MLS-C01 practice test guide
How to use practice tests most effectively before exam day
Related practice questions
Use these pages to review the topic behind this question. This is how one missed question becomes focused revision.
Practise MLS-C01 questions linked to Data Engineering.
Practise MLS-C01 questions linked to Machine Learning Implementation and Operations.
Practise MLS-C01 questions linked to Modeling.
Practise MLS-C01 questions linked to Exploratory Data Analysis.
Practise MLS-C01 questions linked to MLS-C01 fundamentals.
Practise MLS-C01 questions linked to MLS-C01 scenario.
Practise MLS-C01 questions linked to MLS-C01 troubleshooting.
Practice this exam
Short sessions build daily habit. Longer sessions build exam-day stamina. Try a timed session to simulate real conditions.
FAQ
Modeling — This question tests Modeling — Read the scenario before looking for a memorised answer..
The correct answer is: F1 score and AUC-ROC — With a highly imbalanced dataset (1% positive class), 99% accuracy is misleading because the model can achieve it by simply predicting the majority class. The low recall (10%) indicates the model fails to identify most positive instances. The F1 score balances precision and recall, providing a single metric for minority class performance, while AUC-ROC evaluates the model's ability to distinguish between classes across all thresholds, making it robust to class imbalance.
Identify which exam domain this question belongs to, review the core concept, then practise similar questions from the same domain.
Read the scenario before looking for a memorised answer.
About these practice questions
Courseiva creates original exam-style practice questions with explanations and wrong-answer analysis. It does not publish real exam questions, exam dumps, or protected exam content. Learn why practice questions differ from exam dumps →
Same concept, more angles
These questions test the same concept from different angles. Work through them to make sure you can recognise it however the exam phrases it.
Variation 1. A data scientist is training a binary classification model on a highly imbalanced dataset where the positive class represents only 1% of the data. The model achieves 99% accuracy but only identifies 5% of the actual positives. Which metric should the data scientist use to evaluate model performance?
easyWhy C: Recall (sensitivity) measures the proportion of actual positives correctly identified by the model. With only 5% of positives detected, recall is 0.05, which directly reveals the model's failure to capture the minority class despite high accuracy. In imbalanced datasets, accuracy is misleading because the model can achieve 99% accuracy by simply predicting the majority class (negative) for all instances.
Variation 2. A data scientist is training a binary classification model on a highly imbalanced dataset where the positive class represents only 1% of the data. Which metric should be used to evaluate model performance during training to ensure the model is learning to detect the positive class?
easyWhy A: Accuracy is misleading for imbalanced datasets because a model that predicts the majority class all the time can achieve 99% accuracy. F1 score balances precision and recall, making it suitable for imbalanced classification. Precision, recall, and AUC are also useful, but F1 is a common single metric for imbalanced binary classification. Option A: Accuracy is not suitable. Option B: Precision alone ignores recall. Option C: F1 score is correct. Option D: Recall alone ignores precision.
Variation 3. A data scientist is training a binary classification model on a highly imbalanced dataset (99% negative class, 1% positive class). The model currently achieves 99% accuracy but only identifies 0.5% of true positives. Which metric should the data scientist focus on to improve model performance?
easyWhy C: Recall (sensitivity) measures the proportion of actual positives correctly identified, which is critical when the dataset is highly imbalanced (99% negative, 1% positive) and the model fails to detect most positives (only 0.5% true positives). Improving recall directly addresses the model's inability to capture the minority class, even if it reduces precision or accuracy. In binary classification with severe class imbalance, accuracy is misleading because a model can achieve 99% accuracy by simply predicting the majority class, as seen here.
Variation 4. A data scientist is training a binary classifier on an imbalanced dataset (95% negative, 5% positive). The model achieves 99% accuracy but only correctly identifies 2% of the positive samples. Which metric should the data scientist focus on to improve the model's performance?
easyWhy C: Option B is correct because recall measures the proportion of actual positives correctly identified, which is critical for imbalanced datasets. Option A is wrong because accuracy is misleading when classes are imbalanced. Option C is wrong because RMSE is for regression. Option D is wrong because precision does not directly address the low identification of positives.
Keep practising
Last reviewed: Jun 11, 2026
This MLS-C01 practice question is part of Courseiva's free Amazon Web Services certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the MLS-C01 exam.
Question Discussion
Share a tip, memory trick, or ask about the reasoning behind this question. Do not post real exam questions, leaked content, braindumps, or copyrighted exam material. Comments are moderated and may be removed without notice.
Sign in to join the discussion.