MLS-C01 Modeling Practice Question
A data scientist is training a binary classifier on an imbalanced dataset where the positive class represents 1% of the data. The model currently achieves 99% accuracy but a recall of only 10% on the positive class. Which metric combination should the data scientist prioritize to evaluate model improvements?
⚠ Common exam trap
The trap here is that candidates see 99% accuracy and assume the model is good, failing to recognize that accuracy is a poor metric for imbalanced datasets, and that metrics like RMSE are for regression, not classification.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
F1 score and AUC-ROC
With a highly imbalanced dataset (1% positive class), 99% accuracy is misleading because the model can achieve it by simply predicting the majority class. The low recall (10%) indicates the model fails to identify most positive instances. The F1 score balances precision and recall, providing a single metric for minority class performance, while AUC-ROC evaluates the model's ability to distinguish between classes across all thresholds, making it robust to class imbalance.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✓
F1 score and AUC-ROC
Why this is correct
F1 score balances precision and recall; AUC-ROC is robust to imbalance.
- ✗
Precision and recall at 90% precision
Why it's wrong here
Focusing only on precision may ignore recall.
- ✗
Accuracy and RMSE
Why it's wrong here
Accuracy is misleading for imbalanced data; RMSE is for regression.
- ✗
Precision and RMSE
Why it's wrong here
RMSE is not suitable for classification.
Go deeper
Related to this question
About these practice questions
One of 1,672 original MLS-C01 practice questions on Courseiva, each with a full explanation and wrong-answer analysis — not exam dumps or protected exam content. Learn why practice questions differ from exam dumps →
Same concept, more angles
4 more ways this is tested on MLS-C01
These questions test the same concept from different angles. Work through them to make sure you can recognise it however the exam phrases it.
Variation 1. A data scientist is training a binary classification model on a highly imbalanced dataset where the positive class represents only 1% of the data. The model achieves 99% accuracy but only identifies 5% of the actual positives. Which metric should the data scientist use to evaluate model performance?
easy- A.Mean squared error
- B.Accuracy
- ✓ C.Recall
- D.Precision
Why C: Recall (sensitivity) measures the proportion of actual positives correctly identified by the model. With only 5% of positives detected, recall is 0.05, which directly reveals the model's failure to capture the minority class despite high accuracy. In imbalanced datasets, accuracy is misleading because the model can achieve 99% accuracy by simply predicting the majority class (negative) for all instances.
Variation 2. A data scientist is training a binary classification model on a highly imbalanced dataset where the positive class represents only 1% of the data. Which metric should be used to evaluate model performance during training to ensure the model is learning to detect the positive class?
easy- ✓ A.F1 score
- B.Accuracy
- C.Precision
- D.Recall
Why A: Accuracy is misleading for imbalanced datasets because a model that predicts the majority class all the time can achieve 99% accuracy. F1 score balances precision and recall, making it suitable for imbalanced classification. Precision, recall, and AUC are also useful, but F1 is a common single metric for imbalanced binary classification. Option A: F1 score correctly balances precision and recall. Option B: Accuracy is not suitable. Option C: Precision alone ignores recall. Option D: Recall alone ignores precision.
Variation 3. A data scientist is training a binary classification model on a highly imbalanced dataset (99% negative class, 1% positive class). The model currently achieves 99% accuracy but only identifies 0.5% of true positives. Which metric should the data scientist focus on to improve model performance?
easy- A.Precision
- B.Root Mean Squared Error (RMSE)
- ✓ C.Recall
- D.Accuracy
Why C: Recall (sensitivity) measures the proportion of actual positives correctly identified, which is critical when the dataset is highly imbalanced (99% negative, 1% positive) and the model fails to detect most positives (only 0.5% true positives). Improving recall directly addresses the model's inability to capture the minority class, even if it reduces precision or accuracy. In binary classification with severe class imbalance, accuracy is misleading because a model can achieve 99% accuracy by simply predicting the majority class, as seen here.
Variation 4. A data scientist is training a binary classifier on an imbalanced dataset (95% negative, 5% positive). The model achieves 99% accuracy but only correctly identifies 2% of the positive samples. Which metric should the data scientist focus on to improve the model's performance?
easy- A.Precision
- B.RMSE
- ✓ C.Recall
- D.Accuracy
Why C: (Recall). Recall measures the proportion of actual positive samples correctly identified, which is critical for an imbalanced dataset where the model fails to detect positives. Option A (Precision) is not the primary focus because it measures the accuracy of positive predictions, not the ability to find all positives. Option B (RMSE) is a regression metric, not suitable for binary classification. Option D (Accuracy) is misleading because a model can achieve high accuracy by simply predicting the majority class, as seen here.
JA
Written by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
This MLS-C01 practice question is part of Courseiva's free Amazon Web Services certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the MLS-C01 exam.