MLS-C01 Modeling Practice Question
A company is building a binary classification model to predict customer churn. The dataset has 10,000 samples with 500 churners (positive class). The data scientist trains a logistic regression model and obtains an accuracy of 95%. However, the model predicts all customers as non-churn. Which metric should the data scientist use to evaluate the model's performance?
⚠ Common exam trap
AWS often tests the trap that candidates choose accuracy because it is high (95%), failing to recognize that accuracy is meaningless in imbalanced datasets when the model predicts only the majority class.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
F1-score
The F1-score is the harmonic mean of precision and recall, making it robust to class imbalance. Since the model predicts all customers as non-churn (accuracy 95% due to 9500 non-churners), precision for the positive class is undefined (0 true positives) and recall is 0, so the F1-score correctly reveals the model's failure to identify any churners.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✗
AUC-ROC
Why it's wrong here
AUC-ROC would be 0.5 for a constant model, indicating no discriminative power, but it doesn't directly highlight the failure to detect positives.
- ✓
F1-score
Why this is correct
F1-score balances precision and recall; with all negatives predicted, recall is 0, so F1 is 0, clearly showing poor performance on churners.
- ✗
Accuracy
Why it's wrong here
Accuracy is high but misleading because the model predicts all non-churn, giving 95% accuracy while missing all churners.
- ✗
Confusion matrix
Why it's wrong here
A confusion matrix provides detailed breakdown but no single threshold-free metric; the question asks for a metric to evaluate performance.
Go deeper
Related to this question
About these practice questions
One of 1,672 original MLS-C01 practice questions on Courseiva, each with a full explanation and wrong-answer analysis — not exam dumps or protected exam content. Learn why practice questions differ from exam dumps →
JA
Written by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
This MLS-C01 practice question is part of Courseiva's free Amazon Web Services certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the MLS-C01 exam.