MLS-C01 Practice Question: Machine Learning Implementation and Operations
A data scientist is using Amazon SageMaker Autopilot to automatically build a binary classification model. After the Autopilot job completes, the best model has an accuracy of 0.85 on the validation set. However, the data scientist notices a class imbalance (90% negative, 10% positive). Which metric should the data scientist use to evaluate the model's performance on the positive class?
⚠ Common exam trap
Many candidates default to accuracy as the primary metric, failing to recognize that class imbalance renders accuracy misleading, and they overlook AUC's threshold-agnostic property which is specifically designed for evaluating model performance on the minority class in imbalanced datasets.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
Area Under the ROC Curve (AUC)
A is correct because Area Under the ROC Curve (AUC) is threshold-independent and evaluates the model's ability to distinguish between positive and negative classes across all classification thresholds. In the presence of severe class imbalance (90% negative, 10% positive), AUC provides a robust measure of model performance on the positive class without being skewed by the majority class, unlike accuracy which would be high even if the model predicts all negatives.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✓
Area Under the ROC Curve (AUC)
Why this is correct
AUC is robust to class imbalance and evaluates overall ranking performance.
- ✗
Recall
Why it's wrong here
Recall alone does not consider false positives.
- ✗
Accuracy
Why it's wrong here
Accuracy can be high even if the model ignores the positive class due to imbalance.
- ✗
Precision
Why it's wrong here
Precision alone does not reflect how many positives are captured.
Go deeper
Related to this question
About these practice questions
Courseiva writes every MLS-C01 question from scratch — 1,672 in total, each with an explanation and a wrong-answer breakdown. None are copied from real exams or dumps. Learn why practice questions differ from exam dumps →
JA
Written by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
This MLS-C01 practice question is part of Courseiva's free Amazon Web Services certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the MLS-C01 exam.