AIF-C01 Fundamentals of AI and ML Practice Question
A data scientist is training a binary classification model to predict customer churn. The dataset has 10,000 records with 9,500 non-churners and 500 churners. After training a logistic regression model, the model achieves 95% accuracy on the test set. However, the business team reports that the model is not useful because it predicts almost all customers as non-churners. Which metric should the data scientist use to evaluate the model's performance in this scenario?
⚠ Common exam trap
The AIF-C01 exam often tests the misconception that high accuracy always indicates a good model, especially in imbalanced datasets, leading candidates to overlook metrics like recall or precision that better reflect model utility for the specific business problem.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
Recall
(Recall) is correct because in this highly imbalanced dataset (95% non-churners vs 5% churners), the model's 95% accuracy is misleading—it can achieve this by simply predicting the majority class (non-churner) for all samples. Recall measures the proportion of actual churners correctly identified (True Positives / (True Positives + False Negatives)), directly addressing the business need to detect churn. A high recall ensures the model captures most churners, even at the cost of some false positives.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✗
Accuracy
Why it's wrong here
Accuracy counts all correct predictions, so predicting every customer as a non-churner yields 95% on this imbalanced set while catching zero churners. Accuracy is appropriate only when classes are roughly balanced and both error types carry similar cost.
- ✗
R-squared
Why it's wrong here
R-squared quantifies how much variance a regression line explains in continuous outcomes, so it cannot be computed meaningfully for a binary churn label. It would be the right metric when evaluating a linear regression predicting a numeric target such as customer lifetime value.
- ✗
Precision
Why it's wrong here
Precision measures only the proportion of predicted churners that truly churn, so a model predicting almost everything as non-churners can still score highly on the few positive predictions it makes, ignoring the missed churners entirely. Precision suits spam filtering, where false positives are the main cost.
- ✓
Recall
Why this is correct
Recall measures the proportion of actual churners correctly identified, directly exposing the model's failure on the 500-positive minority class. With 95% accuracy achievable by predicting all non-churners, recall reveals that sensitivity to churn is near zero, satisfying the need for a metric robust to this class imbalance.
Go deeper
Related to this question
About these practice questions
Courseiva writes every AIF-C01 question from scratch — 862 in total, each with an explanation and a wrong-answer breakdown. None are copied from real exams or dumps. Learn why practice questions differ from exam dumps →
JA
Written by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
This AIF-C01 practice question is part of Courseiva's free Amazon Web Services certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the AIF-C01 exam.