AI0-001 Implementing AI Solutions Practice Question
During the evaluation phase of an AI project, the team measures the model's F1 score on a held-out test set. They find the F1 score is 0.92, but when deployed in production, the model performs poorly on new data. What is the MOST likely cause of this discrepancy?
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
Data leakage occurred between the training and test sets during preparation
Data leakage during preparation can cause overly optimistic evaluation scores. If the test set contains information from the training set, the model appears better than it really is. Overfitting is possible but less likely with a proper hold-out. Concept drift occurs over time, not immediately. Poor hyperparameter tuning usually yields lower scores, not inflated ones.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✗
The production data has a different distribution than the training data (concept drift)
Why it's wrong here
Concept drift typically occurs over time, not immediately upon deployment; the question describes poor performance right away.
- ✗
The model's hyperparameters were not properly tuned
Why it's wrong here
Improper tuning usually leads to lower evaluation scores, not high ones, so this does not explain the discrepancy.
- ✗
The model is overfitting to the training data
Why it's wrong here
Overfitting would typically show a large gap between training and validation scores, but here the test score is high, suggesting leakage or distribution shift.
- ✓
Data leakage occurred between the training and test sets during preparation
Why this is correct
Data leakage artificially inflates evaluation metrics; the model may have seen test data during training, leading to a false sense of performance.
About these practice questions
Courseiva writes every AI0-001 question from scratch — 754 in total, each with an explanation and a wrong-answer breakdown. None are copied from real exams or dumps. Learn why practice questions differ from exam dumps →
JA
Written by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
This AI0-001 practice question is part of Courseiva's free CompTIA certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the AI0-001 exam.