AI0-001 AI Concepts and Techniques Practice Question
A model trained on customer reviews achieves 98% accuracy on the test set. However, when deployed, it performs poorly on real-world data. The data scientist suspects distribution shift. Which action is MOST important to address this?
⚠ Common exam trap
CompTIA often tests the misconception that high test accuracy guarantees real-world performance, leading candidates to focus on training improvements (like tuning hyperparameters or adding features) rather than addressing the root cause of distribution shift through monitoring and retraining.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
Implement a monitoring system to detect data drift and retrain with fresh data
Distribution shift (data drift) causes the model's training distribution to differ from the real-world distribution, degrading performance despite high test accuracy. Implementing a monitoring system to detect drift and retraining with fresh data directly addresses this by ensuring the model adapts to the current data distribution, which is the most critical action for maintaining performance in production.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✗
Reduce the learning rate during training
Why it's wrong here
Learning rate governs optimisation convergence, not the input distribution the model encounters; a lower rate cannot reconcile training and deployment data. Tuning it is tempting when training loss is unstable or the model overfits, but distribution shift requires detecting and correcting the mismatch, for example through drift monitoring or retraining on current data.
- ✓
Implement a monitoring system to detect data drift and retrain with fresh data
Why this is correct
Continuous monitoring detects when production input distributions diverge from training data, triggering retraining on fresh samples. This directly addresses distribution shift by closing the feedback loop between deployed predictions and current data, restoring performance that static evaluation on the original test set cannot capture.
- ✗
Add more features to the model
Why it's wrong here
Adding features cannot correct a covariate or concept shift between training and deployment distributions; it risks overfitting the original sample further. Feature engineering is tempting because it raises accuracy when the training data genuinely lacks predictive signal, but here the test set already scores 98%, so the gap is distributional, not representational.
- ✗
Increase the number of cross-validation folds
Why it's wrong here
Cross-validation folds estimate model performance on held-out splits of the same training distribution, so they cannot correct a mismatch between training and deployment data. It is tempting because more folds appear to improve rigour, but cross-validation addresses variance in evaluation, not distribution shift.
About these practice questions
Courseiva writes every AI0-001 question from scratch — 962 in total, each with an explanation and a wrong-answer breakdown. None are copied from real exams or dumps. Learn why practice questions differ from exam dumps →
JA
Written by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
This AI0-001 practice question is part of Courseiva's free CompTIA certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the AI0-001 exam.