AI0-001 AI Implementation and Operations Practice Question
A large e-commerce company has deployed a real-time product recommendation system using a neural collaborative filtering model. The model was trained on six months of user click and purchase data. For the first three months after deployment, the click-through rate (CTR) improved by 15%. However, starting in the fourth month, CTR began decreasing steadily despite no changes to the system infrastructure or data pipeline. The product manager suspects model decay but the engineering team insists the model is static and should not degrade. The data science lead suggests investigating further. They have access to production logs, A/B testing framework, and historical model versions. What is the BEST course of action to diagnose and address the issue?
⚠ Common exam trap
CompTIA often tests the principle that diagnosing model decay requires a controlled comparison (A/B test) rather than immediately retraining or adding features, and the trap here is assuming that a static model cannot degrade when the underlying data distribution changes.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
Set up an A/B test comparing the current model against the original baseline model using recent traffic.
Setting up an A/B test comparing the current model against the original baseline model using recent traffic directly isolates whether the model's predictive performance has degraded due to concept drift (changes in user behavior over time). Since the model is static but the data distribution has shifted, the A/B test provides empirical evidence of decay by measuring CTR differences under identical conditions, which is the standard diagnostic step before any retraining or feature engineering.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✗
Re-deploy the model with additional features such as time of day and user device.
Why it's wrong here
Adding features without understanding the drop may not address decay and could complicate diagnosis.
- ✗
Increase the frequency of batch inference from hourly to every 10 minutes to improve responsiveness.
Why it's wrong here
Inference frequency is unrelated to model relevance; it does not address underlying performance decline.
- ✓
Set up an A/B test comparing the current model against the original baseline model using recent traffic.
Why this is correct
A/B testing isolates whether the current model underperforms relative to a known good version, confirming decay.
- ✗
Retrain the model on only the most recent 30 days of data and replace the current model.
Why it's wrong here
Retraining on a short window may cause overfitting to recent trends and lacks baseline comparison.
About these practice questions
This AI0-001 question is part of Courseiva's 754-question bank — original exam-style content with full explanations and wrong-answer analysis, never real exam questions or exam dumps. Learn why practice questions differ from exam dumps →
JA
Written by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
This AI0-001 practice question is part of Courseiva's free CompTIA certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the AI0-001 exam.