AI0-001 AI Implementation and Operations Practice Question
A large e-commerce company has deployed a real-time product recommendation system using a neural collaborative filtering model. The model was trained on six months of user click and purchase data. For the first three months after deployment, the click-through rate (CTR) improved by 15%. However, starting in the fourth month, CTR began decreasing steadily despite no changes to the system infrastructure or data pipeline. The product manager suspects model decay but the engineering team insists the model is static and should not degrade. The data science lead suggests investigating further. They have access to production logs, A/B testing framework, and historical model versions. What is the BEST course of action to diagnose and address the issue?
⚠ Common exam trap
CompTIA often tests the principle that diagnosing model decay requires a controlled comparison (A/B test) rather than immediately retraining or adding features, and the trap here is assuming that a static model cannot degrade when the underlying data distribution changes.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
Set up an A/B test comparing the current model against the original baseline model using recent traffic.
Setting up an A/B test comparing the current model against the original baseline model using recent traffic directly isolates whether the model's predictive performance has degraded due to concept drift (changes in user behavior over time). Since the model is static but the data distribution has shifted, the A/B test provides empirical evidence of decay by measuring CTR differences under identical conditions, which is the standard diagnostic step before any retraining or feature engineering.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✗
Re-deploy the model with additional features such as time of day and user device.
Why it's wrong here
Adding time-of-day and device features addresses feature coverage, not the drift in user-item interaction patterns that degrades a collaborative filtering model over time. Feature enrichment is the right choice when the model lacks signal the business already holds, not when CTR decays without pipeline changes.
- ✗
Increase the frequency of batch inference from hourly to every 10 minutes to improve responsiveness.
Why it's wrong here
Shortening batch inference intervals changes scoring latency, not the learned user-item relationships that decay as tastes shift. Higher inference frequency suits freshness-sensitive serving where stale scores are the bottleneck, not a steady CTR decline caused by distribution drift in the underlying interaction data.
- ✓
Set up an A/B test comparing the current model against the original baseline model using recent traffic.
Why this is correct
Running an A/B test against the original baseline on recent traffic isolates whether the current model has decayed relative to its starting performance, separating genuine model drift from shifting user behaviour. This satisfies the stem's diagnostic need using the available framework and historical versions.
- ✗
Retrain the model on only the most recent 30 days of data and replace the current model.
Why it's wrong here
Retraining on only 30 days discards the six-month interaction history the neural collaborative filtering model relies on, and replaces the model before any diagnosis confirms drift. Short-window retraining suits rapidly shifting domains; here it would remove signal without first validating the decay hypothesis.
About these practice questions
This AI0-001 question is part of Courseiva's 962-question bank — original exam-style content with full explanations and wrong-answer analysis, never real exam questions or exam dumps. Learn why practice questions differ from exam dumps →
JA
Written by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
This AI0-001 practice question is part of Courseiva's free CompTIA certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the AI0-001 exam.