AI0-001 AI Implementation and Operations Practice Question
A company wants to roll out a new recommendation model to production. They decide to run an A/B test where 10% of users see the new model and 90% see the old model. After one week, the new model shows a 5% improvement in click-through rate. What is the next best action?
⚠ Common exam trap
CompTIA often tests the misconception that a short-term observed improvement is automatically statistically significant, tempting candidates to choose immediate full rollout (Option A) or premature reversion (Option B), when the correct answer emphasizes incremental deployment with continuous monitoring.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
Increase the testing percentage gradually while monitoring performance metrics and guardrails
A 5% improvement observed over only one week with a 10% traffic split is insufficient to confirm statistical significance or rule out novelty effects, data drift, or seasonal bias. The recommended best practice in AI deployment is to gradually increase the testing percentage (e.g., 10% → 25% → 50% → 100%) while continuously monitoring performance metrics and guardrails (e.g., click-through rate, conversion rate, latency, and error rates) to ensure the new model generalizes safely across the full user population.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✗
Immediately roll out the new model to 100% of users
Why it's wrong here
Immediate full rollout without monitoring could cause widespread issues.
- ✗
Revert to the old model because the improvement is minimal
Why it's wrong here
A 5% improvement is significant; reverting would lose gains.
- ✗
Run the test for another month to ensure statistical significance
Why it's wrong here
While further testing may confirm significance, it delays deployment; gradual rollout is more balanced.
- ✓
Increase the testing percentage gradually while monitoring performance metrics and guardrails
Why this is correct
This approach captures benefits while controlling risk.
About these practice questions
One of 754 original AI0-001 practice questions on Courseiva, each with a full explanation and wrong-answer analysis — not exam dumps or protected exam content. Learn why practice questions differ from exam dumps →
JA
Written by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
This AI0-001 practice question is part of Courseiva's free CompTIA certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the AI0-001 exam.