AI0-001 AI Implementation and Operations Practice Question
A company wants to roll out a new recommendation model to production. They decide to run an A/B test where 10% of users see the new model and 90% see the old model. After one week, the new model shows a 5% improvement in click-through rate. What is the next best action?
⚠ Common exam trap
CompTIA often tests the misconception that a short-term observed improvement is automatically statistically significant, tempting candidates to choose immediate full rollout (Option A) or premature reversion (Option B), when the correct answer emphasizes incremental deployment with continuous monitoring.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
Increase the testing percentage gradually while monitoring performance metrics and guardrails
A 5% improvement observed over only one week with a 10% traffic split is insufficient to confirm statistical significance or rule out novelty effects, data drift, or seasonal bias. The recommended best practice in AI deployment is to gradually increase the testing percentage (e.g., 10% → 25% → 50% → 100%) while continuously monitoring performance metrics and guardrails (e.g., click-through rate, conversion rate, latency, and error rates) to ensure the new model generalizes safely across the full user population.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✗
Immediately roll out the new model to 100% of users
Why it's wrong here
A one-week, 10% test showing 5% lift does not establish significance or guard against novelty effects, so full rollout risks unvalidated regression. It is tempting because a positive result invites immediate scaling, and would be correct after significance and guardrail metrics are confirmed on a broader population.
- ✗
Revert to the old model because the improvement is minimal
Why it's wrong here
A 5% click-through gain is a positive signal, not evidence of harm, so reverting discards a potentially valuable model without cause. It is tempting because small lifts can look negligible, and reverting would be right if the test showed degradation or breached a guardrail metric, which it did not.
- ✗
Run the test for another month to ensure statistical significance
Why it's wrong here
Extending to a month delays value and exposes users to an unvalidated model while significance could be reached sooner with a larger sample or sequential testing. It is tempting because longer tests reduce variance, and would be correct if the current sample were genuinely underpowered, which a 10% split over a week may already resolve.
- ✓
Increase the testing percentage gradually while monitoring performance metrics and guardrails
Why this is correct
A one-week 5% lift on 10% traffic is promising but not conclusive, so gradually raising exposure while watching guardrail metrics limits blast radius if the model degrades. Immediate full rollout risks undetected regressions; stopping discards a positive signal.
About these practice questions
One of 962 original AI0-001 practice questions on Courseiva, each with a full explanation and wrong-answer analysis — not exam dumps or protected exam content. Learn why practice questions differ from exam dumps →
JA
Written by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
This AI0-001 practice question is part of Courseiva's free CompTIA certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the AI0-001 exam.