AI0-001 AI Implementation and Operations Practice Question
A startup has developed a natural language processing model for sentiment analysis. Their CI/CD pipeline includes a step that runs unit tests on the model's output format and a validation step that checks accuracy on a static test dataset. Recently, the pipeline often fails during the validation step, but the failures are inconsistent—sometimes the same model version passes, sometimes fails. The team suspects the test dataset is small and randomly sampled. They need a reliable validation process to deploy models with confidence. Which approach should the team implement?
⚠ Common exam trap
CompTIA often tests the misconception that increasing the accuracy threshold or using cross-validation alone can fix validation instability, when the real solution is to address the root cause of small, non-representative test data with statistical rigor.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
Fix the test dataset to be larger and more representative, and use a statistical test to compare against baseline
The core issue is that the static test dataset is too small and randomly sampled, leading to inconsistent validation results. By fixing the dataset to be larger and more representative, and using a statistical test (e.g., a paired t-test or McNemar's test) to compare the model's accuracy against a baseline, the team can reliably determine if performance changes are statistically significant, eliminating the randomness that causes pipeline failures to be inconsistent.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✗
Replace the static test set with k-fold cross-validation in each pipeline run
Why it's wrong here
Cross-validation is computationally heavy for CI/CD.
- ✗
Increase the accuracy threshold to 95% so only very good models pass
Why it's wrong here
This doesn't address the inconsistency.
- ✗
Remove the validation step and rely on unit tests only
Why it's wrong here
Removing validation increases risk of deploying poor models.
- ✓
Fix the test dataset to be larger and more representative, and use a statistical test to compare against baseline
Why this is correct
A fixed dataset and statistical test provide consistent and objective validation.
About these practice questions
Courseiva writes every AI0-001 question from scratch — 754 in total, each with an explanation and a wrong-answer breakdown. None are copied from real exams or dumps. Learn why practice questions differ from exam dumps →
JA
Written by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
This AI0-001 practice question is part of Courseiva's free CompTIA certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the AI0-001 exam.