AI0-001 Implementing AI Solutions Practice Question
A hospital's AI triage assistant was validated on data from its own emergency department. Before rolling it out to three affiliated hospitals with different patient demographics, imaging equipment, and documentation habits, the governance committee requires evidence that the model will not silently underperform at the new sites. Which activity BEST provides that evidence?
⚠ Common exam trap
The trap here is treating a strong internal validation AUC as proof of generalization, when internal test data shares the exact site characteristics that differ at the new hospitals.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
Run an external validation using held-out data from each receiving hospital and compare subgroup performance against the original site.
Generalization to new sites can only be demonstrated with data the model has never seen from those sites. External validation on each hospital's held-out records, broken down by subgroup, reveals demographic, equipment, and workflow-related performance gaps before patients are exposed. Reusing the internal test set, tuning thresholds, or fine-tuning prematurely all fail to produce that evidence and would leave the committee without a defensible basis for approval.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✗
Fine-tune the model on a sample of records from the three new hospitals before any prospective evaluation.
Why it's wrong here
Fine-tuning before measuring baseline performance destroys the ability to distinguish intrinsic generalization failure from adaptation effects, and if the sample is unlabeled or small it risks overfitting to site quirks. Governance wants evidence of current behavior on new populations, which requires evaluation first; adaptation is a later remediation step informed by those results.
- ✓
Run an external validation using held-out data from each receiving hospital and compare subgroup performance against the original site.
Why this is correct
External validation on each target site's own held-out data directly measures whether performance generalizes across demographics, equipment, and documentation differences, and subgroup analysis exposes disparities that aggregate accuracy would hide. This is the accepted method for pre-deployment generalization evidence in clinical AI governance. The other approaches either do not test the new populations or cannot reveal site-specific failure.
- ✗
Increase the model's confidence threshold at the new sites until the override rate matches the original hospital's rate.
Why it's wrong here
Matching override or alert rates is an operational tuning exercise, not a performance measurement. A model can produce the same volume of alerts while being systematically wrong on the new population, and threshold changes can mask subgroup degradation. Without labeled outcomes from the receiving sites, this approach provides no evidence about accuracy or safety and could delay detection of harm.
- ✗
Re-run the original internal test set and confirm that the AUC is unchanged from the validation report.
Why it's wrong here
The original test set comes from the same emergency department and shares its demographics, imaging devices, and note-writing conventions, so it cannot detect site shift. Reproducing the same AUC only confirms the model and pipeline are deterministic; it says nothing about the three affiliated hospitals. Governance would rightly reject this as circular evidence of generalization.
About these practice questions
One of 962 original AI0-001 practice questions on Courseiva, each with a full explanation and wrong-answer analysis — not exam dumps or protected exam content. Learn why practice questions differ from exam dumps →
JA
Written and reviewed by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
Last reviewed September 2026 · checked against the official CompTIA exam blueprint
This AI0-001 practice question is part of Courseiva's free CompTIA certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the AI0-001 exam.