AI0-001 AI Implementation and Operations Practice Question
An AI operations team is monitoring a deployed image classification model. They notice a gradual increase in prediction confidence but a drop in accuracy. Which THREE actions should they take to diagnose the issue?
⚠ Common exam trap
CompTIA often tests the distinction between diagnostic actions and corrective actions—candidates mistakenly jump to retraining (Option E) or data collection (Option B) instead of first analyzing calibration and data distribution (Options A, C, D) to identify the specific type of drift or miscalibration.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
Analyze the model's calibration curve to see if confidence scores align with actual accuracy.
Option A is correct because a calibration curve (reliability diagram) directly compares predicted confidence against observed accuracy, revealing whether the model has become overconfident — exactly the symptom of rising confidence with falling accuracy. Option C is correct because comparing training versus production input feature distributions (e.g., via drift metrics like PSI or KL divergence) detects data drift or covariate shift that can degrade accuracy while inflating confidence. Option D is correct because evaluating on a held-out test set collected at deployment time establishes a stable baseline, distinguishing genuine model degradation from changes in the production data distribution. Option B is not appropriate because collecting unlabeled data does not diagnose the cause and cannot be used for supervised evaluation without labels. Option E is not appropriate because immediate retraining without root-cause analysis risks masking the problem and may reinforce the drift or labeling issues causing the confidence-accuracy mismatch.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✓
Analyze the model's calibration curve to see if confidence scores align with actual accuracy.
Why this is correct
Rising confidence alongside falling accuracy signals overconfident misclassification, so plotting the calibration curve quantifies the gap between predicted probability and observed correctness. This reveals whether the model's probability outputs have drifted from true likelihoods, guiding recalibration or retraining decisions.
- ✗
Increase the size of the training dataset by collecting more unlabeled data.
Why it's wrong here
Collecting more unlabeled data does not diagnose why confidence rises while accuracy falls; without labels it cannot even measure the drift. It is tempting because extra training data is the standard remedy for poor generalisation, but this scenario calls for investigation of data drift, label quality and calibration first.
- ✓
Compare the distribution of input features between training and recent production data.
Why this is correct
Accuracy loss with inflated confidence typically stems from distribution shift, so comparing training feature distributions against recent production inputs exposes covariate drift. Detecting shifted pixel statistics, lighting or class frequencies identifies the data change responsible for degraded generalisation.
- ✓
Evaluate model performance on a held-out test set collected at deployment time.
Why this is correct
A held-out test set captured at deployment time provides a stable, uncontaminated reference point. Scoring the current model against it separates genuine performance decay from monitoring artefacts, confirming whether accuracy has truly dropped since release.
- ✗
Retrain the model immediately with the most recent data.
Why it's wrong here
Retraining immediately treats the symptom without identifying the cause, and training on drifted or mislabelled recent data can entrench the fault. It is tempting because retraining is the usual response to degradation, but it is the correct action only once diagnosis has confirmed the underlying data or concept drift.
About these practice questions
One of 962 original AI0-001 practice questions on Courseiva, each with a full explanation and wrong-answer analysis — not exam dumps or protected exam content. Learn why practice questions differ from exam dumps →
JA
Written by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
This AI0-001 practice question is part of Courseiva's free CompTIA certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the AI0-001 exam.