How to Monitor and Retrain Custom Vision Models After Data Drift
A manufacturing company uses Azure Custom Vision to detect defects on an assembly line. The model is deployed to a container on a local edge server. Recently, the model's accuracy dropped. You suspect data drift. What should you do to monitor and retrain the model?
Quick Answer
The correct answer is to periodically collect new images with labels, retrain the model in Custom Vision, and redeploy the updated container. This is necessary because Azure Custom Vision lacks built-in drift detection or automatic retraining capabilities; when data drift occurs—such as changes in lighting or product appearance on an assembly line—the model’s accuracy degrades, and the only way to recover is by manually capturing representative new data, retraining the model in the cloud, and redeploying the updated container to the edge server. On the AI-102 exam, this question tests your understanding of Custom Vision’s limitations versus Azure Machine Learning’s capabilities, often appearing as a trap where candidates assume automated drift monitoring exists. A common memory tip is “Custom Vision is custom work—no auto-detect, no auto-retrain,” reminding you that you must implement your own monitoring loop.
⚠ Common exam trap
A common mix-up: candidates assume Azure Machine Learning's data drift monitoring works with any deployed model, but it specifically requires an Azure-hosted endpoint, not a local container, and Custom Vision lacks native drift detection or auto-retraining features.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
Periodically collect new images with labels, retrain the model in Custom Vision, and redeploy the updated container.
Custom Vision models deployed to containers on edge devices do not expose a REST endpoint that Azure Machine Learning's data drift monitoring can directly access. The only way to detect drift and retrain is to periodically collect new labeled images from the production line, retrain the model in Custom Vision, and redeploy the updated container to the edge server.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✗
Use Azure Machine Learning data drift monitoring on the Custom Vision endpoint.
Why it's wrong here
Azure ML drift monitoring is not natively integrated with Custom Vision.
- ✓
Periodically collect new images with labels, retrain the model in Custom Vision, and redeploy the updated container.
Why this is correct
Manual retraining is required to address drift.
- ✗
Configure Custom Vision to send alerts when drift is detected.
Why it's wrong here
Custom Vision does not have drift detection.
- ✗
Enable active learning in Custom Vision to automatically retrain the model.
Why it's wrong here
Active learning helps with labeling, not automatic retraining.
Go deeper
Related to this question
About these practice questions
One of 945 original AI-102 practice questions on Courseiva, each with a full explanation and wrong-answer analysis — not exam dumps or protected exam content. Learn why practice questions differ from exam dumps →
Same concept, more angles
1 more way this is tested on AI-102
These questions test the same concept from different angles. Work through them to make sure you can recognise it however the exam phrases it.
Variation 1. A manufacturing company uses Azure AI Custom Vision to detect defects on a production line. The model was trained with 500 images per class and achieves 95% accuracy. After deployment, the model's accuracy drops to 80% due to changes in lighting conditions. What is the most effective first step to improve the model's robustness?
hard- A.Reduce the probability threshold to increase recall.
- ✓ B.Capture additional images under the new lighting and retrain the model.
- C.Use Azure AutoML to automatically find the best algorithm.
- D.Add more images from the original lighting conditions to the training set.
Why B: The drop in accuracy is caused by a domain shift—specifically, new lighting conditions that were not represented in the original training set. The most effective first step is to capture additional images under the new lighting and retrain the model, as Custom Vision relies on diverse, representative training data to generalize to real-world variations. This directly addresses the root cause by expanding the training distribution to include the new lighting scenario, which is a fundamental principle of supervised learning in computer vision.
JA
Written by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
This AI-102 practice question is part of Courseiva's free Microsoft certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the AI-102 exam.