Custom Vision Models: Training and Evaluation

Azure Custom Vision is a cloud-based service that allows you to build, train, and deploy custom image classifiers and object detectors without requiring machine learning expertise. It is part of Azure Cognitive Services, specifically under Computer Vision. The service is designed for scenarios where pre-built models (like those in Computer Vision) are insufficient because you need to recognize domain-specific objects or concepts—for example, identifying defects in manufactured parts, distinguishing between different species of plants, or detecting specific logos in images.

Custom Vision eliminates the need to write code for model architecture, training loops, or hyperparameter tuning. Instead, you provide labeled images, and the service automatically handles feature extraction, model selection, and optimization. The underlying architecture uses transfer learning from a pre-trained deep neural network (such as ResNet or MobileNet), which is fine-tuned on your custom dataset. This approach dramatically reduces the amount of data and time required compared to training from scratch.

When you upload images and assign tags (for classification) or draw bounding boxes (for object detection), Azure Custom Vision performs several steps:

- Project Types: - Image Classification: Assigns one or more tags to an entire image. Two subtypes: - Multiclass: Each image has exactly one tag (e.g., 'cat' or 'dog'). - Multilabel: Each image can have multiple tags (e.g., 'cat' and 'sleeping'). - Object Detection: Identifies objects within an image and draws bounding boxes around them. This requires you to draw bounding boxes for each object instance.

- Domains: Custom Vision offers specialized domains that optimize the model for specific scenarios: - General: Default, works for most everyday objects. - Food: Optimized for recognizing food items. - Landmarks: For recognizing natural and man-made landmarks. - Retail: For products on shelves. - Compact Domains: Lightweight models for edge devices (e.g., 'General (compact)', 'Landmarks (compact)'). These are smaller and faster but may have lower accuracy.

- Evaluation Metrics: - Precision: Percentage of correct positive predictions (True Positives / (True Positives + False Positives)). - Recall: Percentage of actual positives correctly identified (True Positives / (True Positives + False Negatives)). - Mean Average Precision (mAP): Standard metric for object detection. It averages the precision across different recall thresholds. Higher is better (range 0-100). - Accuracy: For classification, the percentage of correctly classified images (True Positives + True Negatives) / Total. This metric can be misleading if classes are imbalanced.

Custom Vision is primarily managed through the Azure portal or the Custom Vision API. There is no direct CLI for training, but you can use the REST API or SDKs (Python, C#, etc.).

Using the Python SDK to create a project and train:

from azure.cognitiveservices.vision.customvision.training import CustomVisionTrainingClient
from msrest.authentication import ApiKeyCredentials

# Replace with your endpoint and training key
endpoint = "https://your-resource.cognitiveservices.azure.com/"
training_key = "your-training-key"

credentials = ApiKeyCredentials(in_headers={"Training-key": training_key})
trainer = CustomVisionTrainingClient(endpoint, credentials)

# Create a new project
project = trainer.create_project("My Custom Vision Project", domain_id="domain-id")

# Add tags and upload images (code omitted for brevity)

# Train the model
iteration = trainer.train_project(project.id)
print("Training completed with iteration ID:", iteration.id)

Verifying training status:

iteration = trainer.get_iteration(project.id, iteration.id)
print("Status:", iteration.status)
print("Trained at:", iteration.trained_time)
print("Is default:", iteration.is_default)

Publishing the model for prediction:

trainer.publish_iteration(project.id, iteration.id, "myModel", prediction_resource_id)

Custom Vision is often used alongside other Azure services: - Azure Storage: Store your training images in Blob storage and import them into Custom Vision. - Azure Functions or Logic Apps: Automate retraining when new images are added. - Azure IoT Edge: Deploy compact models to edge devices for offline inference. - Azure Cognitive Services Container: Run Custom Vision containers on-premises for low-latency scenarios.

The service feeds into a larger AI pipeline: images are captured, preprocessed, sent to Custom Vision for inference, and results are stored or trigger actions.

A car parts manufacturer uses Custom Vision to detect scratches on painted surfaces. They train an object detection model with hundreds of images of scratched and flawless parts. The model is deployed on the factory floor using Azure IoT Edge on a local GPU server. The system processes images from cameras in real time, flagging defective parts. Key considerations: the model must achieve high recall (to catch almost all defects) even if precision is slightly lower (some false alarms are acceptable). The team uses a compact domain for low latency. They continuously add new images of rare defects to improve recall. Misconfiguration: if the training set contains only images under ideal lighting, the model fails under variable factory lighting. They solved this by augmenting the dataset with synthetic variations.

A retail chain uses Custom Vision to identify products on shelves. They train a multilabel classifier to recognize multiple brands and categories in a single shelf image. The model runs on cameras in stores, updating inventory in real time. The challenge is scale: hundreds of stores, each with thousands of products. They use the General domain and train on images from multiple stores to handle lighting differences. Performance metric: mAP is used because it balances precision and recall across many classes. Common issue: class imbalance—popular products dominate, while rare items have few examples. They mitigate by oversampling rare classes. The model is retrained weekly with new product images.

A healthcare startup uses Custom Vision to classify X-ray images for signs of pneumonia. They use a multiclass classifier with tags 'normal' and 'pneumonia'. Due to regulatory requirements, the model must have extremely high precision (to avoid false positives that could lead to unnecessary treatment). They set the probability threshold to 95% and only act on high-confidence predictions. The model is deployed in a container on-premises for data privacy. They continuously monitor performance using a separate test set. Failure mode: if the training set has more normal images than pneumonia, the model becomes biased toward 'normal'. They carefully balance the dataset and use weighted loss functions.