Computer vision for video analysis and custom vision. This chapter is about teaching computers to not just see, but to understand what they are looking at in video footage and to solve unique visual problems that off-the-shelf AI can't handle. For the AI-102 exam, you need to know how to use Azure AI services to extract insights from videos and how to train your own image classification or object detection model when the standard tools aren't enough.
Jump to a section
A simple way to picture Computer Vision: Video Analysis and Custom Vision
A supermarket quality inspector is trained to spot a very specific problem: a bruised apple. They walk the aisles, look at the live feed from the store cameras, and instantly flag any apple that doesn't meet the standard. This is pre-built video analysis – the inspector knows exactly what to look for.
Now, imagine a small bakery that makes artisan sourdough loaves. The inspector knows apples, not bread. To check their loaves, the bakery hires a different expert who spends a day learning what a perfect loaf looks like: the golden crust, the scoring pattern, the size. This expert is a custom vision model. You train it on your own photos – your own 'loaves of bread' – so it can spot what you care about, even if no one else in the world has ever looked for it.
The inspector (pre-built video analysis) uses off-the-shelf training. The loaf expert (Custom Vision) is a specialist you build yourself. For AI-102, you need to know when to use the supermarket inspector (Azure Video Indexer or pre-built models) and when to train your own loaf expert (Custom Vision) for a business's unique needs, like finding cracks in a specific car part or identifying a rare bird species.
This section covers two related but distinct Azure AI services: Azure Video Indexer and Custom Vision. Let's break them down from the ground up.
What is Computer Vision? At its simplest, computer vision is the field of AI that gives machines the ability to 'see'. By 'see', we mean they can take an image or a video frame, break it down into pixels, and then use sophisticated mathematical models to recognise patterns – like edges, shapes, colours, textures – and then label what they see. It's not magic; it's pattern recognition on a massive scale, trained on millions of examples.
What is Video Analysis with Azure Video Indexer? Think of Azure Video Indexer as a highly specialised media analyst. You give it a video file (like an MP4), and it returns a detailed JSON file containing:
A transcript of every spoken word (speech-to-text).
Identified speakers (who said what, if voices are known).
Keyframes (the most important single images from the video).
OCR (optical character recognition) to read text that appears in the video, like a street sign or a product label.
Recognised objects, people, and brands.
Sentiment analysis (was the mood of the clip positive, negative, or neutral?).
And even a summary of the main topics or concepts discussed.
It works by feeding the video through a pipeline of pre-trained AI models. These models are the same ones Microsoft has trained on huge datasets, so you don't need to show them what a 'dog' or a 'car' looks like – they already know.
What is Custom Vision? Custom Vision is the exact opposite of the 'already-trained' approach. It is a service that lets you train your own model on your own images. The primary use cases are:
- Image Classification: Answering the question, 'What is the main object in this picture?' For example, 'Is this X-ray showing a fracture or not a fracture?' - Object Detection: Answering the question, 'Where in the picture are all the objects of a particular type?' For example, 'Where are all the stop signs in this street scene?' It draws bounding boxes around them. Here is the workflow for Custom Vision: 1. Upload Images: You start by uploading a minimum of 15-30 images per category you want to recognise. These are your 'training data'. 2. Label Images: For classification, you tag each image with a class label ('happy face', 'sad face', 'angry face'). For detection, you draw rectangles (bounding boxes) around each object in the image and assign them a class tag. 3. Train the Model: You click the 'Train' button. The service uses a technique called 'transfer learning'. It starts with a model that already knows how to recognise basic shapes and edges, then fine-tunes it using your specific images. This is much faster than training from scratch. 4. Evaluate the Model: After training, the service shows you a performance report with metrics like precision and recall. You can test the model with new images it has never seen. 5. Publish as a Prediction Endpoint: Once satisfied, you publish the trained model as an API endpoint. Applications can then send new images to this endpoint and get predictions back. Why do these two services exist separately? Because they solve different business problems. Video Indexer is perfect for extracting broad, pre-defined insights from video content – like a media company analysing thousands of hours of news footage to find every mention of a specific politician. Custom Vision is perfect when you have a very narrow, specific visual task that the pre-built models simply don't cover – like a furniture factory needing to detect small, unique scratches on a specific wood finish.
What problem does this solve for IT professionals? Before these services existed, analysing video or creating a custom image classifier required a team of data scientists and software engineers. They would need to manually code image processing algorithms, manage training data, and deploy custom infrastructure. Azure Video Indexer and Custom Vision remove that barrier. They provide a 'no-code' or 'low-code' way to get these powerful capabilities running, which dramatically saves time and money. Key concepts to understand for the exam: - The difference between a pre-built model (Video Indexer) and a trained model (Custom Vision). - The two types of Custom Vision projects: Image Classification and Object Detection. - The concepts of 'Precision', 'Recall', and 'mAP' (mean Average Precision) as performance metrics for Custom Vision. - The requirement for a minimum number of images (roughly 30 per tag) for reasonable model performance. - How to export a Custom Vision model for offline use (e.g., on a Docker container or on a phone).
Identify the Visual Task
Determine if the problem is pre-built (Video Indexer for standard video analysis) or requires a unique model (Custom Vision for specific objects). This decision is vital for the exam and for real-world efficiency.
Prepare and Upload Training Data
Gather at least 30 high-quality images per category (or per object for detection). Ensure variety in angles, lighting, and backgrounds to make the model robust. Upload them to the Custom Vision portal.
Label the Images
For classification, assign a tag (e.g., 'defective part', 'good part') to each image. For object detection, draw bounding boxes around each object of interest and assign a tag. Accurate labelling directly determines model accuracy.
Train the Model
Click the 'Train' button. The service uses transfer learning to fine-tune a pre-existing model on your data. This creates an 'iteration'. The faster training uses the Quick Training option; the slower but more accurate option uses Advanced Training.
Evaluate and Iterate
Review the performance metrics (Precision, Recall, mAP). If the model is not accurate enough, add more images, correct mislabelled images, or adjust the training domain. Retrain to create a new iteration. Publish the best performing iteration as the default endpoint.
Publish and Integrate
Publish the model to a prediction endpoint URL. Write code in your application (e.g., a warehouse management system) that calls this endpoint with a new image. The response contains the prediction results (class labels and confidence scores).
Export for Offline Use (if needed)
If the model needs to run on an edge device, export it as a TensorFlow, ONNX, CoreML, or Docker container file. Deploy that file to the device. The model can then operate without internet connectivity.
Meet Sarah, a solutions architect at a mid-sized retail company. Her boss comes to her with a problem: they have 50 security cameras covering the shop floor, and they want to automatically count the number of people entering a specific 'end-of-aisle' display area to measure engagement with a new product.
Sarah immediately thinks about two approaches.
Step 1: Assessing the Need The hard part is not just counting people – any basic camera system can do that. The real challenge is identifying the specific area boundaries and only counting people who cross into that area. Sarah determines this is a 'spatial analysis' problem. She could use Azure Video Indexer to get a basic count of people in the frame, but it won't understand the custom 'zone' she needs.
Step 2: Choosing the Right Tool Sarah decides against Video Indexer for the core task because its pre-built models don't know about 'the end of aisle 5 near the cereal'. Instead, she chooses Azure Video Analyzer (which is part of the AI-102 scope) to create a custom pipeline. She sets up a real-time video analysis system that uses a Custom Vision model to detect a specific object – a laminated 'promo sign' she placed near the display. The system tracks when the sign is 'entered' by a person (bounding box overlapping).
Step 3: Building the Custom Vision Solution 1. Data Collection: Sarah's intern takes 200 photos of the display area from different angles, with and without people standing near the sign. 2. Labeling: They use the Custom Vision portal to draw bounding boxes around the promo sign in each image. They also add a class called 'person' and draw bounding boxes around people. 3. Training: Sarah uploads the data to Custom Vision and starts a training run. The first iteration has a precision of only 60% – it mistakes a shopping cart for a person. 4. Iteration: She adds 50 more photos of shopping carts and labels them as 'not a person'. After retraining, precision rises to 90%. 5. Deployment: She publishes the model and connects it to the video stream via a Docker container running on the edge. The system now sends an alert to the store manager whenever the 'promo sign' area has more than three people. Step 4: Scaling the Solution Sarah's boss is thrilled. They now want to roll this out to 100 stores. Sarah realises she cannot manually retrain for every store. She creates a master 'promo sign detection' model using images from five stores and then uses the Custom Vision portal's 'Export' feature to create a TensorFlow model that runs on the store's local server, reducing cloud costs. What an IT professional actually does: - Evaluates the business need and decides between pre-built analysis (Video Indexer) and custom training (Custom Vision). - Collects and curates high-quality, diverse training images. - Labels images with bounding boxes or classification tags – a tedious but critical step. - Trains, evaluates, and iterates on the model until performance meets the business requirement. - Writes code to integrate the model's predictions into a wider application, like a store dashboard or an alerting system. - Deploys the model to the cloud or to edge devices (cameras, IoT devices) using Docker containers.
The AI-102 exam focuses heavily on the practical differences between these two services, not on deep theory. Here is what they will test directly.
Question types that appear often: - Scenario-based questions: They describe a business problem (e.g., 'A manufacturing plant wants to detect defective parts on a conveyor belt') and ask you to choose between Azure Video Indexer, Custom Vision, or another service like Face API. The correct answer nearly always hinges on whether the objects to detect are common (pre-built) or unique (custom). Defective parts are unique, so Custom Vision is the right choice. - Process flow questions: They ask you to order the steps of a Custom Vision workflow. The correct order is: 1. Upload images, 2. Tag words/label bounding boxes, 3. Train model, 4. Evaluate, 5. Publish. They love to reverse steps or insert a 'deploy to Azure Kubernetes Service' step too early. - Definition questions: They ask you to define 'Object Detection' versus 'Image Classification'. Answer: Classification tells you *what* is in the picture (one label per image). Detection tells you *where* objects are (multiple bounding boxes per image). - Metric questions: They ask about 'Precision' and 'Recall'. Precision = 'Of all the things the model said were there, how many were actually there?' (avoid false positives). Recall = 'Of all the things that were actually there, how many did the model find?' (avoid false negatives). They often give you a scenario where one is more important than the other (e.g., medical diagnosis: high recall is critical, even if it means lower precision). Traps they set: - The 'Video Indexer can do anything' trap: They describe a highly specific task (like identifying a company-specific hand gesture) and hint that Video Indexer can do it because it can analyse video. The correct answer is Custom Vision, because pre-built models don't know about your unique gesture. - The 'Export model' trap: They ask you to deploy a Custom Vision model to a robot that has no internet connection. You must know you can export the model as a Docker container (via ONNX, TensorFlow, or CoreML) for offline use. - The 'Minimum images' trap: They claim you can train with 5 images. The official documentation recommends 30+ per tag. The exam will test this. - The 'Face API' confusion: They describe a scenario where you need to identify a specific person (e.g., 'Is this John Smith in the video?') but they try to make you use Custom Vision. The correct answer is Face API (for person identification) or Video Indexer (for face detection), not Custom Vision, which is for objects, not specific people. Key definitions to memorise for the exam: - mAP (mean Average Precision): The primary performance metric for Object Detection. Higher is better. - Iteration: Each time you train a Custom Vision model, you create a new 'iteration'. You can compare them and set one as 'default'. - Domain: The type of images you are training on (e.g., 'General', 'Food', 'Landmarks'). Choosing the correct domain improves model accuracy. - Compact Domain: A special type of domain that produces smaller, faster models suitable for export to mobile or edge devices. What you should NOT waste time on: - The underlying mathematics of convolutional neural networks (CNNs) – not tested. - The exact pricing tiers – just know it exists but don't memorise the prices. - How to use the Video Indexer API in Python or C# syntax – you are tested on the *concept*, not the exact code, but you should recognise the API endpoints (e.g., /insights, /faces).
Azure Video Indexer uses pre-built models to extract insights like transcripts, faces, and brands from video without any custom training.
Custom Vision requires you to upload and label at least 30 images per class to train a model that specialises in objects the pre-built models cannot recognise.
The two primary project types in Custom Vision are Image Classification (what is in the image) and Object Detection (where in the image the object is).
Precision measures false positives (what you thought was there but wasn't), while Recall measures false negatives (what you missed).
You can export a Custom Vision model to run offline on a Docker container or a mobile device, which is critical for edge scenarios with no internet.
For exam scenarios, if the problem involves identifying unique objects (defective parts, custom products), choose Custom Vision; if it involves standard visual elements (faces, text, common objects), choose Video Indexer or Computer Vision.
An 'iteration' in Custom Vision is a version of your trained model; you can have multiple iterations and compare their performance before publishing one as the default.
Choosing the correct 'Domain' (General, Food, Landmarks) in Custom Vision optimises the model's architecture for the type of images you are working with.
These come up on the exam all the time. Here's how to tell them apart.
Image Classification
Assigns a single label to the entire image (e.g., 'this is a cat').
Does not tell you where in the image the object is located.
Simpler to implement and requires less detailed labelling effort.
Object Detection
Draws bounding boxes around multiple objects in the image (e.g., 'there is a cat at x:10, y:20').
Provides spatial location for each detected object.
Requires more detailed labelling (drawing boxes) but gives richer data.
Azure Video Indexer
Uses pre-built, general-purpose AI models trained by Microsoft.
Extracts standard insights from video: transcripts, faces, brands.
No custom training data required; works out of the box.
Custom Vision
You train the model from scratch using your own labelled images.
Solves a unique, specific visual problem (e.g., detecting a rare car part).
Requires you to collect and label a minimum number of images (around 30 per tag).
Precision
Measures how many of the positive predictions were actually correct (avoiding false alarms).
Formula: True Positives / (True Positives + False Positives).
Crucial when false positives are very costly (e.g., firing your billing system for a non-existent customer).
Recall
Measures how many of the actual positive cases were correctly identified (avoiding misses).
Formula: True Positives / (True Positives + False Negatives).
Crucial when false negatives are very costly (e.g., missing a cancerous tumour in a scan).
General Domain
Optimised for high accuracy on general images with no special constraints.
Produces larger model files (hundreds of MBs).
Best for cloud-based predictions where latency and file size are not critical.
Compact Domain
Optimised for a smaller, faster model suitable for mobile or edge devices.
Produces much smaller model files (tens of MBs).
Slightly lower accuracy than General Domain but much faster inference speed.
Mistake
Custom Vision can be trained with just a few images (5-10) and will work perfectly.
Correct
Microsoft's official guidance recommends at least 30 images per tag, and often more for reliable real-world performance. Fewer images leads to severe overfitting (the model memorises the training images but fails on new ones).
Beginners see demos where a model works immediately and assume it is easy. They underestimate the variety needed to cover angles, lighting, and backgrounds.
Mistake
Azure Video Indexer can identify any specific person by name out of the box, like a celebrity database.
Correct
Video Indexer can detect faces and match them to known people you have enrolled (using Face API), but it does not have a built-in database of every person. You must provide photos and names beforehand.
Media often shows facial recognition 'finding a person' in a video, creating unrealistic expectations. The actual system requires setup.
Mistake
Once a Custom Vision model is trained, you never need to update it.
Correct
Real-world data changes (lighting, new product versions). Models degrade over time. You should retrain periodically with fresh images to maintain accuracy.
People think AI is 'fire and forget'. They don't understand machine learning requires ongoing maintenance.
Mistake
You can use Custom Vision to read text from a video, like a license plate.
Correct
Custom Vision is for classifying images or detecting objects by shape and texture. For reading text, you need Azure Computer Vision's OCR (Optical Character Recognition) capability. They are different services.
Beginners hear 'computer vision' and assume every visual task is covered. They don't distinguish between 'seeing an object' and 'reading characters'.
Mistake
Video Indexer and Custom Vision are completely separate and cannot work together.
Correct
They can be complementary. For example, you could use Video Indexer to extract keyframes from a video, then send those frames to a Custom Vision model to classify them for a specific business need.
People think of cloud services as silos. Architects often combine multiple services to solve a problem.
Reveal each answer, then mark whether you got it right. Score 60%+ to unlock the next chapter.
Video Indexer is a pre-built service for extracting insights (transcripts, faces) from a video file. Video Analyzer is a platform for building custom real-time video analytics pipelines, often used with edge devices and Custom Vision.
Microsoft recommends a minimum of 30 images per tag for classification and at least 15 per tag for object detection, but more is better, especially to cover varied backgrounds and lighting.
No. Custom Vision is for objects, not for identifying specific people. For person identification, use Azure Face API (with a PersonGroup). Custom Vision could detect 'a face' but not 'John's face'.
mAP stands for mean Average Precision. It is the standard metric for evaluating the accuracy of object detection models. A higher mAP (closer to 100%) means the model can correctly locate and identify objects more accurately.
For basic demos, yes. The Custom Vision portal allows you to upload, label, and train via a web GUI. However, real integration and the exam's coding-level questions require you to know the REST API calls or SDK usage.
A compact domain is a setting that trains a smaller, faster model that is optimised for export to mobile devices or edge devices with limited computing power. It sacrifices some accuracy for performance.
You've finished Computer Vision: Video Analysis and Custom Vision. Continue through the AI-102 study guide to build a complete picture of the exam.
Done with this chapter?