Image Analysis: Tags, Captions, Objects

Azure Computer Vision provides three distinct but related services for analyzing images:

All three features use deep convolutional neural networks (CNNs) trained on the Microsoft Common Objects in Context (COCO) dataset and other large image corpora. The pipeline is:

3. Task-Specific Heads: - Tagging: The feature maps are fed into a multi-label classifier that outputs probabilities for each of the thousands of predefined tags. A threshold (default 0.5) filters out low-confidence tags. - Captioning: The feature maps are passed to a recurrent neural network (RNN) or transformer-based language model that generates a sequence of words. The model uses attention mechanisms to focus on relevant image regions while generating each word. - Object Detection: The feature maps are processed by a region proposal network (RPN) that generates candidate bounding boxes. Then a classifier assigns labels to each box, and a regression head refines box coordinates. Non-maximum suppression removes duplicate detections.

To use these features, you need an Azure AI Vision resource (formerly Computer Vision) and its endpoint and key. The REST API endpoint is:

https://<endpoint>/computervision/imageanalysis:analyze?api-version=2023-10-01

Example request body (JSON):

{
  "url": "https://example.com/image.jpg"
}

Or send binary image data directly.

Query parameters to specify features:

Example cURL command:

curl -X POST "https://<endpoint>/computervision/imageanalysis:analyze?api-version=2023-10-01&features=tags,caption,objects" \
-H "Ocp-Apim-Subscription-Key: <key>" \
-H "Content-Type: application/json" \
-d '{"url":"https://example.com/image.jpg"}'

Sample response (truncated):

{
  "captionResult": {
    "text": "A brown dog running on a sandy beach",
    "confidence": 0.95
  },
  "tagsResult": {
    "tags": [
      {"name": "dog", "confidence": 0.99},
      {"name": "beach", "confidence": 0.98},
      {"name": "sand", "confidence": 0.97},
      {"name": "animal", "confidence": 0.96},
      {"name": "outdoor", "confidence": 0.95}
    ]
  },
  "objectsResult": {
    "objects": [
      {
        "name": "dog",
        "confidence": 0.98,
        "boundingBox": {"x": 100, "y": 200, "w": 150, "h": 120}
      },
      {
        "name": "frisbee",
        "confidence": 0.85,
        "boundingBox": {"x": 300, "y": 250, "w": 50, "h": 50}
      }
    ]
  }
}

Tags and captions both describe content but at different granularities. Tags are atomic keywords; captions are sentences. Object detection adds spatial awareness. The exam often asks which service to use for a given scenario:

All three can be called in a single API request by specifying multiple features. This is efficient and commonly done in production.

A large online retailer needs to automatically tag millions of product images with keywords for search engine optimization and internal search. They use Azure Computer Vision's image tagging feature. They send each product image with features=tags and maxCandidates=20. The tags (e.g., 'shoes', 'red', 'leather', 'sneakers') are stored in a database and indexed by Azure Cognitive Search. In production, they process images in batches using Azure Functions triggered by blob storage uploads. They set up a queue to handle throttling (20 TPS per resource). They learned to avoid sending very similar images (e.g., same product from different angles) because tags are consistent, but they still pay per transaction. To reduce costs, they cache results for identical images using a hash of the image bytes. A common misconfiguration is not setting maxCandidates high enough, resulting in missing relevant tags like 'running' for athletic shoes. They also discovered that the service sometimes tags 'person' even if only a hand is visible, so they filter out tags with confidence below 0.7 for search indexing.

A news media company wants to automatically generate alt text for images to comply with WCAG accessibility guidelines. They use the caption feature. Each news article image is sent with features=caption. The returned caption becomes the alt attribute. They use the Standard tier and process about 50,000 images per day. They found that captions are generally accurate for common scenes but struggle with abstract or artistic images. For those, they fall back to manual entry. They also noticed that the caption model sometimes describes irrelevant background details, so they use the confidence score to decide whether to use the caption (only if confidence > 0.8). The exam may ask: 'Which feature should you use to generate a sentence describing an image?' The answer is caption, not tags.

A logistics company uses object detection to count items on shelves from camera feeds. They deploy Azure Computer Vision object detection on edge devices using Azure IoT Edge. Each frame is analyzed with features=objects. They detect 'box', 'pallet', 'forklift', and 'person'. Bounding boxes allow them to estimate occupancy. They set the confidence threshold to 0.6 to reduce false positives. A common problem is overlapping boxes (e.g., a person standing in front of a box) – non-maximum suppression handles this internally, but occasionally two objects merge into one box. They also use the 'parent' tag (e.g., 'furniture' for 'chair') to group objects. Performance is critical; they use GPU-accelerated VMs for lower latency. The exam may test that object detection returns bounding box coordinates (x, y, width, height) and that you can count objects programmatically.