Face Attributes and Emotion Detection

Face attributes are structured data points extracted from a detected face in an image. They go beyond simply locating a face (bounding box) and include characteristics like age, gender, emotion, facial hair, glasses, and more. Emotion detection is a subset of attribute extraction that classifies facial expressions into predefined emotional states. On the AI-900 exam, you are expected to know which attributes the Face API can return and how emotion confidence scores work.

The Face API uses deep neural networks trained on millions of labeled faces. The process has three steps:

The Face API can return the following attributes:

Emotion detection is based on the Facial Action Coding System (FACS), which maps facial muscle movements (Action Units) to emotions. The Azure model uses a convolutional neural network (CNN) trained on a large dataset of labeled expressions. The output is a probability distribution over eight emotion classes. Importantly, the scores are not mutually exclusive—a face could show both surprise and fear. However, the API returns confidence scores that sum to 1. The highest score indicates the most likely emotion.

Exam tip: The AI-900 exam will ask which emotions are supported. The list is exactly: anger, contempt, disgust, fear, happiness, neutral, sadness, surprise. Contempt is often the one candidates forget.

To retrieve attributes, you must specify them in the request. Here's a sample REST call using curl:

curl -v -X POST "https://<your-endpoint>/face/v1.0/detect?returnFaceId=true&returnFaceLandmarks=true&returnFaceAttributes=age,gender,emotion" \
-H "Content-Type: application/json" \
-H "Ocp-Apim-Subscription-Key: <your-key>" \
-d '{
  "url": "https://example.com/photo.jpg"
}'

The response will include a JSON array of face objects. Each face object contains:

{
  "faceId": "abc123",
  "faceRectangle": {
    "top": 100,
    "left": 200,
    "width": 150,
    "height": 150
  },
  "faceAttributes": {
    "age": 30.5,
    "gender": "female",
    "emotion": {
      "anger": 0.001,
      "contempt": 0.002,
      "disgust": 0.001,
      "fear": 0.003,
      "happiness": 0.95,
      "neutral": 0.03,
      "sadness": 0.01,
      "surprise": 0.002
    }
  }
}

Note: If you don't specify returnFaceAttributes, no attributes are returned. The exam often tests this configuration requirement.

Attributes are separate from face identification. Face identification requires a PersonGroup and compares a detected face against enrolled faces. Attributes can be extracted during detection without any enrollment. However, you can combine both—for example, detect a face, get its attributes, and then identify it. The exam will ask you to differentiate: detection finds faces and optionally returns attributes; identification matches faces to known persons.

The Face API is priced per transaction. Each API call counts as one transaction regardless of how many faces are in the image. However, there is a limit on the number of faces returned per image (default 100). For attribute extraction, the same transaction cost applies. The exam may test that attribute extraction does not incur additional cost beyond the detection call.

The Face API is available in many Azure regions. However, some features like emotion detection may not be available in all regions due to compliance. For the exam, know that the Face API is generally available in West US, East US, West Europe, Southeast Asia, and others.

Microsoft has retired the use of Face API for emotion detection in certain scenarios due to privacy and bias concerns. As of June 2023, new customers cannot use the Face API to infer emotional states; existing customers have limited access. The AI-900 exam reflects this: you should know that emotion detection is a sensitive capability and Microsoft restricts its use. However, the exam still tests the technical understanding of how it works.

A large retail chain wants to gauge customer reactions to new product displays. They install cameras at eye level near displays. The Face API is called on each frame to detect faces and extract emotion attributes. The system aggregates emotion data over time to measure happiness and surprise levels. This helps the marketing team decide which displays are most engaging. In production, the system must handle high throughput—up to 30 frames per second from multiple cameras. The solution uses Azure Functions to process images asynchronously and stores results in Cosmos DB. A common misconfiguration is not setting returnFaceAttributes to include 'emotion', resulting in no emotion data. Also, the system must comply with privacy regulations; faces are not stored, only aggregated emotion counts. When misconfigured, the system might return neutral for all faces if the image quality is poor (e.g., low light). The team learned to preprocess images to enhance brightness before calling the API.

A financial institution uses face detection to verify identity at ATMs. They use Face API to detect a face and extract attributes like glasses and facial hair to compare with a stored profile. However, they need to prevent spoofing with photos. They combine attribute extraction with liveness detection (not part of Face API) to ensure a real person. In this scenario, attribute extraction helps filter out faces that don't match the expected features (e.g., wrong gender). Performance considerations: the API must respond within 2 seconds to avoid user frustration. The team uses the S0 tier with a dedicated endpoint. A common mistake is to assume that the Face API includes liveness detection—it does not. The exam may test that liveness detection is a separate feature.

A social media platform uses Face API to automatically tag users in photos. They first detect faces and extract attributes like age and gender to suggest tags. Then they use face identification to match against friends. The platform processes millions of photos daily. They use Azure Blob Storage to store images and Azure Functions to orchestrate. They found that attribute extraction sometimes fails for faces with heavy makeup or unusual angles. They mitigated by requesting multiple attributes and using the ones with highest confidence. Cost is a major factor—each call costs money, so they cache results for identical images. The exam may ask about cost optimization: you can reduce costs by not requesting attributes you don't need.