Anomaly Detection

Anomaly detection is the process of identifying data points, events, or observations that deviate significantly from the majority of the data. In the context of Azure AI, the Anomaly Detector service is a pre-built AI model that automatically detects anomalies in time-series data without requiring you to train a custom model. It is part of Azure Cognitive Services and can be accessed via a REST API or client SDK.

While you could build a custom anomaly detection model using Azure Machine Learning, the Anomaly Detector service is designed for rapid integration and minimal setup. It is optimized for time-series data—data points indexed in time order, such as server CPU usage over hours, daily sales figures, or IoT sensor readings. The service handles seasonality, trends, and noise, which are common challenges in real-world data.

The Anomaly Detector uses a combination of statistical and machine learning models. The core algorithm is based on SR-CNN (Spectral Residual and Convolutional Neural Network) for batch detection and a sliding window approach for real-time detection. Here’s a step-by-step mechanism:

The Anomaly Detector service offers two main detection modes:

To use the service, you need an Azure Cognitive Services resource of type 'Anomaly Detector'. You can create one via the Azure portal or CLI:

az cognitiveservices account create \
    --name myAnomalyDetector \
    --resource-group myResourceGroup \
    --kind AnomalyDetector \
    --sku F0 \
    --location eastus

After creation, obtain the endpoint and key:

az cognitiveservices account show --name myAnomalyDetector --resource-group myResourceGroup --query "properties.endpoint"
az cognitiveservices account keys list --name myAnomalyDetector --resource-group myResourceGroup --query "key1"

Then, you can call the API using curl:

curl -X POST "https://<your-endpoint>/anomalydetector/v1.0/timeseries/entire/detect" \
-H "Content-Type: application/json" \
-H "Ocp-Apim-Subscription-Key: <your-key>" \
-d '{
  "series": [
    {"timestamp": "2023-01-01T00:00:00Z", "value": 100},
    {"timestamp": "2023-01-01T01:00:00Z", "value": 200},
    {"timestamp": "2023-01-01T02:00:00Z", "value": 150}
  ],
  "granularity": "hourly",
  "sensitivity": 95
}'

The response includes isAnomaly boolean for each point, expectedValue, upperMargin, and lowerMargin.

Anomaly Detector can be integrated with: - Azure Stream Analytics: For real-time anomaly detection on streaming data. - Azure Data Explorer: For historical analysis of large time-series datasets. - Power BI: To visualize anomalies in dashboards. - Azure Logic Apps: To trigger alerts or workflows when anomalies are detected.

A large e-commerce company uses Azure Anomaly Detector to monitor server CPU utilization across thousands of VMs. They collect metrics every 5 minutes into Azure Monitor, then stream the data via Azure Stream Analytics to the Anomaly Detector API. The goal is to detect unusual spikes that may indicate a DDoS attack or a runaway process. They use the 'last/detect' endpoint for real-time detection. The sensitivity is set to 90 to catch moderate anomalies while tolerating normal load spikes during flash sales. In production, they discovered that the default sensitivity of 95 caused too many false positives during peak hours, so they tuned it down. They also set up Azure Logic Apps to automatically scale out VMs when an anomaly is detected. A common misconfiguration is not providing the correct granularity—if timestamps are 5 minutes apart but granularity is set to 'hourly', the model misinterprets the seasonality, leading to poor detection.

A factory uses IoT sensors to record vibration levels of machinery every second. They want to detect early signs of bearing failure. They use the 'entire/detect' endpoint on historical data to identify patterns that precede failures. The data has strong daily seasonality (machines run at different speeds during shifts). They set the 'period' to 86400 seconds (24 hours) to help the model. The sensitivity is set to 80 because the cost of missing a real anomaly (catastrophic failure) is much higher than false positives (unnecessary maintenance checks). They also use the change point detection endpoint to identify when the machine's baseline vibration shifts permanently. A challenge is that the data is noisy, so they preprocess it with a moving average filter before sending it to the API. They also had to adjust the 'maxAnomalyRatio' to 0.1 because only about 10% of data points correspond to pre-failure conditions.

A fintech startup uses Anomaly Detector to monitor daily transaction volumes. They look for sudden drops or spikes that might indicate fraud or system issues. They use the 'changepoint/detect' endpoint to detect shifts in the trend. For example, if the average daily transactions suddenly drop from 10,000 to 5,000, that's a change point. They set the sensitivity to 95 to catch even small shifts. They also use the 'last/detect' endpoint to flag individual days that are unusually high or low compared to the expected seasonal pattern. A key consideration is that financial data often has weekly seasonality (weekdays vs weekends), so they set 'granularity' to 'daily' and 'period' to 7. They found that the service works well out-of-the-box, but they had to exclude holidays manually because the model cannot predict irregular events.