Application Insights Sampling and Cost Control

Application Insights collects telemetry data (requests, dependencies, exceptions, traces, etc.) from your application. In high-traffic scenarios, the volume of data can become enormous, leading to high costs and storage overhead. Sampling is a technique that reduces the amount of telemetry data sent to Application Insights while preserving the ability to detect statistically significant events and trends. The goal is to keep a representative subset of all telemetry, so that queries and metrics remain accurate.

Sampling operates by selecting a subset of telemetry items based on a sampling rate. For example, a sampling rate of 10% means that only 10 out of every 100 telemetry items are sent. The key challenge is to ensure that related telemetry items (e.g., a request and its associated dependencies and exceptions) are either all kept or all dropped. This is called 'correlated sampling.' Application Insights uses a hashing algorithm on the operation ID (or a combination of IDs) to deterministically decide whether to keep a telemetry item. This ensures that all items belonging to the same operation are sampled together.

There are three types of sampling in Application Insights:

#### Adaptive Sampling Defaults

For ASP.NET and ASP.NET Core, adaptive sampling is enabled by default. The default settings are:

You can modify these settings in the appsettings.json or via code.

#### Fixed-rate Sampling Configuration

In code, you can set fixed-rate sampling like this:

using Microsoft.ApplicationInsights.Extensibility;
using Microsoft.ApplicationInsights.WindowsServer.TelemetryChannel;

var config = TelemetryConfiguration.Active;
var samplingProcessor = new SamplingTelemetryProcessor(config);
samplingProcessor.SamplingPercentage = 10.0; // 10%
config.TelemetryProcessorChainBuilder.Use(samplingProcessor);
config.TelemetryProcessorChainBuilder.Build();

In appsettings.json:

{
  "ApplicationInsights": {
    "SamplingPercentage": 10.0
  }
}

#### Ingestion Sampling Configuration

In the Azure portal, navigate to your Application Insights resource, click on 'Usage and estimated costs', then 'Data sampling'. You can set a percentage from 0 to 100. Note that ingestion sampling is applied after data is sent, so it does not affect the amount of data transmitted over the network.

Application Insights has a daily cap feature that limits the amount of data ingested per day. This is essential for cost control. The daily cap is set in GB per day. When the cap is reached, data ingestion stops for the rest of the day. You can also set a 'notification threshold' at a percentage (e.g., 80%) that sends an alert when usage approaches the cap.

Key points about daily cap:

How to set daily cap:

In Azure portal, go to your Application Insights resource, click 'Usage and estimated costs', then 'Daily cap'. Enter the daily cap in GB. You can also set the notification threshold.

Sampling and daily caps work together. Sampling reduces the volume before it reaches the cap, so you are less likely to hit the cap. However, if sampling is too aggressive, you may lose important data. Conversely, if you set a low daily cap, you might hit it even with sampling, causing data loss. Best practice is to use adaptive sampling to keep volume within a target rate, and set a daily cap as a safety net to prevent cost overruns.

To check current sampling settings for adaptive sampling, you can use Application Insights Analytics queries:

// Check sampling rate over time
let start = ago(1h);
requests
| where timestamp > start
| summarize count() by bin(timestamp, 5m)
| extend sampledCount = count_ * (1 / samplingRate)

To see if daily cap has been reached, query:

// Check daily cap events
union traces, exceptions, requests
| where timestamp > ago(1d)
| where message contains "Daily cap"

Understanding these mechanisms is crucial for the AZ-204 exam, where you may be asked to recommend a sampling strategy based on requirements like cost, accuracy, and complexity.

A large e-commerce platform experiences massive traffic spikes during Black Friday sales. Without sampling, telemetry volume could increase 100x, leading to exorbitant costs. The engineering team implements adaptive sampling with a target rate of 5 events/sec/host. During normal traffic, sampling percentage is near 100%. During spikes, it drops to 1-2%, keeping costs predictable. However, they also set a daily cap of 50 GB as a safety net. During the spike, the cap is reached after 6 hours, causing data loss for the remainder of the day. They learn to increase the cap temporarily during known events. They also set a notification threshold at 80% to get early warnings.

A financial services application must retain 100% of audit-related telemetry (e.g., user authentication events). They use fixed-rate sampling of 25% for all telemetry but exclude 'Event' telemetry type from sampling. They configure the SDK to exclude 'Event' telemetry: excludedTypes: "Event". This ensures that all custom events (which include audit logs) are sent. They also set a daily cap of 10 GB. Because the audit events are low volume, they rarely hit the cap. However, they monitor the cap usage daily and have an alert at 80%. This approach balances cost and compliance.

An IoT backend processes telemetry from millions of devices. Each device sends a heartbeat every minute. The total volume is enormous. They use ingestion sampling set to 10% at the Application Insights resource level. This is simpler than modifying each device's SDK. The downside is that network bandwidth is still consumed (since ingestion sampling is server-side). To reduce bandwidth, they also implement client-side fixed-rate sampling at 50% on the devices. This reduces network traffic by half. The combined effect: only 5% of original telemetry reaches the storage. They also set a daily cap of 100 GB. This multi-layer approach is common in high-volume scenarios.