# Custom Telemetry with Application Insights SDK

> Chapter 60 of the Courseiva AZ-204 curriculum — https://courseiva.com/learn/az-204/app-insights-telemetry

**Official objective:** 4.1 — Monitor

## Introduction

Using the Application Insights SDK for custom telemetry is a critical skill for the AZ-204 exam. You'll learn how to instrument applications to send custom events, metrics, and traces beyond what auto-collection provides. Approximately 10-15% of exam questions touch on telemetry customization, sampling, and SDK configuration, making this a high-yield topic. Mastery of the TelemetryClient, telemetry initializers, processors, and channel configuration is essential for passing the 'Monitor' objective.

## Custom Telemetry as a Black Box Recorder

With capacity for hundreds of parameters, an airplane's flight data recorder (black box) logs standard metrics like altitude, speed, and heading, as well as custom events such as engine vibration spikes, pilot control inputs, and cabin pressure fluctuations. The recorder is installed by an SDK (the data collection module) that hooks into the aircraft's systems. When a custom event occurs—say, a sudden drop in oil pressure—the SDK immediately writes a timestamped entry with the event name and any additional measurements (e.g., pressure value, temperature) into a local buffer. The buffer periodically flushes its contents to a central ground station (Application Insights) via a secure radio link, using a compression algorithm to minimize bandwidth. On the ground, analysts can query these custom events to correlate failures with specific conditions, set alerts for abnormal patterns, and create dashboards for real-time monitoring. Without the custom telemetry, they would only see generic flight data and miss the root cause of the problem. Similarly, Application Insights SDK allows developers to instrument their code with custom events and metrics, capturing business-specific data that standard auto-collected telemetry misses. The SDK handles buffering, retry logic, and sampling to ensure reliable delivery without overwhelming the application.

## Core explanation

### What is Custom Telemetry and Why Does It Exist?

Application Insights automatically collects a wealth of telemetry: requests, exceptions, dependencies, page views, and performance counters. However, auto-collection cannot capture business-specific events or custom metrics that are unique to your application. For example, an e-commerce app might need to track 'CheckoutCompleted' or 'ItemAddedToCart' events with custom properties like product ID and price. This is where custom telemetry comes in. By using the Application Insights SDK, you can programmatically send:
- **Custom Events**: Named occurrences (e.g., 'UserLoggedIn', 'PaymentFailed') with optional metrics and properties.
- **Custom Metrics**: Numeric values that can be aggregated (e.g., queue length, processing time).
- **Traces**: Diagnostic log messages with severity levels.
- **Page Views**: Custom page view tracking for non-web contexts.

### How It Works Internally

The Application Insights SDK provides a `TelemetryClient` object that acts as the primary interface for sending custom telemetry. When you call a method like `TrackEvent()`, the SDK creates a telemetry item (e.g., `EventTelemetry`) that includes:
- **Timestamp**: Automatically set to UTC now.
- **Sequence number**: For ordering.
- **Custom properties**: A dictionary of string key-value pairs.
- **Custom measurements**: A dictionary of string-double pairs for numeric data.
- **IKey**: The instrumentation key (or connection string) identifying the Application Insights resource.

The telemetry item is then passed through a **telemetry processor chain** (if configured) where you can modify or filter items. After processing, it enters the **telemetry channel**, which buffers items and sends them to the ingestion endpoint. The channel uses a default buffer size of 500 items and a flush interval of 30 seconds (configurable). If the buffer overflows, items are dropped (by default) or you can configure a custom overflow handler.

### Key Components, Values, Defaults, and Timers

- **TelemetryClient**: The main object. Must be created once per application (singleton) to avoid resource issues. Example:
```csharp
var telemetryClient = new TelemetryClient(new TelemetryConfiguration(connectionString));
```
- **Connection String**: Preferred over instrumentation key. Format: `InstrumentationKey=...;IngestionEndpoint=...`. Default ingestion endpoint is `https://dc.services.visualstudio.com`.
- **TrackEvent**: Sends a custom event. Example:
```csharp
telemetryClient.TrackEvent("OrderPlaced", new Dictionary<string, string> { {"Currency", "USD"} }, new Dictionary<string, double> { {"Revenue", 99.95} });
```
- **TrackMetric**: Sends a custom metric. Note: Pre-aggregate metrics to avoid sending too many data points. Use `GetMetric` for pre-aggregation.
- **TrackTrace**: Sends log traces. Example:
```csharp
telemetryClient.TrackTrace("Processing started", SeverityLevel.Information);
```
- **TelemetryProcessor**: Implements `ITelemetryProcessor`. Use to filter, modify, or enrich telemetry. Example:
```csharp
public class MyTelemetryProcessor : ITelemetryProcessor
{
    private ITelemetryProcessor Next { get; set; }
    public MyTelemetryProcessor(ITelemetryProcessor next) => Next = next;
    public void Process(ITelemetry item)
    {
        if (item is EventTelemetry eventTelemetry && eventTelemetry.Name == "SensitiveEvent")
            return; // Drop the event
        Next.Process(item);
    }
}
```
- **TelemetryInitializer**: Adds properties to all telemetry. Implements `ITelemetryInitializer`. Example:
```csharp
public class MyTelemetryInitializer : ITelemetryInitializer
{
    public void Initialize(ITelemetry telemetry)
    {
        telemetry.Context.Cloud.RoleName = "MyRole";
        telemetry.Context.GlobalProperties["Environment"] = "Production";
    }
}
```
- **Sampling**: Adaptive or fixed-rate sampling can reduce volume. Default adaptive sampling target is 5 items per second. Fixed-rate sampling uses a percentage (e.g., 10%).
- **Channel**: `InMemoryChannel` is default for .NET. It has a buffer size (default 500) and send interval (default 30s). You can configure:
```csharp
var channel = new InMemoryChannel { MaxTelemetryBufferCapacity = 1000, SendingInterval = TimeSpan.FromSeconds(10) };
```

### Configuration and Verification Commands

To configure the SDK in an ASP.NET Core app, add the following to `ConfigureServices`:
```csharp
services.AddApplicationInsightsTelemetry(connectionString);
```
For custom telemetry, inject `TelemetryClient`:
```csharp
public class MyService
{
    private readonly TelemetryClient _telemetryClient;
    public MyService(TelemetryClient telemetryClient) => _telemetryClient = telemetryClient;
}
```
To verify telemetry is flowing, use **Application Insights Search** in the Azure portal or the **Live Metrics** blade for real-time validation. You can also use **Diagnostic Search** to query for custom events.

### Interaction with Related Technologies

- **Azure Monitor Metrics**: Custom metrics sent via `TrackMetric` appear in Azure Monitor Metrics, but with limitations (e.g., 10 custom metrics per resource). Use pre-aggregation for better performance.
- **Log Analytics**: Custom events and traces are stored in `customEvents` and `traces` tables, queryable via Kusto.
- **Azure Functions**: App Insights integration is automatic; use `TelemetryClient` in function code.
- **OpenTelemetry**: Application Insights can receive telemetry via OpenTelemetry exporter. Custom events can be sent using OpenTelemetry SDK.

### Advanced: Telemetry Channel and Reliability

The channel handles retries with exponential backoff (up to 10 retries). If the endpoint is unreachable, items are stored in a local buffer (default 1 MB) and sent when connectivity resumes. You can implement a custom `ITelemetryChannel` for persistent storage (e.g., to disk) to prevent data loss during extended outages.

### Code Example: Full Custom Telemetry Flow

```csharp
using Microsoft.ApplicationInsights;
using Microsoft.ApplicationInsights.Extensibility;
using Microsoft.ApplicationInsights.DataContracts;

var config = new TelemetryConfiguration("InstrumentationKey=...;IngestionEndpoint=https://...");
var client = new TelemetryClient(config);

// Add a telemetry initializer
config.TelemetryInitializers.Add(new MyTelemetryInitializer());

// Add a telemetry processor
var builder = config.TelemetryProcessorChainBuilder;
builder.Use((next) => new MyTelemetryProcessor(next));
builder.Build();

// Track a custom event
client.TrackEvent("VideoStarted", new Dictionary<string, string> { {"VideoId", "123"} });

// Track a custom metric
client.GetMetric("VideoDuration").TrackValue(120.5);

// Flush before shutdown
client.Flush();
Task.Delay(5000).Wait(); // Allow time for flush
```

## Real-world context

### Scenario 1: E-commerce Checkout Tracking

A large online retailer needs to track every step of the checkout process to identify bottlenecks. They use `TrackEvent` with properties like `StepName`, `CartValue`, and `PaymentMethod`. They also track custom metrics like `TimeSpentOnStep` using `GetMetric` for pre-aggregation. The SDK is configured with a custom telemetry initializer that adds `SessionId` and `UserId` to all events. In production, the application handles thousands of checkouts per minute. To avoid overwhelming the ingestion endpoint, they enable adaptive sampling (default target 5 events/sec). However, they mark critical events (e.g., 'PaymentFailed') with `SamplingPercentage = 100` by using a telemetry processor that sets `item.SamplingPercentage = 100` for those events, ensuring no sampling. They also set up alerts on custom metrics to trigger when the checkout abandonment rate exceeds 20%.

### Scenario 2: IoT Device Telemetry

A manufacturing company collects sensor data from thousands of IoT devices. Each device runs a .NET Core application that sends custom metrics (temperature, pressure, vibration) using `TrackMetric`. To reduce cost and storage, they configure fixed-rate sampling at 10% in the SDK. They also use a telemetry processor to drop metrics that are within normal range (e.g., temperature between 20-30°C) and only send anomalies. The telemetry channel is configured with a larger buffer (MaxTelemetryBufferCapacity=2000) to handle bursts. They also implement a custom channel that writes to a local file if the network is down, then replays when connectivity returns. This ensures no data loss during network outages.

### Scenario 3: Multi-tier Microservices

A SaaS provider has a microservices architecture with 20 services. They use Application Insights for distributed tracing. Each service sends custom events for business operations (e.g., 'UserRegistered', 'SubscriptionUpgraded'). They use a telemetry initializer to set `Cloud.RoleName` to the service name, so they can filter telemetry by service. They also use a telemetry processor to add `CorrelationId` from the request headers to all custom events, enabling end-to-end tracking. The ingestion endpoint is configured with a custom endpoint for compliance reasons (e.g., `https://dc.applicationinsights.azure.com`). They monitor the live metrics to ensure no data loss and set up alerts on the rate of custom events per second to detect anomalies.

### Common Pitfalls

- Forgetting to flush before shutdown leads to data loss.
- Creating multiple `TelemetryClient` instances causes excessive memory use and potential duplicate telemetry.
- Not pre-aggregating custom metrics results in high volume and cost. Always use `GetMetric`.
- Overusing custom properties (max 512 characters per property, 100 properties per event) can cause truncation or rejection.
- Misconfiguring sampling can cause critical events to be dropped. Use telemetry processors to exempt important events.

## Exam focus

### What AZ-204 Tests on Custom Telemetry

The AZ-204 exam under objective 'Monitor' (4.1) tests your ability to instrument applications with custom telemetry. You must know:
- How to create and configure a `TelemetryClient` with connection string.
- The difference between `TrackEvent`, `TrackMetric`, `TrackTrace`, and `TrackException`.
- How to add telemetry initializers and processors.
- How to configure sampling (adaptive vs. fixed-rate) and its impact.
- How to use `GetMetric` for pre-aggregation.
- How to flush telemetry before shutdown.

### Common Wrong Answers and Why Candidates Choose Them

1. **Using instrumentation key instead of connection string**: Many candidates still use the deprecated instrumentation key. The exam emphasizes connection strings because they are more secure and support sovereign clouds. Wrong: 'InstrumentationKey=...'. Correct: 'InstrumentationKey=...;IngestionEndpoint=...'.
2. **Creating a new TelemetryClient for each request**: Candidates think it's stateless, but it causes resource leaks. The SDK is designed to be a singleton. Wrong: Creating multiple instances. Correct: Use dependency injection as a singleton.
3. **Not pre-aggregating metrics**: Candidates send raw metrics individually, causing high volume. Wrong: `TrackMetric` for every data point. Correct: Use `GetMetric` to pre-aggregate.
4. **Setting sampling percentage in code incorrectly**: Candidates assume sampling is disabled by default. Wrong: Not configuring sampling leads to high data volume. Correct: Understand adaptive sampling is enabled by default (target 5 items/sec).

### Specific Numbers and Terms That Appear on the Exam

- Default buffer size: 500 items.
- Default send interval: 30 seconds.
- Adaptive sampling target: 5 items per second.
- Maximum custom properties per event: 100.
- Maximum property key length: 150 characters.
- Maximum property value length: 8192 characters.
- Connection string format: `InstrumentationKey=...;IngestionEndpoint=...`.
- `GetMetric` returns a `Metric` object that pre-aggregates before sending.

### Edge Cases and Exceptions

- **TelemetryProcessor vs. TelemetryInitializer**: Initializers always run and modify items; processors can drop items. The exam may ask which to use for filtering.
- **Sampling exemption**: Use `item.SamplingPercentage = 100` in a processor to prevent sampling.
- **Channel configuration**: Custom channel for persistent storage is advanced but may appear.
- **Flush behavior**: `Flush()` is synchronous but the actual send is async; need to delay before exit.
- **Multiple telemetry clients**: If using multiple `TelemetryClient` with different configurations, they may share the same channel, leading to unexpected behavior.

### How to Eliminate Wrong Answers

- If the question mentions 'filtering telemetry based on content', the answer is a **telemetry processor**, not an initializer.
- If the question asks about 'pre-aggregating metrics', the answer is `GetMetric`, not `TrackMetric`.
- If the question asks about 'adding a property to all telemetry', the answer is a **telemetry initializer**.
- If the question involves 'reducing cost by dropping non-critical telemetry', the answer is **sampling** or a **processor**.
- If the question mentions 'ensuring no data loss during network outage', the answer is a **custom channel** with persistent storage.

## Step by step

1. **Initialize TelemetryClient Singleton** — Create a single instance of `TelemetryClient` using the Application Insights connection string. This instance should be reused across the application to avoid excessive resource consumption and to ensure proper batching. Use dependency injection in ASP.NET Core to register it as a singleton. The `TelemetryConfiguration` object can be configured with initializers and processors before creating the client. If you create multiple clients, they may share the same channel, but it's best practice to have one.
2. **Add Telemetry Initializers** — Implement `ITelemetryInitializer` to automatically add properties to every telemetry item, such as role name, environment, or user ID. The initializer's `Initialize` method is called for each telemetry item before processing. This ensures consistent context across all custom telemetry. For example, set `telemetry.Context.Cloud.RoleName` to the service name. Initializers are added to `TelemetryConfiguration.TelemetryInitializers`.
3. **Add Telemetry Processors** — Implement `ITelemetryProcessor` to filter or modify telemetry items. Processors form a chain; each processor passes the item to the next. Use for sampling, dropping sensitive events, or adding derived properties. The chain is built using `TelemetryProcessorChainBuilder`. Processors can inspect the item and decide to drop it (by not calling `Next.Process`). This runs after initializers but before the channel.
4. **Track Custom Telemetry** — Use methods like `TrackEvent`, `TrackMetric`, `TrackTrace`, `TrackDependency`, `TrackRequest`, and `TrackException` to send custom telemetry. For events and metrics, provide a name and optional properties/measurements. For metrics, prefer `GetMetric` for pre-aggregation to reduce volume. Each call creates a telemetry item (e.g., `EventTelemetry`) that is timestamped and queued in the channel's buffer.
5. **Flush and Shutdown Gracefully** — Before the application exits, call `TelemetryClient.Flush()` to send all buffered telemetry. The flush is asynchronous; wait a few seconds (e.g., `Task.Delay(5000)`) to allow the channel to send data. In ASP.NET Core, the SDK automatically flushes on shutdown if configured. Failure to flush may result in data loss. The channel's `SendingInterval` (default 30s) controls how often buffered items are sent.

## Comparisons

### TelemetryInitializer vs TelemetryProcessor

**TelemetryInitializer:**
- Adds or modifies properties on all telemetry items.
- Cannot drop telemetry items.
- Runs before telemetry processors.
- Implements ITelemetryInitializer.
- Example: Add 'Environment' property to all events.

**TelemetryProcessor:**
- Can filter, modify, or drop telemetry items.
- Can drop items by not calling Next.Process().
- Runs after telemetry initializers.
- Implements ITelemetryProcessor.
- Example: Drop events with name 'SensitiveEvent'.

### TrackMetric vs GetMetric

**TrackMetric:**
- Sends each metric data point individually.
- No pre-aggregation; high volume.
- Suitable for low-frequency metrics.
- Deprecated in favor of GetMetric.
- Simple to use but costly.

**GetMetric:**
- Pre-aggregates metrics before sending.
- Sends aggregated values (count, sum, min, max, stddev).
- Recommended for high-frequency metrics.
- Returns a Metric object that tracks aggregations.
- Reduces telemetry volume and cost.

## Common misconceptions

- **Misconception:** You can use multiple TelemetryClient instances for different telemetry types. **Reality:** You should use a single TelemetryClient instance. The SDK is designed to be thread-safe and handles all telemetry types. Multiple instances waste resources and may cause duplicate telemetry or channel conflicts.
- **Misconception:** Custom metrics should be sent individually with TrackMetric for each data point. **Reality:** Sending each data point individually generates high volume and cost. Use GetMetric to pre-aggregate metrics (e.g., count, sum, min, max) and send aggregated values at intervals. This reduces telemetry volume significantly.
- **Misconception:** Instrumentation key is the preferred way to configure Application Insights. **Reality:** Connection strings are now preferred because they include the ingestion endpoint, supporting sovereign clouds and reducing configuration errors. Instrumentation keys are deprecated for new applications.
- **Misconception:** Telemetry initializers can filter out telemetry items. **Reality:** Initializers cannot drop items; they only add or modify properties. To filter telemetry, use a telemetry processor, which can choose not to call Next.Process() to drop an item.
- **Misconception:** Flush() immediately sends all telemetry to the server. **Reality:** Flush() triggers the channel to send buffered items, but the send is asynchronous. You must wait a few seconds (e.g., Task.Delay(5000)) before the application exits to ensure data is transmitted. The channel uses a background thread.

## Key takeaways

- Use a single TelemetryClient singleton per application.
- Prefer connection string over instrumentation key.
- Use TelemetryInitializer to add common properties to all telemetry.
- Use TelemetryProcessor to filter or modify telemetry (can drop items).
- Use GetMetric for custom metrics to pre-aggregate and reduce volume.
- Call Flush() and wait before application shutdown to avoid data loss.
- Adaptive sampling is enabled by default with target 5 items/sec.
- To exempt an event from sampling, set SamplingPercentage = 100 in a processor.
- Maximum custom properties per event: 100; max property key length: 150; max value length: 8192.
- Default channel buffer size: 500 items; default send interval: 30 seconds.

## FAQ

**How do I send custom telemetry from a .NET Core console app?**

Install the `Microsoft.ApplicationInsights` NuGet package. Create a `TelemetryClient` with a connection string. Use `TrackEvent`, `TrackMetric`, etc. Call `Flush()` and `Task.Delay(5000)` before exit. Example: `var client = new TelemetryClient(new TelemetryConfiguration("connectionString")); client.TrackEvent("AppStarted"); client.Flush(); Task.Delay(5000).Wait();`

**What is the difference between TrackEvent and TrackTrace?**

`TrackEvent` is for custom events that represent significant occurrences (e.g., 'OrderPlaced'), with optional metrics. `TrackTrace` is for diagnostic log messages with severity levels (e.g., 'Processing file X'). Both support custom properties, but events are typically used for business analytics, traces for debugging.

**How do I prevent a critical custom event from being sampled?**

Implement a telemetry processor that sets `item.SamplingPercentage = 100` for the event. This marks the event as exempt from sampling. Example: `if (item is EventTelemetry eventTelemetry && eventTelemetry.Name == "CriticalEvent") { eventTelemetry.SamplingPercentage = 100; }`

**Can I send custom telemetry without using the SDK?**

Yes, you can send HTTP POST requests directly to the Application Insights ingestion endpoint. The endpoint is `https://dc.services.visualstudio.com/v2/track`. You must format the payload as JSON with an array of telemetry items. However, using the SDK is recommended for reliability, retries, and ease of use.

**How do I configure Application Insights for an Azure Function?**

Azure Functions automatically integrate with Application Insights when you enable it in the function app settings. You can inject `TelemetryClient` into your function using dependency injection. The SDK is already configured; you just need to add the NuGet package and use the client.

**What happens if the telemetry channel buffer overflows?**

By default, items are dropped (lost). You can configure a custom overflow handler or increase the buffer size (`MaxTelemetryBufferCapacity`). Alternatively, you can implement a custom channel that persists to disk. The default `InMemoryChannel` drops items silently.

**How do I view custom events in Application Insights?**

In the Azure portal, go to your Application Insights resource, then under 'Investigate' select 'Events' or 'Search'. You can query custom events using the 'customEvents' table in Log Analytics. Use Kusto queries like `customEvents | where name == "MyEvent"`.

---

Interactive version with quiz and diagrams: https://courseiva.com/learn/az-204/app-insights-telemetry