Distributed Tracing with Application Insights

Distributed tracing is a method used to profile and monitor applications, especially those built using a microservices architecture. In a monolithic application, a single process handles a request end-to-end, making it easy to log and trace. However, in a distributed system, a single user request may trigger calls to multiple services, databases, queues, and caches. Traditional logging produces isolated logs in each component, making it nearly impossible to correlate them to understand the full flow. Distributed tracing solves this by assigning a unique trace ID to the root request and propagating it across all service boundaries. Each unit of work within a service is recorded as a span with a span ID and a parent span ID, forming a tree structure. This allows developers and operators to reconstruct the entire request path, measure latency at each hop, and identify bottlenecks or failures.

The mechanism relies on three key identifiers: trace ID, span ID, and parent span ID. When a request first enters the system (e.g., via an HTTP request to a front-end service), the tracing system generates a globally unique trace ID (usually a 128-bit or 64-bit number, often represented as a hex string). It also creates the first span, called the root span, with a unique span ID. The root span has no parent (parent span ID is null or empty). As the front-end service calls downstream services (e.g., via HTTP, gRPC, or message queues), it propagates the trace ID and its own span ID as the parent span ID to the downstream service. The downstream service then creates a new span with its own span ID, sets the parent span ID to the incoming span ID, and continues. This creates a hierarchical tree where each span represents a logical unit of work (e.g., a database query, a cache lookup, an external API call).

Application Insights uses the OpenTelemetry standard (formerly OpenCensus) for distributed tracing. The Application Insights SDK automatically instruments common frameworks (ASP.NET Core, Java, Node.js, etc.) to create and propagate spans. For example, in an ASP.NET Core application, the SDK intercepts incoming HTTP requests and outgoing HTTP calls, generating DependencyTelemetry for each outbound call. Each DependencyTelemetry object contains the trace ID, span ID, and parent span ID. The SDK also supports manual instrumentation using the TelemetryClient API to create custom dependencies or track custom operations.

To enable distributed tracing in an ASP.NET Core application, you need to add the Application Insights SDK and configure the connection string. In the appsettings.json file:

{
  "ApplicationInsights": {
    "ConnectionString": "InstrumentationKey=...;IngestionEndpoint=..."
  }
}

In Startup.cs (or Program.cs for .NET 6+), add:

using Microsoft.ApplicationInsights.AspNetCore.Extensions;
using Microsoft.Extensions.DependencyInjection;

services.AddApplicationInsightsTelemetry();

For manual instrumentation, you can use the TelemetryClient:

using Microsoft.ApplicationInsights;
using Microsoft.ApplicationInsights.DataContracts;

var telemetryClient = new TelemetryClient();
using (var operation = telemetryClient.StartOperation<RequestTelemetry>("MyCustomOperation"))
{
    // Your code
    operation.Telemetry.Success = true;
}

To verify that distributed tracing is working, you can query Application Insights using Log Analytics (Kusto). For example, to find all telemetry for a specific trace:

union requests, dependencies, exceptions, traces
| where operation_Id == "<trace_id>"
| order by timestamp asc
| project timestamp, itemType, name, id, operation_ParentId, duration

Distributed tracing in Application Insights integrates with Azure Monitor and other Azure services. For example, you can correlate traces with Azure Functions, Azure Kubernetes Service (AKS), and Azure Service Bus. When using Azure Functions, the host automatically propagates the trace context via the traceparent header, and the Application Insights SDK in the function app creates spans for each function execution. For AKS, you can enable Azure Monitor for containers, which collects Prometheus metrics and container logs, and integrates with Application Insights for distributed tracing. Additionally, Application Insights supports OpenTelemetry, allowing you to use the OpenTelemetry SDK to export traces to Application Insights. This enables interoperability with other observability tools like Jaeger or Zipkin.

Distributed tracing generates additional telemetry data, which can impact performance and cost. The overhead of generating and sending telemetry is minimal (usually <1% CPU), but the volume of data can be significant in high-throughput systems. Use sampling to control costs: head-based sampling (decide at the start of the trace) ensures that entire traces are either sampled or not, which preserves the trace tree. Adaptive sampling adjusts automatically based on traffic. You can also set limits on the number of telemetry items sent per second using the TelemetryChannel configuration. In production, it is common to sample at 10% or lower for high-traffic services while keeping 100% sampling for critical paths like payment processing.

A large e-commerce platform uses a microservices architecture with separate services for user authentication, product catalog, shopping cart, order processing, payment, and shipping. Each service is deployed in Azure Kubernetes Service (AKS). When a customer places an order, the front-end service calls the order service, which then calls payment and shipping services, each of which may call other internal APIs. Without distributed tracing, diagnosing a slow order is nearly impossible. The team implements Application Insights distributed tracing across all services using the OpenTelemetry SDK. They configure a fixed sampling rate of 20% to balance cost and visibility. In production, they notice that the payment service occasionally takes >10 seconds. Using the transaction search, they find a specific trace where the payment service's span shows a long duration. They drill down and see that the payment service made a call to an external fraud detection API that timed out. The team adds a circuit breaker and reduces the timeout. They also set up alerts using Application Insights metrics for p95 latency. The result is a 50% reduction in order processing time.

A company manages thousands of IoT devices that send telemetry to an Azure IoT Hub, which then processes data through Azure Functions and stores results in Azure Cosmos DB. Each device message triggers a chain of functions. To monitor the end-to-end flow, the team enables distributed tracing in the Azure Functions host (using the built-in Application Insights integration). They manually propagate the trace context through IoT Hub messages by adding the traceparent header to the device-to-cloud messages. In the portal, they can see a trace for each message, showing the time spent in each function and in Cosmos DB. They discover that one function is a bottleneck because it makes a synchronous HTTP call to a weather API. They change it to an asynchronous pattern and reduce the function timeout. They also use sampling to avoid high costs from the high volume of device messages.