X-Ray Segments, Annotations, and Sampling

AWS X-Ray models a single request through a distributed application as a trace, which is composed of segments and subsegments. A segment represents the work done by a single service (e.g., an API Gateway endpoint, a Lambda function, or an EC2 instance) as it processes a request. Each segment has a start time, end time, duration, and optional error/fault/throttle flags. The first segment created for a trace is the root segment, which typically represents the entry point (e.g., API Gateway or an instrumented web server).

Subsegments break down the work within a segment into finer-grained units, such as downstream calls (e.g., DynamoDB queries, HTTP calls to another microservice), or internal logic (e.g., database query, file processing). Subsegments are nested inside a segment and can themselves have subsegments. They are critical for pinpointing where time is spent within a single service.

When a request enters your application, the X-Ray SDK creates a trace ID (a unique identifier, e.g., 1-5e3b3b3b-abcdef1234567890abcdef12) and a segment ID. The SDK sends the segment data to the X-Ray daemon, which buffers and forwards it to the X-Ray API. If the application makes downstream calls, the SDK propagates the trace header (containing the trace ID, parent segment ID, and sampling decision) via HTTP headers (e.g., X-Amzn-Trace-Id) or other protocols. The receiving service creates a new segment as a child of the calling segment, forming a trace tree.

Annotations are simple key-value pairs with string, number, or boolean values. They are indexed by X-Ray for filtering and searching traces. For example, you can annotate a segment with "customer_tier": "premium" and later search for all traces where customer_tier = premium. Annotations are limited to 50 per segment, and keys can be up to 512 characters, values up to 1024 characters (strings). They are ideal for high-cardinality attributes you need to query on.

Metadata are arbitrary key-value pairs that can hold any data type (including objects and arrays). They are not indexed and cannot be used for filtering. Metadata is stored alongside the segment but only appears in the raw segment details when you view a trace. Use metadata for debugging information that you don't need to search on, such as full request/response payloads, stack traces, or internal state.

X-Ray sampling determines which requests are traced. By default, X-Ray uses centralized sampling where the X-Ray service manages sampling rules. You can also define local sampling rules in the SDK configuration. The default sampling rate is 1 request per second (reservoir) and 5% of additional requests (rate). This means the first request each second is always traced, and then 5% of the remaining requests are traced. This is called reservoir and rate sampling.

You can create custom sampling rules to override defaults for specific services, HTTP methods, URLs, or hostnames. Rules are evaluated in order of priority (lowest number = highest priority). The first matching rule is applied. If no rule matches, the default rule is used.

The sampling decision (trace or not) is made at the first service that receives the request (e.g., API Gateway or load balancer). That decision is propagated downstream via the trace header. Downstream services must honor the decision; they cannot override it to start tracing a request that was not sampled. However, they can generate subsegments even if the parent segment was not sampled? No—if the root segment is not sampled, no segments are sent. But the SDK will still propagate the decision. If a downstream service receives a request with a sampling decision of "not sampled," it should not create a segment. However, the SDK allows a service to generate a segment for its own work if it has a local sampling rule that overrides? Actually, the SDK follows the propagated decision; local rules only apply when the service is the origin of the trace (i.e., the first service). For incoming requests with a trace header, the SDK uses the decision in the header.

Subsegments are created for synchronous calls automatically by the SDK (e.g., DynamoDB, S3). For asynchronous work (e.g., SQS, Lambda invocation), you must manually create subsegments using AWSXRay.beginSubsegment() and AWSXRay.endSubsegment(). The exam tests that you know subsegments can be created manually for custom instrumentation.

When you enable X-Ray tracing on a Lambda function, Lambda automatically sends trace data for the Lambda service segment (invocation, cold start, duration). Your function code can use the X-Ray SDK to add custom subsegments and annotations. Lambda also propagates the _X_AMZN_TRACE_ID environment variable containing the trace ID, parent segment ID, and sampling decision. The X-Ray daemon is automatically run inside the Lambda execution environment (since the runtime supports it).

API Gateway can be configured to pass trace headers and can also generate its own segment. When X-Ray tracing is enabled on an API Gateway stage, API Gateway sends a segment for each request. The segment includes the request path, method, and response status. API Gateway can also propagate the trace header to the backend integration.

To enable X-Ray on Lambda (using the AWS CLI):

aws lambda update-function-configuration --function-name my-function --tracing-config Mode=Active

To create a sampling rule:

aws xray create-sampling-rule --sampling-rule file://sampling-rule.json

Example sampling-rule.json:

{
  "RuleName": "MyRule",
  "ResourceARN": "*",
  "ServiceName": "*",
  "ServiceType": "*",
  "Host": "*",
  "HTTPMethod": "GET",
  "URLPath": "/api/*",
  "Priority": 10,
  "FixedRate": 0.1,
  "ReservoirSize": 5
}

X-Ray integrates with many AWS services: Elastic Beanstalk, EC2, ECS, Lambda, API Gateway, SNS, SQS, DynamoDB, and more. These services can automatically send segments and propagate trace headers. For example, SQS can propagate the trace header from producer to consumer, allowing end-to-end tracing across async boundaries.

This depth of understanding is exactly what the DVA-C02 exam requires.

A large e-commerce platform uses a microservice architecture for checkout: API Gateway → Lambda (order processing) → DynamoDB (write order) → SNS (notify shipping). They enable X-Ray on all services. Initially, they only see segments from API Gateway and Lambda, but the DynamoDB call appears as a generic subsegment. They add annotations like orderId, customerTier, and paymentMethod. This allows the operations team to search for all failed checkouts for premium customers. They also add metadata containing the full order payload for debugging. They configure a custom sampling rule to trace 100% of requests for customerTier = "premium" by setting a high priority rule with FixedRate=1.0 and ReservoirSize=1000. For other customers, they use the default sampling. This saves costs while ensuring premium customers are fully traced.

A video processing pipeline uses S3 → Lambda (thumbnail generation) → SQS → EC2 (video encoding). They enable X-Ray on Lambda and EC2. However, the EC2 instances are not instrumented with the X-Ray SDK, so traces end at Lambda. They install the X-Ray daemon on EC2 and instrument the encoding code. They add annotations for videoId and resolution. They face a problem: the default sampling (1 reservoir, 5% rate) is too low for the high volume (10,000 requests/sec). They increase the reservoir to 100 and rate to 10% but still miss important traces. They then create a rule to trace 100% of requests for a specific video that is being tested. They also realize that the SQS async call breaks the trace; they manually propagate the trace header by reading it from the SQS message attributes and passing it to the EC2 worker. This restores the end-to-end trace.

A company builds a serverless API using API Gateway with a Lambda authorizer. They enable X-Ray on both the authorizer and the backend Lambda. They notice that the authorizer segment appears as a separate trace because the authorizer runs before the backend. They add annotations to both segments: userId from the JWT token. They create a sampling rule that traces 100% of requests for users in the "admin" group. They also add metadata containing the decoded JWT payload for debugging authorization failures. They discover that the authorizer sometimes throws errors; by examining the subsegments, they pinpoint a DynamoDB call that times out. They increase the DynamoDB timeout and add retry logic. X-Ray helped them reduce mean time to resolution from hours to minutes.