# Monitoring and Debugging with AWS X-Ray

> Chapter 16 of the Courseiva AWS-DEVELOPER-ASSOCIATE curriculum — https://courseiva.com/learn/aws-developer-associate/x-ray-distributed-tracing

**Official objective:** Domain 4 — Instrument applications, analyze traces, and identify performance bottlenecks.

## Introduction

AWS X-Ray is a service that helps you see inside your application, tracing each user request as it moves through different parts of your system. For the DVA-C02 exam, you need to know how to instrument your code with X-Ray to find performance problems and fix them — a skill that separates junior developers from senior ones.

## The Kitchen Order Tracking Analogy

3 seconds. That is how long it takes the head chef to realise the error when she checks the digital order board and sees that ticket #47 has been sitting in the 'prep station' status for 12 minutes — 8 minutes longer than normal. The ticket was created when the waiter entered the table's order into the system. As it moved from the salad station to the grill station to the plating station, each chef tapped a button to log the start and end time of their work. A green line on the board traced the ticket's entire path through the kitchen. Now the head chef knows the holdup: the new junior cook at the grill is taking too long on medium-rare steaks. She can see every step of that single ticket's journey, compare it to the average time for similar orders, and even check if the delay was caused by a missing ingredient from the walk-in cooler. This is exactly how AWS X-Ray works for your application's code. X-Ray creates a 'trace' for each user request — like a digital ticket — and tracks it as it travels through every service, database, and API call. You can see exactly where each request slows down, fails, or succeeds. You can find the 'head chef' view that pinpoints the 'grill station' bottleneck in milliseconds, without guessing or adding print statements to your code.

## Core explanation

When you run a modern application in the cloud, especially one made of many small services (called microservices), a single user request might touch 10 or 20 different components. It might hit a load balancer, then a web server, then a database, then another API service, then a cache. If that request is slow or fails, how do you find the culprit? Before X-Ray, developers had to add logging statements everywhere — 'I am here', 'now I am here' — and then search through thousands of log lines to piece together what happened. This is like trying to find a single wrong turn in a cross-country road trip by reading every mile marker sign in a stack of papers.

AWS X-Ray solves this by giving every incoming request a unique identifier called a trace ID. Think of the trace ID as a shipping tracking number for a parcel. As the request travels through your application, X-Ray collects data about each step, or segment. Each segment records what happened (like 'user logged in'), how long it took (like '47 milliseconds'), and whether it succeeded or failed. A subsegment is a smaller piece of work inside a segment, like the database query that runs when a user logs in.

X-Ray works by adding a small piece of code — called the X-Ray SDK (Software Development Kit) — to your application. The SDK automatically captures data for common services like AWS Lambda, Amazon API Gateway, Amazon DynamoDB, and many others. You can also add custom instrumentation to track specific parts of your own code. The SDK sends this data to the X-Ray service, which processes it and creates a service graph — a visual map that shows all the services in your application and how they connect.

Here is a key concept: sampling. If you traced every single request, X-Ray would generate a huge amount of data and cost a lot. Instead, X-Ray uses a sampling rate — by default, it records the first request per second, and then 5% of all subsequent requests. You can adjust this. For the exam, remember that you need to set sampling rules carefully to control costs while still getting enough data to spot trends.

The X-Ray SDK provides two main ways to add tracing to your own code:

- Custom subsegments: You wrap a block of code with a subsegment to measure its performance. For example, if you have a function that resizes an image, you can create a subsegment called 'image_resize' to see exactly how long it takes.
- Annotations and metadata: Annotations are simple key-value pairs (like 'user_tier: premium') that you can use to filter traces in the X-Ray console. Metadata is richer data (like a full JSON object) that you can view in trace details but cannot filter on.

When a request causes an error, X-Ray can capture details about the exception, including the stack trace — a list of the function calls that led to the error. This helps you identify which line of code caused the problem.

X-Ray integrates directly with other AWS services. For example, if you use AWS Lambda, you can enable X-Ray tracing in the Lambda console with a single click. API Gateway can also be configured to send traces to X-Ray. Elastic Load Balancing can pass trace headers to your backend instances so the tracing is continuous.

You view all this data in the AWS Management Console. The console shows you:
- The service map: a visual graph of your application.
- Trace list: a table of recent traces you can sort by duration, status, or time.
- Trace details: a timeline view of a single request, showing every segment and subsegment in order.

X-Ray also provides a way to detect anomalies. You can create groups of traces that share a common characteristic (like 'all requests that go to the checkout service') and set alarms when those groups show high error rates or long durations.

## Real-world context

Imagine you work for an e-commerce company that sells custom sneakers. The website lets users design their own shoes, add them to a cart, and check out. The application has several services: a web frontend, a user authentication service, a design rendering service, an inventory service, a payment service, and an order processing service. One day, customers start complaining that the checkout page takes over 30 seconds to load. Orders are timing out and sales are dropping.

As the developer, you need to find the bottleneck fast. You cannot guess — you need data. This is where X-Ray comes in.

First, you check the X-Ray service map in the console. The map shows each service as a circle, with arrows between them showing traffic flow. The circles are coloured: green for normal, yellow for slowing down, red for errors. You see that the payment service circle is red and has a thick arrow connecting to the order processing service. That is where you focus.

You open the trace list and filter for traces that went through the payment service in the last hour. You sort by duration, longest first. You open the slowest trace — a checkout request that took 45 seconds. The trace details show a timeline. Each segment is a horizontal bar. You see:
- Web frontend: 45 seconds total.
- User auth: 2 milliseconds.
- Design rendering: 150 milliseconds.
- Inventory check: 200 milliseconds.
- Payment processing: 44 seconds. 
- Inside payment processing, there is a subsegment called 'charge_credit_card' that took 43.8 seconds.

You now know the exact service (payment) and the exact function (charge_credit_card) causing the delay. The subsegment details show an error: 'timeout waiting for upstream provider'. The payment service is calling an external credit card processor, and that external service is slow.

You then look at the annotations on this trace. You have added an annotation called 'payment_provider' to each payment subsegment. You can see that 90% of slow traces use 'provider_B', while traces using 'provider_A' are fast. You tell your team to switch all traffic to provider_A while you investigate why provider_B is failing.

With X-Ray, you diagnose a production issue in minutes without restarting servers or adding temporary logging code. You can also set up a recurring report using X-Ray analytics to monitor the average duration of the checkout flow and receive an alert if it goes above 5 seconds.

In practice, IT professionals use X-Ray for:
- Root cause analysis: Finding exactly which service or code path causes a failure.
- Performance optimisation: Identifying bottlenecks to reduce latency.
- Capacity planning: Understanding how request patterns affect resource usage.
- Compliance auditing: Demonstrating that requests are processed correctly.
- Cost reduction: By finding and fixing slow code that uses more compute time.

## Exam focus

The DVA-C02 exam tests X-Ray in several specific ways. You will not be asked to configure X-Ray from scratch, but you need to know how it works and how to use it with common services.

The exam focuses on these concepts:

- Tracing with Lambda: You must know that you enable X-Ray tracing on a Lambda function via the console or the AWS SAM template. When tracing is enabled, Lambda sends traces for each invocation. The exam will ask what happens when you enable active tracing on a Lambda function — the answer is that Lambda sends trace data to X-Ray.
- Sampling rules: The exam loves to test sampling. Remember the default: 1 request per second plus 5% of additional requests. You can adjust the sampling rate using the X-Ray SDK or the X-Ray API. A common question: 'How can you reduce the number of traces recorded to save costs?' Answer: adjust the sampling rate.
- X-Ray with API Gateway: You can enable X-Ray tracing on your API Gateway stages. When you do, API Gateway adds a trace header to incoming requests and forwards it to the backend. You need to ensure your backend code (or Lambda) passes the trace header correctly to keep the trace continuous.
- Segments and subsegments: You will be asked to distinguish between them. A segment covers the top-level work in a service. A subsegment is for lower-level work, like a database query or an HTTP call to another service. A subsegment must be closed before its parent segment can be closed.
- Annotations vs. Metadata: Annotations are for filtering — you define a key and a value, and you can search on them. Metadata is for debugging — detailed data you can look at in a trace but cannot filter on. The exam expects you to know when to use each.
- X-Ray daemon: This is a process you run on your EC2 instances or on-premises servers. The daemon receives trace data from the X-Ray SDK and sends it to the X-Ray service. You need to install and configure the daemon for EC2-based applications.
- Trace headers: The X-Ray SDK uses the `X-Amzn-Trace-Id` header to pass trace information between services. If you have a custom service, you must forward this header to keep the trace intact. The exam might test you on why a trace is broken — the cause is often a missing or corrupted trace header.

Common traps:
- Thinking that X-Ray can automatically trace any application without any code changes. Reality: you must instrument your code with the SDK.
- Confusing X-Ray with CloudWatch Logs. CloudWatch Logs collects log data, but X-Ray traces the path of a request across services. They complement each other.
- Believing that X-Ray traces 100% of requests by default. Reality: it uses sampling to limit data volume.
- Assuming that enabling X-Ray on a Lambda function automatically traces all downstream calls. Reality: you still need to instrument downstream services with the SDK to get full traces.

Key definitions to memorise:
- Trace: the complete journey of a single request.
- Segment: the record of work done by a single service for a request.
- Subsegment: a smaller unit of work within a segment.
- Service graph: a visual map of all services and their connections.
- Sampling: deciding which requests to trace to control data volume and cost.

## Step by step

1. **Enable tracing on your AWS resources** — Turn on X-Ray tracing for each service in your application. For AWS Lambda, you check a box in the console or set `TracingConfig` to `Active` in your SAM template. For API Gateway, you enable tracing on the stage. This tells these services to generate trace data and send it to X-Ray.
2. **Instrument your application code with the X-Ray SDK** — Install the X-Ray SDK (one exists for Python, Node.js, Java, .NET, and Go) into your application. Import the SDK and use its methods to wrap your code in subsegments. For example, in Python: `from aws_xray_sdk.core import xray_recorder` and then `xray_recorder.begin_subsegment('process_payment')`. This captures timing and error data for that specific block of code.
3. **Add annotations and metadata to your traces** — Annotate your subsegments with key-value pairs like 'user_role: admin' or 'region: eu-west-1'. You can later filter the trace list by these annotations. Metadata can be used to attach a full JSON object with detailed request data, which you can view in the trace details but cannot search on.
4. **Install and run the X-Ray daemon on non-AWS servers** — If your application runs on EC2 or an on-premises server, you must install the X-Ray daemon. The daemon is a background process that listens for trace data from the SDK on a local port (UDP 2000) and forwards it to the AWS X-Ray API. Without the daemon, the SDK cannot send traces.
5. **Analyse traces in the X-Ray console** — Open the AWS Management Console and go to X-Ray. Use the service map to see a visual graph of your application's architecture. Use the trace list to sort traces by duration, status, or time. Click on a single trace to see a timeline of every segment and subsegment, including which ones failed or were slow.
6. **Create sampling rules and alarms** — Define custom sampling rules using the X-Ray console or API. For example, you can set a higher sampling rate for requests that go to your checkout service so you capture more data on that critical flow. Then, use CloudWatch alarms to trigger if error rates or durations exceed thresholds for specific X-Ray trace groups.

## Comparisons

### Annotations vs Metadata

**Annotations:**
- Simple key-value pairs (string, number, boolean)
- You can filter and search traces based on annotation values
- Used for indexing and categorising traces for quick analysis

**Metadata:**
- Can store larger data like JSON objects
- You cannot filter or search traces on metadata values
- Used for debugging details you view in a specific trace

### Segment vs Subsegment

**Segment:**
- Represents the top-level work done by one service for a request
- Covers the entire time the service spends on the request
- Can contain multiple subsegments inside it

**Subsegment:**
- Represents a smaller unit of work within a segment (e.g., a function call)
- Must be opened and closed within its parent segment
- Cannot exist without a parent segment

### X-Ray vs CloudWatch Logs

**X-Ray:**
- Traces the path of a request across multiple services
- Shows latency and errors per segment or subsegment
- Presents data in a service graph and timeline view

**CloudWatch Logs:**
- Stores plain text log output from your application
- Used to read error messages, stack traces, and custom log statements
- Presents data as a list of log events with timestamps

### Active Tracing (Lambda) vs Pass Through Tracing (Lambda)

**Active Tracing (Lambda):**
- Lambda creates and sends trace segments to X-Ray
- Requires no explicit SDK code in the function itself
- Captures basic Lambda execution time and result

**Pass Through Tracing (Lambda):**
- Lambda forwards the incoming trace header but does not generate its own segments
- Used when you want to propagate tracing from an upstream service without adding Lambda instrumentation
- Trace will show a gap in the timeline for Lambda's internal work

## Common misconceptions

- **Misconception:** X-Ray automatically traces all AWS services without any configuration. **Reality:** You must enable tracing on each service (like Lambda or API Gateway) and instrument your code with the X-Ray SDK. Some services support one-click enablement, but custom code still needs SDK instrumentation. (Many beginners assume 'serverless' means everything is automatic. In reality, distributed tracing requires explicit opt-in and code changes.)
- **Misconception:** X-Ray works without any changes to your application code. **Reality:** You need to add the X-Ray SDK to your application and wrap code in subsegments to get custom tracing. Without the SDK, X-Ray will only capture basic data from integrated services like Lambda. (The name 'X-Ray' sounds like a passive inspection tool. Beginners think it is a 'magic glass' that sees through code, not an active monitoring system that requires instrumentation.)
- **Misconception:** X-Ray records every single request if you enable it on a Lambda function. **Reality:** By default, X-Ray uses sampling: it records the first request per second and then 5% of subsequent requests. You can change the sampling rate, but recording 100% of requests is usually too expensive and unnecessary. (The exam often traps candidates who think 'enabled' means 'all'. Sampling is a counter-intuitive concept for beginners who want full visibility.)
- **Misconception:** X-Ray and CloudWatch Logs serve the same purpose and you only need one. **Reality:** CloudWatch Logs stores log output from your application (text messages, errors, timestamps). X-Ray traces the path of a request across services and shows latency and errors per segment. They serve different audiences and are used together for complete observability. (Beginners see both as 'monitoring' tools and assume they are redundant. The exam tests their complementary roles.)
- **Misconception:** If I enable X-Ray on API Gateway, the trace automatically continues through my backend Lambda function. **Reality:** API Gateway will add a trace header and forward it, but your Lambda function must have X-Ray tracing enabled separately. If the Lambda function does not have active tracing, the trace will appear broken in the X-Ray console. (The concept of 'header propagation' is subtle. Beginners assume a single toggle controls the entire flow, but each service independently decides whether to participate in tracing.)

## Key takeaways

- AWS X-Ray traces the complete journey of a single user request as it travels across multiple services, showing latency and errors at each step.
- You must enable X-Ray tracing on each service individually — enabling it on one service does not automatically trace downstream services.
- The default sampling rule is one request per second plus 5% of additional requests; you can adjust this to balance cost and visibility.
- Use annotations for filtering traces in the console (key-value pairs you can search on), and metadata for storing detailed debugging data you do not need to search on.
- A subsegment must be closed before its parent segment can be closed; otherwise, the trace will be incomplete or generate an error.
- The X-Ray daemon must be running on EC2 instances or on-premises servers to collect and send trace data to the AWS X-Ray service.
- A broken trace is often caused by a missing or corrupted 'X-Amzn-Trace-Id' header when a request crosses between services.

## FAQ

**Does AWS X-Ray cost extra money on top of my other AWS services?**

Yes. AWS X-Ray charges based on the number of traces recorded and the amount of data stored. You set the sampling rate to control costs. The first 100,000 traces per month are free, then you pay per trace.

**Can I use X-Ray with an application that is not running on AWS?**

Yes, you can run the X-Ray daemon on your own servers (on-premises) and send traces to X-Ray. The daemon sends data over the internet to the AWS X-Ray API. You need to configure your firewall to allow outbound traffic.

**What happens if my Lambda function fails to send a trace to X-Ray?**

The Lambda function will still run normally — X-Ray tracing is non-blocking. If the trace fails to send, you lose that trace data but the function execution is unaffected. Errors are logged to CloudWatch Logs by the X-Ray SDK.

**How do I see which database query caused a slow response using X-Ray?**

If you instrument your database calls with the X-Ray SDK, each query becomes a subsegment with its duration and SQL statement (if you choose to capture it). In the trace details timeline, you will see the database subsegment as a horizontal bar. Click it to see the query text and duration.

**Is X-Ray the same as AWS CloudTrail?**

No. CloudTrail records API calls made to AWS services (like 'someone created an S3 bucket'). X-Ray traces application-level requests (like 'a user logged in'). They serve different purposes. X-Ray is for debugging application performance, CloudTrail is for auditing AWS account activity.

**Can I use X-Ray to trace requests that go through a load balancer?**

Yes. Elastic Load Balancing (ALB/NLB) can be configured to pass the X-Ray trace header to your backend instances. You must also enable X-Ray tracing on the backend instances or containers for the trace to continue.

---

Interactive version with quiz and diagrams: https://courseiva.com/learn/aws-developer-associate/x-ray-distributed-tracing