AWS Distro for OpenTelemetry

AWS Distro for OpenTelemetry (ADOT) is a distribution of the open-source OpenTelemetry project, maintained and supported by AWS. It provides a single set of libraries, agents, and collectors to collect distributed traces and metrics from your applications and send them to AWS monitoring services such as Amazon CloudWatch, AWS X-Ray, and Amazon Managed Service for Prometheus. ADOT is designed to be a drop-in replacement for the AWS X-Ray SDK and other proprietary instrumentation libraries, allowing you to instrument your code once and send telemetry data to multiple backends.

Before ADOT, developers had to use different SDKs for different monitoring backends. For example, to send traces to AWS X-Ray, you had to use the X-Ray SDK, which was tightly coupled to X-Ray. If you later wanted to send the same traces to a third-party tool like Jaeger or Zipkin, you needed additional instrumentation. OpenTelemetry solves this by providing a vendor-agnostic standard for telemetry data. ADOT is AWS's supported distribution that includes additional components to integrate seamlessly with AWS services, such as the AWS X-Ray exporter and the AWS CloudWatch exporter. The exam tests your understanding that ADOT is the recommended path for new projects and that it can replace the X-Ray SDK.

ADOT consists of three main components: the OpenTelemetry API/SDK (instrumentation libraries), the OpenTelemetry Collector (an agent that receives, processes, and exports telemetry data), and AWS-specific exporters and extensions.

#### Instrumentation Libraries

You instrument your application code using OpenTelemetry API/SDKs. These libraries automatically generate traces and metrics from your application without requiring manual code changes. For example, the OpenTelemetry Java SDK can automatically instrument common libraries like HTTP clients, JDBC drivers, and AWS SDK calls. The SDK produces telemetry data in the OpenTelemetry Protocol (OTLP) format.

#### OpenTelemetry Collector

The Collector is a vendor-agnostic agent that can receive telemetry data in multiple formats (including OTLP, Jaeger, Zipkin, and Prometheus), process it (e.g., batch, filter, sample, add attributes), and export it to one or more backends. ADOT provides a pre-built distribution of the Collector with AWS-specific exporters. The Collector can be deployed as a sidecar, a daemonset on Kubernetes, or as a standalone service.

#### AWS Exporters

ADOT includes exporters that convert OTLP data into AWS-native formats: - AWS X-Ray Exporter: Converts OTLP traces into X-Ray segment documents and sends them to the X-Ray API. This allows you to use X-Ray's service maps, traces, and analytics. - AWS CloudWatch Exporter: Converts OTLP metrics into CloudWatch metrics and sends them via the CloudWatch PutMetricData API. - AWS Managed Service for Prometheus Exporter: Converts OTLP metrics into Prometheus remote write format and sends them to Amazon Managed Service for Prometheus.

#### Installing ADOT Collector on Amazon ECS

You can deploy the ADOT Collector as a sidecar container in your ECS task definition. Example task definition snippet:

{
  "containerDefinitions": [
    {
      "name": "adot-collector",
      "image": "public.ecr.aws/aws-observability/aws-otel-collector:latest",
      "environment": [
        {
          "name": "AWS_REGION",
          "value": "us-east-1"
        }
      ],
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "/ecs/adot-collector",
          "awslogs-region": "us-east-1",
          "awslogs-stream-prefix": "ecs"
        }
      }
    },
    {
      "name": "my-app",
      "image": "my-app:latest",
      "environment": [
        {
          "name": "OTEL_EXPORTER_OTLP_ENDPOINT",
          "value": "http://localhost:4317"
        }
      ]
    }
  ]
}

#### Configuring the Collector (otel-config.yaml)

The Collector configuration file defines receivers, processors, and exporters. Example:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 200ms
    send_batch_size: 8192

exporters:
  awsxray:
    region: us-east-1
  awscloudwatch:
    namespace: "MyAppMetrics"
    region: us-east-1

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [awsxray]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [awscloudwatch]

#### Verifying ADOT Operation

A large e-commerce company has been using the AWS X-Ray SDK for Java to trace requests across their microservices. They want to adopt a vendor-neutral instrumentation approach to also send traces to a self-hosted Jaeger instance for internal analysis. The X-Ray SDK is tightly coupled to X-Ray, making it difficult to export to multiple backends. The team decides to migrate to ADOT. They replace the X-Ray SDK with the OpenTelemetry Java agent and deploy the ADOT Collector as a sidecar in each ECS task. The Collector is configured with two exporters: awsxray and otlp (to send to Jaeger). This allows them to maintain X-Ray integration while adding Jaeger support without changing application code. The migration is seamless because ADOT maps OpenTelemetry spans to X-Ray segments automatically. The team also adds sampling rules in the Collector to reduce trace volume by 90%, saving costs. In production, they run 200 microservices across 500 ECS tasks, with each Collector processing ~10,000 spans/second. They tune the batch processor to send 8192 spans every 200ms to optimize API call costs.

A financial services company uses CloudWatch for monitoring but also wants to use Prometheus for alerting. They have a mix of Java and Python applications running on Amazon EKS. Previously, they ran the CloudWatch agent and a Prometheus server separately. With ADOT, they deploy the ADOT Collector as a DaemonSet on each EKS node. The Collector is configured to receive OTLP metrics from instrumented applications and also scrape Prometheus endpoints. It then exports metrics to both CloudWatch and Amazon Managed Service for Prometheus (AMP). This reduces operational overhead by having a single agent. The Collector also adds metadata like Kubernetes pod labels to metrics. They configure the batch processor to aggregate metrics every 60 seconds to reduce CloudWatch costs. A common misconfiguration is forgetting to set IAM permissions for AMP remote write, causing metrics to be silently dropped. The team monitors the Collector's internal metrics (exposed on port 8888) to detect export failures.

A serverless application uses AWS Lambda for data processing. The team wants to trace invocations and send custom metrics to CloudWatch. They use the ADOT Lambda layer for Python, which includes the OpenTelemetry Python SDK and the ADOT Collector as a Lambda extension. The layer automatically instruments the Lambda runtime and AWS SDK calls. Traces are sent to X-Ray, and metrics are sent to CloudWatch. The team configures the Collector in the Lambda function's extensions directory. A common pitfall is exceeding the Lambda execution time due to the Collector's overhead; they mitigate this by increasing the Lambda timeout to 30 seconds and using asynchronous export. They also set the OTEL_BSP_SCHEDULE_DELAY environment variable to 1000ms to reduce export frequency.