Route 53 Health Checks and DNS Failover

Route 53 health checks are automated probes that monitor the availability and health of your endpoints (e.g., web servers, load balancers, or other AWS resources). They exist to enable DNS-level failover: if an endpoint becomes unhealthy, Route 53 can automatically route traffic away from it to a healthy endpoint. This is critical for building fault-tolerant applications that can survive individual component failures without manual intervention.

Route 53 health checks operate from a global network of health checkers distributed across AWS regions. When you create a health check, Route 53 assigns a set of these checkers to probe your endpoint from multiple locations. The health checkers send requests based on the type of check: - TCP: Opens a TCP connection to the specified IP address and port. If the connection is successful within the timeout, it's healthy. - HTTP/HTTPS: Opens a TCP connection, sends an HTTP request, and expects a 2xx or 3xx status code (configurable). Additional options include string matching in the response body. - Calculated checks: Combine multiple child health checks using logical AND/OR/NOT. - CloudWatch alarm checks: Use the state of a CloudWatch alarm (OK, ALARM, INSUFFICIENT_DATA) as the health status.

Each health checker independently reports the endpoint status. Route 53 aggregates these reports: if a majority (more than half) of the health checkers report the endpoint as healthy, the overall health check is considered healthy. This distributed approach prevents false positives from isolated network issues.

You create a health check via AWS Console, CLI, or SDK. Example using AWS CLI:

aws route53 create-health-check --caller-reference unique-string --health-check-config '{
    "IPAddress": "203.0.113.10",
    "Port": 443,
    "Type": "HTTPS",
    "ResourcePath": "/health",
    "FullyQualifiedDomainName": "example.com",
    "RequestInterval": 30,
    "FailureThreshold": 3
}'

To associate a health check with a DNS record, you use a routing policy that supports health checks: failover, weighted, latency, geolocation, geoproximity, or multivalue answer. For example, for failover routing:

aws route53 change-resource-record-sets --hosted-zone-id ZONEID --change-batch '{
    "Changes": [{
        "Action": "CREATE",
        "ResourceRecordSet": {
            "Name": "www.example.com.",
            "Type": "A",
            "SetIdentifier": "primary",
            "Failover": "PRIMARY",
            "HealthCheckId": "health-check-id",
            "TTL": 60,
            "ResourceRecords": [{"Value": "203.0.113.10"}]
        }
    }]
}'

To monitor health check status, use:

aws route53 get-health-check-status --health-check-id id

When a health check fails, Route 53 automatically updates the DNS response for any record that references that health check. For failover routing, if the primary record's health check is unhealthy, Route 53 returns the secondary record's IP. If both are unhealthy, Route 53 returns the primary record (by default). This is because Route 53 still returns the primary if no healthy record exists; you can configure a 'last resort' behavior.

Failover can be active-passive (primary/standby) or active-active (using weighted routing with health checks). For active-passive, you create two records with the same name, one PRIMARY and one SECONDARY. The primary is served until its health check fails, then the secondary is served. When the primary recovers, traffic shifts back (unless you set Invert Health Check).

Calculated health checks allow you to combine multiple child health checks using a logical operator. For example, you can create a health check that is healthy only if at least 2 out of 3 regional endpoints are healthy. This is useful for multi-region deployments where you want failover only if multiple regions fail.

By default, Route 53 health checkers run from public IP addresses and cannot reach endpoints in private subnets (VPC). To health check private resources, you must use a CloudWatch alarm that monitors a custom metric (e.g., from a Lambda function) and then associate that alarm with the health check (CloudWatch alarm health check type). Alternatively, you can use a proxy endpoint in a public subnet that forwards to the private resource.

Use CloudWatch metrics: HealthCheckStatus (1=healthy, 0=unhealthy), HealthCheckPercentageHealthy (percentage of health checkers reporting healthy). Common failure reasons: endpoint not responding, incorrect port, timeout, string mismatch. Check health check status via CLI: aws route53 get-health-check-status --health-check-id id returns last status and failure reason.

A global e-commerce company runs a web application in us-east-1 (primary) and eu-west-1 (secondary). They use Route 53 failover routing with health checks pointing to an Application Load Balancer (ALB) in each region. The health checks are HTTP checks on the ALB's /health endpoint. The primary ALB is configured with a 30-second interval and 3 failure threshold. When us-east-1 experiences an outage, Route 53 detects three consecutive failures after 90 seconds and automatically returns the eu-west-1 ALB IP. The TTL is set to 60 seconds, so most clients fail over within 2-3 minutes. A common misconfiguration is forgetting to set the health check path to a lightweight endpoint that reflects application health, not just the load balancer's health. In production, they also use a CloudWatch alarm health check to monitor a custom metric that checks database connectivity, ensuring the health check is application-aware.

A SaaS provider uses weighted routing with health checks to gradually shift traffic from blue to green deployments. They create two A records with weights 100 and 0, each associated with a health check. The green deployment's health check is configured to fail until the deployment is verified. Once verified, they update the green health check to be healthy, and adjust weights to 50/50. If the green deployment shows errors, the health check fails, and Route 53 automatically stops sending traffic to it. This prevents manual rollback. A common pitfall is not setting the health check interval low enough (10 seconds) for rapid detection; with 30-second interval and 3 failures, it takes 90 seconds to detect a problem, which may be too slow for critical deployments.

A financial services company hosts a critical API behind an internal Network Load Balancer (NLB) in a private subnet. Since Route 53 health checkers are public, they cannot directly probe the NLB. They deploy a Lambda function inside the VPC that periodically calls the API and publishes a custom metric to CloudWatch (e.g., api_health with value 1 for healthy, 0 for unhealthy). They create a CloudWatch alarm that triggers when the metric is below 1 for 1 minute. Then they create a Route 53 health check of type 'CloudWatch Alarm' and associate it with the alarm. This health check is used in a failover record. A common issue is the Lambda function's execution role not having permissions to publish metrics, causing the alarm to never trigger. They also set the alarm's evaluation period to match the health check's failure threshold to avoid flapping.