A company has a production VPC with a public and private subnet across two Availability Zones. The public subnet hosts a Network Load Balancer (NLB) that distributes traffic to EC2 instances in the private subnet. The application experiences periodic failures where the NLB marks all targets as unhealthy for about 2 minutes, then they recover. The health checks are HTTP on port 80 with a 5-second interval, 2 consecutive successes to be healthy, and 2 consecutive failures to be unhealthy. The target group health check timeout is 5 seconds. The EC2 instances are behind an Auto Scaling group with a minimum of 2 instances per AZ. CPU utilization on the instances is stable at 40%. The NLB's CloudWatch metrics show HealthyHostCount drops to zero suddenly. The network engineer suspects a network issue. What is the most likely cause?
If timeout equals interval, any delay in response results in consecutive failures, marking the instance unhealthy. Increasing the interval or decreasing the timeout would help.
Why this answer
The health check timeout equals the interval (both 5 seconds). Under transient network delays, a health check response can be delayed beyond the timeout, causing the NLB to count a failure. With 2 consecutive failures required to mark unhealthy, a brief period of latency can cause all targets to be marked unhealthy for about 2 minutes (2 intervals × 5 seconds = 10 seconds of failures, but the recovery requires 2 consecutive successes, leading to the observed ~2-minute duration due to repeated timeouts).
The instances are not overloaded (CPU 40%), and security groups and route tables are configured correctly since normal operation resumes. This configuration violates the best practice of setting the timeout lower than the interval to allow for retries.