During a load test, a Vertex AI endpoint serving a large language model experiences high latency and increased error rates. The endpoint is configured with autoscaling. What is the most likely cause?
GPU-bound models require GPU-based metrics for effective autoscaling.
Why this answer
When a model is GPU-bound, CPU utilization remains low even under heavy inference load, so autoscaling based on CPU metrics fails to trigger additional replicas. This leads to queued requests, increased latency, and eventual error rates as the existing GPU instances become saturated. Vertex AI endpoints default to CPU-based autoscaling unless explicitly configured with GPU metrics like 'gpu_utilization' or custom metrics.
Exam trap
The trap here is that candidates assume autoscaling always works generically, but Vertex AI's default CPU-based metric is irrelevant for GPU-accelerated inference, causing silent failures under load.
How to eliminate wrong answers
Option A is wrong because a network bottleneck would typically manifest as packet loss or timeouts across all requests, not specifically correlated with GPU-bound model behavior, and autoscaling would still trigger if CPU metrics were affected. Option B is wrong because model size exceeding machine type limits would cause deployment failures or out-of-memory errors, not gradual latency increases during load tests. Option C is wrong because a global load balancer distributes traffic across regions and reduces latency, not increases it; the issue is autoscaling misconfiguration, not load balancing.