You are serving a model on a Vertex AI endpoint that requires a GPU. The model is used for interactive predictions with a strict latency SLO. You notice that during peak hours, some requests time out because the endpoint's autoscaler is slow to add GPU replicas. Which action should you take to meet the SLO?
Increasing the minimum replica count ensures that more GPU replicas are already running and ready to serve traffic before the peak arrives. Since GPU replicas take longer to start, pre-warming them avoids the delay caused by slow autoscaling. This directly addresses the timeout issue by providing sufficient capacity during peak hours without waiting for the autoscaler to react, helping to meet the strict latency SLO.
Why this answer
GPU replicas take longer to provision and initialize than CPU replicas, so autoscaling may not react quickly enough during sudden peaks. Setting a higher minimum replica count keeps additional GPU replicas warm and ready before demand spikes. This pre-provisioned capacity absorbs peak traffic immediately, preventing request queuing and timeouts.
It addresses the slow autoscaler response directly, ensuring the endpoint can meet its strict latency SLO during peak hours.
Exam trap
The trap here is assuming that batching or reducing input size solves timeout issues, when the real cause is insufficient warm GPU capacity during peak demand.