PMLE Serving and Scaling Models Practice Question

You are serving a model on a Vertex AI endpoint that requires a GPU. The model is used for interactive predictions with a strict latency SLO. You notice that during peak hours, some requests time out because the endpoint's autoscaler is slow to add GPU replicas. Which action should you take to meet the SLO?

⚠ Common exam trap

The trap here is assuming that batching or reducing input size solves timeout issues, when the real cause is insufficient warm GPU capacity during peak demand.

Answer choices

Why each option matters

Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.

Correct answer & explanation

✓

Set a higher minimum replica count on the endpoint to keep more GPU replicas warm during peak hours.

GPU replicas take longer to provision and initialize than CPU replicas, so autoscaling may not react quickly enough during sudden peaks. Setting a higher minimum replica count keeps additional GPU replicas warm and ready before demand spikes. This pre-provisioned capacity absorbs peak traffic immediately, preventing request queuing and timeouts. It addresses the slow autoscaler response directly, ensuring the endpoint can meet its strict latency SLO during peak hours.

Answer analysis

Option-by-option breakdown

For each option: why learners choose it and why it is or isn't the right answer here.

✗
Enable request batching on the endpoint to process multiple requests per replica.
Why it's wrong here
Batching can improve throughput but may increase latency for individual requests because they wait to be grouped. With a strict latency SLO, batching could worsen timeouts. It also does not solve the root cause, which is insufficient GPU replicas during peak hours. Without more warm replicas, batching alone cannot prevent request timeouts when demand exceeds capacity.
✓
Set a higher minimum replica count on the endpoint to keep more GPU replicas warm during peak hours.
Why this is correct
Increasing the minimum replica count ensures that more GPU replicas are already running and ready to serve traffic before the peak arrives. Since GPU replicas take longer to start, pre-warming them avoids the delay caused by slow autoscaling. This directly addresses the timeout issue by providing sufficient capacity during peak hours without waiting for the autoscaler to react, helping to meet the strict latency SLO.
✗
Reduce the model's input size by truncating features to lower per-request compute.
Why it's wrong here
Truncating features may reduce compute per request but can degrade model accuracy and does not address the autoscaler's slow response to GPU demand. The timeout is caused by insufficient replicas during peak hours, not by excessive input size. Even with smaller inputs, if the autoscaler cannot add GPU replicas fast enough, requests will still queue and time out.
✗
Switch to a CPU-based machine type to avoid GPU provisioning delays.
Why it's wrong here
The model requires a GPU, so switching to CPU would either fail to deploy or cause much slower inference that violates the latency SLO. GPU provisioning delays are a scaling characteristic, not a reason to abandon GPU. This option does not solve the timeout problem and likely makes latency worse, as CPU inference for a GPU-optimized model is significantly slower.

Go deeper

PMLE Serving and Scaling Models Practice Question

Why each option matters

Option-by-option breakdown

Related to this question