hardMultiple SelectObjective-mapped
Reducing Tail Latency for LLMs on Vertex AI: Key Strategies
A team is serving a large language model (LLM) on Vertex AI using a custom container. They want to reduce tail latency. Which THREE strategies should they consider?
Quick Answer
The answer is to quantize the model to INT8, implement dynamic batching, and optimize the custom container’s inference pipeline. Quantizing to INT8 reduces the computational load per token, directly lowering the processing time for each request and smoothing out latency spikes. Dynamic batching groups multiple inference requests into a single forward pass, maximizing GPU utilization and amortizing overhead, which is especially effective for LLMs because it reduces the per-request variance that creates tail latency. On the Google Professional Machine Learning Engineer exam, this question tests your understanding of latency optimization trade-offs: the common trap is to focus only on hardware scaling (like adding more GPUs) rather than algorithmic and batching strategies. A key memory tip is “Quantize, Batch, Optimize” — remember that reducing precision, grouping requests, and tuning the serving stack are the three pillars for cutting tail latency on Vertex AI.
⚠ Common exam trap
Watch out — candidates often confuse scaling strategies (like increasing replicas or upgrading hardware) with latency-optimization techniques, failing to recognize that tail latency is primarily reduced by batching and caching, not by adding more compute resources.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
Use dynamic batching to combine requests.
Dynamic batching (B) reduces tail latency by grouping multiple inference requests into a single batch, which improves GPU utilization and amortizes overhead across requests. This is particularly effective for LLMs because it allows the model to process more tokens per forward pass, reducing the per-request latency variance that contributes to tail latency.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✗
Increase the number of replicas.
Why it's wrong here
Helps with throughput, not tail latency.
- ✓
Use dynamic batching to combine requests.
Why this is correct
Improves GPU utilization and reduces per-request latency.
- ✓
Implement response caching for common queries.
Why this is correct
Avoids inference for repeated requests.
- ✓
Quantize the model to INT8 to reduce computation.
Why this is correct
Reduces latency per request.
- ✗
Upgrade to a more powerful GPU type.
Why it's wrong here
May reduce overall latency but not specifically tail.
Go deeper
Related to this question
About these practice questions
This PMLE question is part of Courseiva's 990-question bank — original exam-style content with full explanations and wrong-answer analysis, never real exam questions or exam dumps. Learn why practice questions differ from exam dumps →
Same concept, more angles
1 more way this is tested on PMLE
These questions test the same concept from different angles. Work through them to make sure you can recognise it however the exam phrases it.
Variation 1. Your team has deployed a PyTorch model using a custom container on Vertex AI Prediction. The model uses dynamic batching to combine incoming requests. You notice that the average latency is 150 ms, but the 99th percentile latency is 2 seconds. Cloud Monitoring shows that the CPU is idle much of the time, and GPU utilization is around 70%. The model is deployed on a single n1-standard-4 with a T4 GPU. You suspect the issue is related to request queuing. Which change would most effectively reduce tail latency?
hard- ✓ A.Add a second replica to share the load.
- B.Increase the batch timeout to allow larger batches to form, reducing the number of batches.
- C.Decrease the batch size to reduce processing time per batch.
- D.Implement a priority queue to handle high-priority requests first.
Why A: Adding a second replica reduces the queue length per replica, thereby decreasing the waiting time for requests. This directly addresses the request queuing issue causing high tail latency. Option A (increase batch timeout) could actually worsen tail latency by allowing more requests to accumulate. Option B (decrease batch size) may reduce per-batch processing time but does not address queuing. Option D (priority queue) adds complexity without solving the root cause of insufficient capacity.
JA
Written by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
This PMLE practice question is part of Courseiva's free Google Cloud certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the PMLE exam.