hardMultiple ChoiceObjective-mapped
Reducing Tail Latency for Online Predictions
A team has successfully trained a deep learning model on Vertex AI using a custom container and distributed training with TensorFlow. They want to serve this model for online predictions with low latency. They deploy the model to Vertex AI Endpoint with a single n1-standard-4 machine. During load testing, they observe that the median latency is 200ms, but the 99th percentile latency spikes to 2 seconds. The model is a complex neural network that takes variable-length text as input. Which approach will best reduce tail latency while maintaining throughput?
Quick Answer
The answer is implementing request batching to process multiple inputs per request. Batching reduces tail latency by amortizing the fixed overhead of model inference across several inputs, which smooths out the variability caused by variable-length text inputs that can create unpredictable compute spikes. On the Google Professional Machine Learning Engineer exam, this scenario tests your understanding that tail latency spikes often stem from per-request overhead rather than raw compute power, and a common trap is reaching for more resources like GPUs or memory when the real fix is architectural. Remember that for online predictions on Vertex AI, batching converts many small, erratic requests into fewer, more predictable batches, directly stabilizing the 99th percentile. A useful memory tip: think of batching as a “traffic jam smoother”—it merges individual cars into a steady bus route, eliminating the stop-and-go that causes the slowest riders to arrive late.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
Implement request batching to process multiple inputs per request.
Request batching (Option B) is the best approach to reduce tail latency for variable-length text inputs because it amortizes the fixed overhead of processing multiple predictions together, reducing per-request latency variability. Option A (autoscaling) helps handle increased traffic but does not reduce per-request latency spikes. Option C (GPU) can improve throughput but may not reduce tail latency caused by variability in input lengths. Option D (more memory) addresses memory constraints but not compute-bound variability.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✗
Use autoscaling with a target CPU utilization of 70%.
Why it's wrong here
Autoscaling adds capacity over time, not reducing per-request latency spikes.
- ✓
Implement request batching to process multiple inputs per request.
Why this is correct
Batching reduces overhead and smooths out latency for variable-length inputs.
- ✗
Use a GPU machine type like n1-standard-4 with an attached GPU.
Why it's wrong here
GPU may speed up inference but not specifically address tail latency from variable-length inputs.
- ✗
Increase the machine type to n1-highmem-8 to allocate more memory.
Why it's wrong here
Memory is not the bottleneck; it's compute variability.
Go deeper
Related to this question
About these practice questions
One of 990 original PMLE practice questions on Courseiva, each with a full explanation and wrong-answer analysis — not exam dumps or protected exam content. Learn why practice questions differ from exam dumps →
Same concept, more angles
1 more way this is tested on PMLE
These questions test the same concept from different angles. Work through them to make sure you can recognise it however the exam phrases it.
Variation 1. A data science team has trained a custom model using Vertex AI and wants to deploy it for online predictions with low latency. Which TWO actions should they take to optimize performance?
medium- A.Use Vertex AI Endpoints with traffic splitting for canary deployments.
- B.Enable autoscaling with a large min replicas count to handle bursts.
- ✓ C.Optimize the model by quantizing to FP16.
- ✓ D.Use a custom prediction routine with pre-processing inside the container.
- E.Use a machine type with GPU for inference.
Why C: Quantizing the model to FP16 reduces its memory footprint and computational requirements, directly lowering inference latency on compatible hardware (e.g., NVIDIA GPUs with Tensor Cores). This optimization is especially effective for online predictions where response time is critical, as it accelerates matrix operations without significantly sacrificing model accuracy.
JA
Written by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
This PMLE practice question is part of Courseiva's free Google Cloud certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the PMLE exam.