mediumMultiple Select
Optimize PyTorch Inference on Vertex AI
A team wants to serve a large PyTorch model (3 GB) for online predictions with low latency. Which THREE actions should they take?
Quick Answer
The answer is to use a GPU accelerator, optimize the model with TorchScript or quantization, and deploy with a custom container that preloads the model. These three actions directly address the core challenge of serving a large PyTorch model for low-latency online predictions on Vertex AI: the GPU provides the raw compute speed, model optimization reduces the computational footprint and inference time, and preloading eliminates cold start delays by keeping the model in memory. On the Google Professional Machine Learning Engineer exam, this question tests your ability to distinguish between actions that reduce prediction latency versus those that improve throughput or network latency—a common trap is confusing multiregion deployment (which reduces network round-trips) with actual inference speed. Remember the memory tip: "GPU, shrink, preload" to recall the three pillars of low-latency PyTorch inference on Vertex AI.
⚠ Common exam trap
PMLE often tests whether candidates pick availability/scaling options (multi-region) when the real issue is model size and inference speed; the trap is confusing latency reduction with redundancy.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
Use a custom container that preloads the model into memory.
Option A is correct because a custom container that preloads the 3 GB PyTorch model into memory eliminates the per-request model-loading overhead, which is essential for low-latency online predictions. Option C is correct because attaching a GPU accelerator provides the parallel compute throughput PyTorch inference needs, drastically reducing latency for a large model compared to CPU-only serving. Option D is correct because TorchScript (tracing/scripting for optimized execution) and quantization (e.g., FP16/INT8) reduce model size and inference cost, directly lowering latency while preserving online serving. Option B is wrong because batch prediction is asynchronous and high-throughput, not low-latency online serving, so it contradicts the stated requirement. Option E is wrong because multi-region deployment with Cloud Load Balancing improves global availability and proximity, but does not address the model's size or per-inference latency for online predictions.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✓
Use a custom container that preloads the model into memory.
Why this is correct
Preloading the 3 GB model into memory inside a custom container eliminates per-request model loading, which otherwise dominates latency for a model this size. This directly addresses the low-latency constraint by removing cold-start and disk-read overhead from the prediction path.
- ✗
Use batch prediction instead of online prediction.
Why it's wrong here
Batch prediction processes stored data in bulk and returns results asynchronously, so it cannot deliver low-latency responses to individual online requests. It is tempting because batching improves throughput and cost for large volumes; it would be correct for offline scoring jobs, not real-time serving.
- ✓
Use a machine type with a GPU accelerator.
Why this is correct
A GPU accelerator provides the parallel compute throughput PyTorch inference needs, cutting per-request processing time substantially compared with CPU-only serving. This satisfies the low-latency requirement for a 3 GB model whose forward passes would otherwise be too slow.
- ✓
Optimize the model using TorchScript or quantization.
Why this is correct
TorchScript compilation and quantization reduce model size and inference cost, lowering both memory footprint and per-request latency. Applied to a 3 GB PyTorch model, these optimisations directly serve the low-latency online prediction constraint rather than merely simplifying deployment.
- ✗
Deploy in multiple regions with Cloud Load Balancing.
Why it's wrong here
Multi-region deployment with Cloud Load Balancing reduces geographic latency but does not address serving a 3 GB PyTorch model with low latency; the model still needs optimised serving infrastructure. It is tempting because regional distribution improves availability and proximity; it would be correct for global user bases, not model-size constraints.
Go deeper
Related to this question
About these practice questions
One of 775 original PMLE practice questions on Courseiva, each with a full explanation and wrong-answer analysis — not exam dumps or protected exam content. Learn why practice questions differ from exam dumps →
Same concept, more angles
1 more way this is tested on PMLE
These questions test the same concept from different angles. Work through them to make sure you can recognise it however the exam phrases it.
Variation 1. A team needs to serve a PyTorch model for production inference with strict latency requirements (p99 < 100ms). The model has dynamic control flow and uses custom kernels compiled with torch.jit. Which serving approach should they recommend?
medium- ✓ A.Build a custom container with PyTorch JIT and deploy it on Vertex AI Prediction.
- B.Convert the model to TensorFlow SavedModel and serve it on Vertex AI Prediction with TensorFlow Serving.
- C.Use Cloud Functions with a PyTorch wrapper to handle inference requests.
- D.Deploy the model on Vertex AI Prediction using the prebuilt PyTorch container.
Why A: A custom container with PyTorch JIT allows full control over model execution, including dynamic control flow and custom kernels, and can be deployed on Vertex AI Prediction, which supports custom containers for low-latency inference. Option B is wrong because converting to TensorFlow SavedModel would lose PyTorch-specific features like custom JIT kernels. Option C is wrong because Cloud Functions have cold start latency and are not suited for low-latency production inference at scale. Option D is wrong because the prebuilt PyTorch container may not support custom JIT kernels or dynamic control flow optimally.
JA
Written and reviewed by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
Last reviewed September 2026 · checked against the official Google Cloud exam blueprint
This PMLE practice question is part of Courseiva's free Google Cloud certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the PMLE exam.