Optimize PyTorch Inference on Vertex AI

A team wants to serve a large PyTorch model (3 GB) for online predictions with low latency. Which THREE actions should they take?

Quick Answer

The answer is to use a GPU accelerator, optimize the model with TorchScript or quantization, and deploy with a custom container that preloads the model. These three actions directly address the core challenge of serving a large PyTorch model for low-latency online predictions on Vertex AI: the GPU provides the raw compute speed, model optimization reduces the computational footprint and inference time, and preloading eliminates cold start delays by keeping the model in memory. On the Google Professional Machine Learning Engineer exam, this question tests your ability to distinguish between actions that reduce prediction latency versus those that improve throughput or network latency—a common trap is confusing multiregion deployment (which reduces network round-trips) with actual inference speed. Remember the memory tip: "GPU, shrink, preload" to recall the three pillars of low-latency PyTorch inference on Vertex AI.

⚠ Common exam trap

Candidates may mistakenly think multi-region deployment or batch prediction reduce online prediction latency. Multi-region reduces network round-trip time but not inference time; batch prediction is not suitable for real-time serving.

Answer choices

Why each option matters

Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.

Correct answer & explanation

✓

Use a custom container that preloads the model into memory.

For online low-latency predictions with a large model, preloading the model into memory via a custom container (A) eliminates cold-start latency. GPU acceleration (C) significantly speeds up inference for large models. Optimizing with TorchScript or quantization (D) reduces model size and inference time. Batch prediction (B) is for offline batch processing, not online. Multi-region deployment (E) improves availability and global latency but does not directly reduce prediction latency for a single request.

Answer analysis

Option-by-option breakdown

For each option: why learners choose it and why it is or isn't the right answer here.

✓
Use a custom container that preloads the model into memory.
Why this is correct
Preloading avoids loading model on each request, reducing latency.
✗
Use batch prediction instead of online prediction.
Why it's wrong here
Batch prediction is asynchronous, not real-time.
✓
Use a machine type with a GPU accelerator.
Why this is correct
GPU accelerates deep learning inference.
✓
Optimize the model using TorchScript or quantization.
Why this is correct
TorchScript and quantization reduce inference time and model size.
✗
Deploy in multiple regions with Cloud Load Balancing.
Why it's wrong here
Reduces network latency but not prediction latency.

About these practice questions

Courseiva creates original exam-style practice questions with explanations and wrong-answer analysis. It does not publish real exam questions, exam dumps, or protected exam content. Learn why practice questions differ from exam dumps →

How Courseiva writes practice questions · Editorial policy

Same concept, more angles

1 more way this is tested on PMLE

These questions test the same concept from different angles. Work through them to make sure you can recognise it however the exam phrases it.

Variation 1. A team needs to serve a PyTorch model for production inference with strict latency requirements (p99 < 100ms). The model has dynamic control flow and uses custom kernels compiled with torch.jit. Which serving approach should they recommend?

medium

✓ A.Build a custom container with PyTorch JIT and deploy it on Vertex AI Prediction.
B.Convert the model to TensorFlow SavedModel and serve it on Vertex AI Prediction with TensorFlow Serving.
C.Use Cloud Functions with a PyTorch wrapper to handle inference requests.
D.Deploy the model on Vertex AI Prediction using the prebuilt PyTorch container.

Why A: A custom container with PyTorch JIT allows full control over model execution, including dynamic control flow and custom kernels, and can be deployed on Vertex AI Prediction, which supports custom containers for low-latency inference. Option B is wrong because converting to TensorFlow SavedModel would lose PyTorch-specific features like custom JIT kernels. Option C is wrong because Cloud Functions have cold start latency and are not suited for low-latency production inference at scale. Option D is wrong because the prebuilt PyTorch container may not support custom JIT kernels or dynamic control flow optimally.

Last reviewed: Jun 24, 2026

Question Discussion

Share a tip, memory trick, or ask about the reasoning behind this question. Do not post real exam questions, leaked content, braindumps, or copyrighted exam material. Comments are moderated and may be removed without notice.

Loading comments…

This PMLE practice question is part of Courseiva's free Google Cloud certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the PMLE exam.