PMLE Serving and Scaling Models — 50 Questions

Question 1

You are deploying a new version of a model to a Vertex AI endpoint that already has a champion model serving 100% of traffic. You want to gradually shift traffic to the new version while monitoring for errors. Which approach should you use?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Deploy the challenger to the same endpoint with initial traffic split, e.g., champion 90%, challenger 10%, and gradually adjust.. Vertex AI endpoints support traffic splitting between model versions deployed to the same endpoint. By deploying the challenger to the same endpoint and setting an initial split (e.g., champion 90%, challenger 10%), you can gradually shift traffic while monitoring for errors. This approach uses the endpoint's built-in traffic management, avoiding the complexity and latency of external load balancers.

Answer

Use Cloud Load Balancing with weighted backend services pointing to different endpoints.

Answer

Delete the champion model and redeploy with the challenger as the new version.

Answer

Create a new endpoint for the challenger and use a load balancer to split traffic.

Question 2

A company is using Vertex AI Prediction with a custom container that performs preprocessing before inference. The preprocessing step is CPU-intensive and the inference step uses a GPU. They want to minimize prediction latency while optimizing cost. Which architecture should they use?

Accepted Answer

Use a single GPU machine (e.g., n1-standard-4 with T4) and perform both preprocessing and inference on the same instance.. Running preprocessing and inference on the same GPU-backed instance (e.g., n1-standard-4 with T4) eliminates the network hop, serialization, and queueing overhead that would otherwise be introduced by splitting the pipeline across services. Because the preprocessing is CPU-intensive and the inference is GPU-bound, co-locating them lets the CPU work overlap with GPU execution on the same VM, minimizing end-to-end latency. It also avoids paying for two separate always-on resources, which is the cost-optimal choice for a tightly coupled preprocess-then-infer pipeline.

Answer

Use Cloud Run for preprocessing and send HTTP requests to a GPU-backed Vertex AI endpoint for inference.

Answer

Use two separate Vertex AI endpoints: one CPU-based for preprocessing, one GPU-based for inference, and chain them with Cloud Tasks.

Answer

Use Dataflow for preprocessing and then invoke the model, but Dataflow is not designed for real-time prediction.

Question 3

You have a Vertex AI endpoint with min_replica_count=2 and max_replica_count=10. You notice that during a traffic spike, the endpoint does not scale up quickly enough, causing increased latency. What should you do to improve autoscaling responsiveness?

Accepted Answer

Reduce the target CPU utilization percentage from default to a lower value.. Reducing the target CPU utilization percentage (e.g., from the default 60% to a lower value like 40%) causes the autoscaler to trigger scale-up actions sooner, as the threshold for adding replicas is reached at a lower CPU load. This improves responsiveness during traffic spikes by initiating scaling earlier, reducing latency. The endpoint's min_replica_count=2 and max_replica_count=10 remain unchanged, so the scaling range is preserved.

Answer

Increase max_replica_count to 20.

Answer

Disable autoscaling and manually manage replicas.

Answer

Increase min_replica_count to 10.

Question 4

An ML team wants to deploy multiple models (e.g., a recommender and a classifier) behind a single Vertex AI endpoint. The models have different resource requirements: the recommender needs GPU, the classifier needs high memory. How should they configure the endpoint?

Accepted Answer

Deploy both models to the same endpoint with different machine types per deployed model.. Vertex AI endpoints support deploying multiple models to the same endpoint, and each deployed model can have its own dedicated resources (machine type, accelerators, min/max replicas). This allows the recommender to use a GPU-backed machine while the classifier uses a high-memory CPU machine, all behind a single endpoint URL. Traffic is split or routed to the appropriate model based on the deployed model ID or traffic split configuration. This is the intended design for heterogeneous model serving.

Answer

Use Cloud Run for one model and Vertex AI for the other.

Answer

Use a single machine type that meets the highest requirements.

Answer

Create separate endpoints for each model.

Question 5

You need to run a batch prediction job on Vertex AI using a model that requires custom preprocessing using a Python script. The preprocessing must be applied before inference. Which approach should you use?

Accepted Answer

Use Dataflow to preprocess the data and write the results to BigQuery or GCS, then launch a batch prediction job on the preprocessed data.. Dataflow (Apache Beam) is the recommended serverless service for distributed, scalable preprocessing of large datasets on Google Cloud. It can read raw data from GCS, apply custom Python preprocessing logic, and write the preprocessed results to GCS or BigQuery. The batch prediction job then reads the preprocessed data directly, avoiding the need to embed preprocessing in the prediction container or handle data on a single VM.

Answer

Use Cloud Functions to preprocess each record individually.

Answer

Preprocess the data on a single VM and then upload to GCS.

Answer

Include the preprocessing logic in the custom container used for batch prediction.

Question 6

An organization wants to deploy a model on edge devices (e.g., Android phones) for offline inference. They trained a model using TensorFlow. Which THREE steps should they take to prepare and deploy the model?

Accepted Answer

Convert the model to TensorFlow Lite format.. Option A is correct because TensorFlow Lite is Google's lightweight runtime format designed for on-device/offline inference on mobile and edge hardware like Android phones, so the trained TensorFlow model must be converted (e.g., via the TFLite Converter) to a .tflite file. Option C is correct because Vertex AI Edge Manager (part of the Vertex AI/Edge AI suite) packages the model and manages its distribution and deployment to edge devices, providing the tooling to push and monitor models at the edge. Option E is correct because after conversion the resulting TFLite model is what actually gets deployed and executed on the edge devices for offline inference, completing the workflow. Option B is not appropriate because deploying to a Vertex AI endpoint provides cloud-hosted online inference, which contradicts the offline, on-device requirement. Option D is not appropriate because ONNX is a different interchange format and is not the target runtime for TensorFlow models on Android edge devices in this scenario.

Answer

Deploy the model to a Vertex AI endpoint for online inference.

Answer

Export the model to ONNX format.

Question 7

You want to deploy a TensorFlow model to a Vertex AI endpoint and enable online predictions. The model requires GPU for inference. Which machine type should you select when deploying the model?

Accepted Answer

a2-highgpu-1g (with A100 GPU). The a2-highgpu-1g machine type is specifically designed for GPU-accelerated workloads on Vertex AI, featuring an NVIDIA A100 GPU that meets the inference requirements of a TensorFlow model. Vertex AI online prediction endpoints require a machine type that supports GPU attachment, and the A2 series is the only option among the choices that provides a dedicated GPU for inference.

Answer

n1-standard-4

Answer

e2-standard-4

Answer

n1-highmem-8

Question 8

Your Vertex AI endpoint is experiencing high latency during traffic spikes. You have set maxReplicas=10 and minReplicas=2. The CPU utilisation target is 60%. During spikes, the endpoint never scales beyond 4 replicas. What is the most likely reason?

Accepted Answer

The maxReplicas limit is set to 10, but the cooldown period is preventing rapid scaling.. Vertex AI online prediction endpoints use a cooldown period (default 5 minutes) after each scaling action to prevent rapid, thrashing scale-ups. During a traffic spike, the autoscaler may add a few replicas (e.g., up to 4) and then enter a cooldown, refusing to add more even if CPU utilization remains above the 60% target. Since maxReplicas=10 is not reached, the cooldown is the bottleneck. This is a known behavior in Vertex AI's autoscaling algorithm.

Answer

You need to enable GPU acceleration.

Answer

The endpoint is using a legacy model framework.

Answer

The machine type is too small.

Question 9

Your team has built a low-latency similarity search service using Vertex AI Matching Engine (Vector Search). The index is updated daily with new embeddings. You need to serve the latest index without downtime. What is the correct deployment strategy?

Accepted Answer

Create a new index version, deploy it to the same endpoint, and then update the endpoint to use the new index version.. Vertex AI Matching Engine supports deploying a new index version to the same endpoint without downtime. You create a new index version from the updated embeddings, deploy it to the existing endpoint, and then update the endpoint to use the new version. This allows traffic to seamlessly switch to the updated index once it is fully deployed, avoiding any service interruption.

Answer

Use streaming updates to continuously update the index.

Answer

Update the existing index in place by calling the index update API.

Answer

Delete the old index and create a new index each day, then deploy the new index to a new endpoint and update DNS.

Question 10

You need to serve a model on an edge device with low latency and offline capability. Which approach should you use?

Accepted Answer

Export the model to TensorFlow Lite and use Vertex AI Edge Manager for deployment.. TensorFlow Lite is specifically designed for on-device inference with low latency and offline capability, converting models into a lightweight format optimized for edge hardware. Vertex AI Edge Manager extends this by providing deployment, monitoring, and management of models on edge devices, ensuring they run efficiently without constant cloud connectivity.

Answer

Use Cloud Run for on-device inference.

Answer

Deploy the model to a Vertex AI endpoint and rely on mobile connectivity.

Answer

Use AI Platform Prediction (not Vertex AI).

Question 11

You have a Vertex AI endpoint with two deployed models: model A (champion) and model B (challenger). Traffic split is 90:10. You want to gradually increase model B's traffic to 50% over a week. What is the best way to update the traffic split?

Accepted Answer

Use the gcloud ai endpoints update command to change traffic split.. The `gcloud ai endpoints update` command allows you to directly modify the traffic split of an existing Vertex AI endpoint without redeploying or recreating the endpoint. This is the intended method for gradually shifting traffic between deployed models, as it supports incremental updates to the `--traffic-split` parameter, enabling a controlled rollout from 90:10 to 50:50 over a week.

Answer

Use gcloud ai models upload to overwrite model B with new settings.

Answer

Create a new endpoint and migrate traffic gradually using a load balancer.

Answer

Delete model B and redeploy with a new traffic split.

Question 12

You are deploying a model for real-time inference with strict latency requirements (<100ms P99). You want to autoscale based on custom metrics. Which TWO actions should you take? (Choose 2)

Accepted Answer

Configure the endpoint to use custom metrics from Cloud Monitoring.. Option B is correct because Vertex AI online prediction endpoints support autoscaling driven by custom metrics that you publish to Cloud Monitoring, which is exactly what the scenario requires for scaling on application-specific signals rather than only CPU/GPU utilization. Option C is correct because an autoscaling policy must define a target value for the chosen metric (for example, requests per replica or a custom latency/throughput metric) so the autoscaler knows when to add or remove replicas. Together, B and C form the required pair: select the custom metric as the scaling signal and set its target threshold. Option A is not required because a regional endpoint does not by itself guarantee sub-100ms P99 and is not the mechanism for custom-metric autoscaling. Option D is not necessary because GPU acceleration may improve inference speed but is unrelated to configuring custom-metric autoscaling. Option E is wrong because setting minReplicas to 0 can introduce cold-start latency that violates the strict <100ms P99 requirement.

Answer

Use a regional endpoint to reduce network latency.

Answer

Enable GPU acceleration for faster inference.

Answer

Set minReplicas to 0 to save cost.

Question 13

Your team is using Vertex AI Prediction for a large-scale NLP model (PyTorch, custom ops). The model currently runs on CPU but you want to optimise inference cost and performance. Which THREE approaches should you consider? (Choose 3)

Accepted Answer

Deploy the model with a GPU machine type and use TensorRT optimisation.. Option A is correct because deploying a PyTorch model on a GPU machine type in Vertex AI Prediction provides the hardware acceleration needed for large-scale NLP inference, and TensorRT optimisation (via torch-tensorrt or ONNX-to-TensorRT) fuses layers and uses FP16/INT8 precision to cut latency and cost. Option B is correct because Vertex AI Model Optimisation automatically applies quantization and compilation (e.g., TensorRT/OpenVINO-style optimisations) to reduce model size and inference cost without manual tuning, which directly addresses the goal of optimising cost and performance. Option C is correct because NVIDIA Triton Inference Server supports dynamic batching, concurrent model execution, and model ensembles, which are essential for maximising GPU utilisation and throughput on large NLP models with custom ops. Option D is not appropriate because TensorFlow Lite targets mobile/edge devices and does not support the custom PyTorch ops or the server-side GPU acceleration required here. Option E is not appropriate because batch prediction is for offline, asynchronous scoring and does not improve online inference performance or reduce per-request serving cost for a deployed endpoint.

Answer

Convert the model to TensorFlow Lite and deploy on Vertex AI endpoint.

Answer

Switch to batch prediction to reduce cost.

Question 14

You need to deploy a model for online predictions with low latency. You want to ensure that the endpoint can handle traffic bursts without cold start. Which TWO configurations should you set? (Choose 2)

Accepted Answer

Set minReplicas to 1.. Option B (Set minReplicas to 1) is correct because keeping at least one replica always running prevents cold starts — with minReplicas=0 the endpoint scales to zero and the first request after idle time incurs model loading latency, whereas minReplicas=1 keeps a warm instance ready to serve immediately. Option E (Use a machine type with sufficient memory for the model) is correct because the model must fit entirely in memory on each replica; if the machine type has insufficient RAM, the model cannot be loaded or will swap/thrash, causing high latency and failed predictions, so adequate memory is a prerequisite for low-latency online serving. Option A is not among the marked answers: maxReplicas only caps the upper bound of scaling and does not by itself prevent cold starts or guarantee burst handling without a warm baseline. Option C is not marked: a custom container is a packaging choice and does not address latency or cold-start behavior. Option D is not marked: autoscaling on CPU at 30% is a scaling policy detail, not a guarantee against cold starts, and may even cause unnecessary scaling churn.

Answer

Set maxReplicas to a high number to handle bursts.

Answer

Deploy the model as a custom container.

Answer

Enable autoscaling with a target CPU utilisation of 30%.

Question 15

A company deploys a model on Vertex AI Endpoints for real-time inference. They need to minimize latency for prediction requests that are identical to previous requests. Which approach should they use?

Accepted Answer

Implement a caching layer using Cloud Memorystore with request hashing.. Caching identical prediction requests using Cloud Memorystore with request hashing reduces latency by serving cached responses directly from an in-memory cache, avoiding redundant model inference. This approach is ideal for real-time inference where many requests are identical, as it bypasses the model endpoint entirely for cached requests, minimizing response time.

Answer

Use a regional load balancer with session affinity.

Answer

Use Cloud CDN to cache prediction responses.

Answer

Enable prediction caching on Vertex AI Endpoints.

Question 16

An ML engineer is optimizing a large model for deployment on Vertex AI with GPU acceleration. They want to reduce model size and improve inference latency without significant accuracy loss. Which tool should they use?

Accepted Answer

Use Vertex AI Model Optimization with TensorRT.. Vertex AI Model Optimization with TensorRT is specifically designed to reduce model size and improve inference latency on NVIDIA GPUs by applying techniques like quantization, pruning, and graph optimizations. TensorRT optimizes the model for the target GPU architecture, enabling faster inference with minimal accuracy loss, which directly addresses the engineer's goals.

Answer

Use gcloud CLI to prune the model.

Answer

Use Cloud TPU for faster inference.

Answer

Use TensorFlow.js converter to optimize the model for web.

Question 17

A company is deploying multiple models on a single Vertex AI endpoint to reduce costs. Each model has different traffic patterns. Which configuration should they use?

Accepted Answer

Use a single endpoint with multiple deployed models and traffic allocation.. Vertex AI endpoints support deploying multiple models behind a single endpoint with traffic splitting, allowing you to route different percentages of requests to each model based on their traffic patterns. This reduces infrastructure costs compared to separate endpoints, as the endpoint's underlying compute resources are shared. Traffic allocation can be adjusted dynamically to match changing model usage without redeploying.

Answer

Use Cloud Run to serve each model as a separate service.

Answer

Deploy each model to separate endpoints and use a load balancer.

Answer

Use Vertex AI Matching Engine to serve models.

Question 18

A company wants to use Vertex AI Vector Search for real-time product recommendations based on user embeddings. They need to update the index frequently with new product embeddings without significant downtime. Which TWO options should they consider? (Choose 2)

Accepted Answer

Maintain two separate indexes: one for building and one for serving, and swap them after building.. Option A is correct because Vertex AI Vector Search supports a blue/green-style pattern where you build a new index while the current index continues serving, then swap the deployed index to the new one, minimizing downtime during frequent updates. Option B is correct because streaming updates let you add or modify embeddings incrementally in near real time without rebuilding the whole index, which suits frequently changing product embeddings. Option C is not ideal because daily full rebuilds cause stale data and require index rebuild/swap cycles rather than continuous freshness. Option D is wrong because increasing replicas improves query throughput and availability, not index update frequency. Option E is wrong because GPU endpoints accelerate serving/query performance, not the indexing update mechanism.

Answer

Use batch updates to rebuild the entire index daily.

Answer

Increase the number of replicas on the deployed index.

Answer

Use a GPU-powered endpoint for faster indexing.

Question 19

You have a Vertex AI endpoint serving a model with min replicas=2 and max replicas=10. You notice that during low traffic hours, the endpoint still runs 2 replicas, incurring costs. You want to reduce costs to zero when there is no traffic. What should you do?

Accepted Answer

Change min replicas to 0 and max replicas to 10.. Setting min replicas to 0 allows Vertex AI to scale down to zero instances when there is no traffic, eliminating costs during idle periods. The endpoint will automatically scale up from 0 to handle incoming requests, while max replicas=10 ensures it can handle peak load. This is the standard approach for cost optimization in Vertex AI endpoints.

Answer

Use a custom metric to trigger scaling down to zero.

Answer

Delete the endpoint when not in use and recreate it on demand.

Answer

Set max replicas to 0.

Question 20

You have a Vertex AI endpoint with autoscaling enabled. You notice that during traffic spikes, the endpoint takes a long time to scale up, causing prediction errors. What is the most effective solution?

Accepted Answer

Increase the minimum number of replicas to maintain a larger buffer.. Increasing the minimum number of replicas ensures that a baseline capacity is always running, so when traffic spikes occur, the endpoint already has enough replicas to handle the initial surge without waiting for autoscaling to add more. Autoscaling in Vertex AI adds replicas based on metrics like CPU utilization, but there is inherent latency in detecting the spike, provisioning new replicas, and warming them up. By maintaining a larger buffer of minimum replicas, you reduce the reliance on rapid scale-up during spikes, thereby minimizing prediction errors. This is the most effective solution because it directly addresses the cold-start delay of autoscaling.

Answer

Use a larger machine type to handle more requests per replica.

Answer

Reduce the target CPU utilization to trigger scaling earlier.

Answer

Increase the maximum number of replicas.