PMLE Serving and scaling models — Timed 20 Questions

Question 1

A company deploys a TensorFlow model on Vertex AI Prediction with a single node. During peak hours, inference latency increases. What should they do first to reduce latency?

Accepted Answer

Enable autoscaling for the deployment. Enabling autoscaling for the deployment is the correct first step because it allows Vertex AI Prediction to dynamically adjust the number of replicas based on incoming traffic. During peak hours, autoscaling can add more nodes to distribute the inference load, directly reducing latency without requiring manual intervention or over-provisioning.

Answer

Increase the machine type of the node

Answer

Decrease the min replicas to 0

Answer

Enable automatic batching of requests

Question 2

A data science team deploys a PyTorch model using Vertex AI Prediction. The model requires GPU for inference, but they notice high costs and underutilized GPUs during off-peak hours. What is the most cost-effective solution?

Accepted Answer

Use a GPU instance with min replicas=0 and autoscaling. Option C is correct because setting min replicas to 0 allows Vertex AI Prediction to scale down to zero instances during off-peak hours, eliminating GPU costs when no requests are being served. Combined with autoscaling, the deployment will spin up GPU-backed instances on demand only when traffic arrives, directly addressing the underutilization issue while maintaining low latency for inference requests.

Answer

Move the model to Cloud Functions

Answer

Use a GPU instance with a fixed number of replicas

Answer

Switch to a CPU-only machine type

Question 3

A company serves a scikit-learn model on Vertex AI Prediction but receives a 400 error with 'Prediction failed: Model evaluation error'. What is the most likely cause?

Accepted Answer

The model uses a scikit-learn version not supported by Vertex AI. Vertex AI Prediction supports specific versions of scikit-learn for serving models. If the model was trained with a version that is not in the supported list (e.g., 0.19, 0.20, 0.22, 0.23, 0.24, 1.0, 1.1), the prediction endpoint will fail with a 'Model evaluation error' because the underlying runtime cannot load the serialized model (e.g., pickle or joblib file). This is the most likely cause of a 400 error when the input format is otherwise correct.

Answer

The input data format is incorrect

Answer

The model was trained with a different framework

Answer

The endpoint is overloaded and timing out

Question 4

A company wants to serve a large XGBoost model that exceeds the 2GB limit for Vertex AI Prediction. What should they do?

Accepted Answer

Use a custom container to serve the model. Vertex AI Prediction has a 2GB limit for the model artifact when using pre-built containers. A custom container bypasses this limit because you package the model and serving code into a Docker image, which can be arbitrarily large. This allows you to serve XGBoost models exceeding 2GB without size constraints imposed by the managed serving infrastructure.

Answer

Reduce model size by removing features

Answer

Compress the model using gzip and upload

Answer

Deploy the model on Cloud Run Functions

Question 5

A company deploys a model on Vertex AI Prediction with autoscaling enabled. They notice that during a traffic spike, new instances take several minutes to become available, causing high latency. What is the best solution?

Accepted Answer

Set a higher min replicas to maintain a baseline of warm instances. Option D is correct because setting a higher min replicas ensures that a baseline number of instances are always warm and ready to serve traffic. During a traffic spike, new instances still take time to provision (cold start), but the warm instances handle the initial surge without latency spikes. This directly addresses the observed high latency during spikes.

Answer

Disable autoscaling and use a fixed number of replicas

Answer

Increase the max replicas setting

Answer

Decrease the machine type to reduce provisioning time

Question 6

A company uses Vertex AI Prediction with a custom container for a TensorFlow model. They notice that after deploying a new model version, requests still go to the old version. What is the most likely cause?

Accepted Answer

Traffic is not split to the new model version. In Vertex AI Prediction, when you deploy a new model version to an existing endpoint, you must explicitly allocate traffic to it. By default, the new version receives 0% traffic, so all requests continue to be served by the old version. The correct fix is to update the endpoint's traffic split, for example via the console or the `gcloud ai endpoints update` command with the `--traffic-split` flag.

Answer

The custom container is not compatible with Vertex AI

Answer

The model is cached and needs cache invalidation

Answer

The new model version was not deployed to the same endpoint

Question 7

A company needs to serve a model with strict latency requirements (<100ms). They are using Vertex AI Prediction with CPU. During testing, latency is 150ms. What should they do?

Accepted Answer

Switch to a GPU machine type. The model's latency of 150ms exceeds the 100ms requirement. Switching to a GPU machine type (Option D) is correct because GPUs are optimized for parallel computation, significantly reducing inference latency for many ML models, especially deep learning models, compared to CPUs. Vertex AI Prediction supports GPU machine types, and this change directly addresses the latency bottleneck without altering the model or its serving configuration.

Answer

Enable batching to improve throughput

Answer

Use a smaller machine type with more replicas

Answer

Export the model to TensorFlow Lite

Question 8

A company is deploying a model for online predictions on Vertex AI. They want to minimize latency while also handling traffic spikes. Which TWO configurations should they choose?

Accepted Answer

Use GPU machine type. Option A is correct because GPU machine types on Vertex AI provide significantly faster inference for deep learning models, reducing latency per prediction. Option B is correct because enabling autoscaling with min replicas=1 ensures the model can handle traffic spikes by dynamically adding replicas while keeping at least one instance running to avoid cold starts.

Answer

Disable autoscaling and use manual scaling

Answer

Use CPU machine type with more memory

Answer

Set a fixed number of replicas equal to peak load

Question 9

A company trains a model using Vertex AI Training and then deploys it to Vertex AI Prediction. They notice that prediction requests fail with 'InvalidArgument: input tensor shape mismatch'. Which THREE are possible causes?

Accepted Answer

The input data types do not match the expected types (e.g., float vs int). Option C is correct because Vertex AI Prediction expects the input tensor data types to exactly match those used during model training. If the model was trained with float32 inputs but the prediction request sends int32 values, the serving infrastructure detects the mismatch and returns an 'InvalidArgument: input tensor shape mismatch' error, as TensorFlow Serving (which underlies Vertex AI Prediction) validates dtype consistency at the graph level.

Answer

The model was exported in a different format than supported

Answer

The batch size in the request is too large

Question 10

A company wants to reduce costs for serving a model on Vertex AI Prediction without sacrificing availability. Which THREE strategies should they consider?

Accepted Answer

Enable automatic batching to improve throughput per instance. Option C is correct because enabling automatic batching on Vertex AI Prediction allows the model server to group multiple inference requests into a single batch, which increases throughput per instance and reduces the total number of compute resources needed. This directly lowers serving costs without sacrificing availability, as the batching is handled transparently by the Vertex AI Prediction infrastructure.

Answer

Use larger machine types to reduce the number of replicas

Answer

Switch to HTTP/2 to reduce network overhead

Question 11

You are an ML engineer at a global e-commerce company. Your team has developed a deep learning model for product recommendation that runs on Vertex AI Prediction. The model is deployed on a single n1-highmem-2 instance (CPU only) with autoscaling enabled (min replicas=1, max replicas=10). During Black Friday, traffic spikes to 1000 requests per second (QPS), and you observe that latency increases from 50ms to over 5000ms, and many requests time out. You check the monitoring dashboard and see that CPU utilization is at 100% on the single instance, and autoscaling is not triggering quickly enough. The team has a budget for this service and wants to handle the spike without compromising latency. What should you do?

Accepted Answer

Switch to GPU instances (e.g., n1-standard-4 with T4) and set min replicas=2 with autoscaling up to 10. Option A is correct because switching to GPU instances (n1-standard-4 with T4) offloads compute-intensive recommendation model inference to GPUs, significantly reducing per-request latency. Setting min replicas=2 ensures that at least two instances are always warm, reducing cold-start delays and allowing autoscaling to handle traffic spikes more responsively. This combination addresses both the CPU bottleneck and the slow scaling trigger, keeping latency under 50ms even at 1000 QPS.

Answer

Increase min replicas to 5 to keep warm instances

Answer

Set min replicas=1 and max replicas=5 to control cost

Answer

Increase max replicas to 20 and keep CPU instances

Question 12

A data science team has trained a TensorFlow model and wants to serve it online with minimal latency. Which Vertex AI deployment option should they use to ensure the model can handle traffic spikes without manual scaling?

Accepted Answer

Deploy the model to a Vertex AI Endpoint with automatic scaling.. Vertex AI Endpoints with automatic scaling (option B) are designed for online serving with minimal latency and can automatically adjust the number of replicas based on traffic load, handling spikes without manual intervention. This is the correct choice for a TensorFlow model requiring real-time inference and elastic scaling.

Answer

Use Vertex AI Model Garden.

Answer

Use Vertex AI Batch Prediction for offline inference.

Answer

Deploy the model to a Compute Engine VM with a load balancer.

Question 13

A financial services company deploys a fraud detection model on Vertex AI. The model must make predictions in under 100ms. After deployment, latency spikes to 300ms during peak hours. The model is a large ensemble with 500MB size. Which action is most likely to reduce latency?

Accepted Answer

Optimize the model using TensorFlow Lite and convert to a smaller format.. The primary cause of latency is the large model size (500MB) combined with real-time inference constraints. Optimizing the model with TensorFlow Lite reduces the model size and computational overhead, directly decreasing inference time. This addresses the root cause—model complexity—rather than scaling infrastructure around an inefficient model.

Answer

Switch to batch prediction to process requests asynchronously.

Answer

Reduce the machine type to a smaller instance.

Answer

Increase the number of replicas on the endpoint.

Question 14

A company serves a PyTorch model using a custom container on Vertex AI Prediction. They notice that after a few hours, the endpoint returns 502 errors. The logs show 'Out of memory' errors. The container has a memory limit of 4GB, and the model loads a 3GB vocabulary file. What is the most likely cause and best fix?

Accepted Answer

Load the vocabulary file once at startup and reuse it.. The 502 errors and 'Out of memory' errors indicate that the container is running out of memory during inference. Since the model loads a 3GB vocabulary file, and the container has only 4GB of memory, loading this file repeatedly for each prediction request (e.g., inside the prediction handler) would quickly exhaust memory. The correct fix is to load the vocabulary file once at container startup and reuse it across all requests, which is a standard best practice for serving models with large static assets.

Answer

Increase the container memory to 8GB.

Answer

Increase the number of replicas to distribute load.

Answer

Switch to Vertex AI Batch Prediction.

Question 15

A team deploys a model on Vertex AI that uses a custom prediction routine (CPR) with a dependency on a native library. The container crashes with 'ImportError: libcudart.so.11.0: cannot open shared object file'. How should they resolve this?

Accepted Answer

Build a custom container image that includes the CUDA runtime library.. The error 'ImportError: libcudart.so.11.0: cannot open shared object file' indicates that the CUDA runtime library (version 11.0) is missing from the container environment. Since the custom prediction routine (CPR) depends on a native library that requires this CUDA runtime, the correct solution is to build a custom container image that includes the CUDA runtime library. This ensures the shared object is available at runtime, resolving the import error.

Answer

Submit the model for batch prediction to avoid the error.

Answer

Request a GPU machine type for the endpoint.

Answer

Use a Vertex AI pre-built container for PyTorch instead.

Question 16

A company is deploying a machine learning model for real-time inference on Vertex AI. Which TWO practices improve serving performance and reliability?

Accepted Answer

Enable autoscaling to handle traffic variations.. Option B is correct because Vertex AI's autoscaling dynamically adjusts the number of replicas based on incoming request traffic, ensuring low latency during spikes and cost savings during lulls. This is critical for real-time inference, where consistent response times are required and manual scaling would either over-provision or under-provision resources. Autoscaling uses metrics like CPU utilization or request count to scale up or down, directly improving serving performance and reliability.

Answer

Use batch prediction for all requests.

Answer

Use manual scaling with a fixed number of replicas.

Answer

Deploy all models on the same machine type for consistency.

Question 17

A team is serving a large language model (LLM) on Vertex AI using a custom container. They want to reduce tail latency. Which THREE strategies should they consider?

Accepted Answer

Use dynamic batching to combine requests.. Dynamic batching (B) reduces tail latency by grouping multiple inference requests into a single batch, which improves GPU utilization and amortizes overhead across requests. This is particularly effective for LLMs because it allows the model to process more tokens per forward pass, reducing the per-request latency variance that contributes to tail latency.

Answer

Increase the number of replicas.

Answer

Upgrade to a more powerful GPU type.

Question 18

A model deployed on Vertex AI Prediction repeatedly exits with code 137. What is the most likely cause?

Accepted Answer

The model is using more memory than allocated (4GB).. Exit code 137 indicates that the container was killed by the Linux kernel's Out-Of-Memory (OOM) killer. In Vertex AI Prediction, each model deployment has a fixed memory allocation (default 4GB for custom containers). When the model's inference process exceeds this limit, the OOM killer terminates the container, resulting in exit code 137. This is the most direct and common cause for this specific exit code in Vertex AI.

Answer

The model has a disk I/O bottleneck.

Answer

The model is using too much CPU.

Answer

The container image is incompatible with the machine type.

Question 19

You are a machine learning engineer at a retail company. You have deployed a product recommendation model on Vertex AI Prediction using a custom container. The model is a TensorFlow SavedModel that computes embeddings using a large lookup table. The endpoint is configured with 2 replicas on n1-standard-4 (4 vCPU, 15 GB memory) machines. After deployment, you notice that the endpoint's memory usage grows over time, eventually reaching 90% and causing requests to fail with 503 errors. The container logs show no errors, but the memory usage graph shows a steady increase. The model loads the embedding table (5 GB) at startup. You suspect a memory leak. Which course of action should you take first to diagnose and resolve the issue?

Accepted Answer

Profile the container's memory usage locally with memory_profiler to find the leak, then fix the code.. Option A is correct because the steady memory growth despite a fixed 5 GB embedding table indicates a memory leak in the custom container code, not a capacity issue. Profiling locally with memory_profiler allows you to trace object allocations and identify the leak source before modifying the serving code, which is the most direct diagnostic step.

Answer

Reduce the number of replicas to 1 to reduce memory contention.

Answer

Increase the machine memory to n1-standard-8 (30 GB).

Answer

Restart the endpoint every hour using a Cloud Scheduler job.

Question 20

A company is deploying a machine learning model for real-time fraud detection. The model must respond to requests within 100ms. The model is a TensorFlow model and will be deployed on Google Kubernetes Engine (GKE). Which Google Cloud service should be used to serve the model to minimize latency?

Accepted Answer

Deploy TensorFlow Serving on GKE with a LoadBalancer service.. Option D is correct because deploying TensorFlow Serving directly on GKE with a LoadBalancer service provides the lowest-latency path for real-time inference. TensorFlow Serving is optimized for high-performance model serving with batching and gRPC support, and GKE allows fine-grained control over node placement, autoscaling, and networking to meet the 100ms SLA. In contrast, serverless options like Cloud Run or Cloud Functions add cold-start latency and lack the low-level optimization for TensorFlow models.

Answer

Deploy the model on Cloud Run with minimum instances set to 1.

Answer

Deploy the model as a Cloud Function triggered by HTTP requests.

Answer

Deploy the model on Vertex AI Prediction with a custom container.