PMLE Serving and Scaling Models — 50 Questions

Question 1

A team is building a batch prediction pipeline that processes raw data from Cloud Storage, performs complex preprocessing, and then runs predictions using a large model. The preprocessing step is compute-intensive and the prediction step is I/O-bound. Which TWO Google Cloud services should they combine to optimize cost and performance? (Choose 2)

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Dataflow for preprocessing and writing results to Cloud Storage. Dataflow is ideal for the compute-intensive preprocessing step because it can horizontally scale across many workers to handle complex transformations in parallel, and it writes results directly to Cloud Storage, which serves as the input source for Vertex AI Batch Prediction. Vertex AI Batch Prediction is optimized for I/O-bound inference workloads: it reads batches of data from Cloud Storage, runs predictions using the large model, and writes results back to Cloud Storage, all without requiring a persistent serving endpoint, which minimizes cost for offline predictions.

Answer

Cloud Functions to preprocess data row by row

Answer

Cloud Run to serve the preprocessed data as an API

Answer

Vertex AI Batch Prediction with BigQuery source

Question 2

A data scientist wants to deploy a trained TensorFlow model to Vertex AI for online predictions. They need to serve predictions with low latency and want to leverage GPU acceleration. Which machine type should they select when creating the Vertex AI endpoint?

Accepted Answer

n1-standard-4 with 1 NVIDIA Tesla T4. The n1-standard-4 machine type supports attaching GPUs such as the NVIDIA Tesla T4, which provides GPU acceleration for low-latency online predictions. Vertex AI endpoints require a machine type that allows GPU attachment, and the n1-series is one of the few families that supports GPUs, while the T4 offers a good balance of cost and performance for inference workloads.

Answer

n1-standard-4

Answer

e2-standard-4

Answer

n1-highmem-8

Question 3

You are deploying a new version of a model to a Vertex AI endpoint that already has a champion model serving 100% of traffic. You want to gradually shift traffic to the new version while monitoring for errors. Which approach should you use?

Accepted Answer

Deploy the challenger to the same endpoint with initial traffic split, e.g., champion 90%, challenger 10%, and gradually adjust.. Vertex AI endpoints support traffic splitting between model versions deployed to the same endpoint. By deploying the challenger to the same endpoint and setting an initial split (e.g., champion 90%, challenger 10%), you can gradually shift traffic while monitoring for errors. This approach uses the endpoint's built-in traffic management, avoiding the complexity and latency of external load balancers.

Answer

Use Cloud Load Balancing with weighted backend services pointing to different endpoints.

Answer

Delete the champion model and redeploy with the challenger as the new version.

Answer

Create a new endpoint for the challenger and use a load balancer to split traffic.

Question 4

A company is using Vertex AI Prediction with a custom container that performs preprocessing before inference. The preprocessing step is CPU-intensive and the inference step uses a GPU. They want to minimize prediction latency while optimizing cost. Which architecture should they use?

Accepted Answer

Use a single GPU machine (e.g., n1-standard-4 with T4) and perform both preprocessing and inference on the same instance.. Using a CPU-only node for preprocessing and then sending the preprocessed data to a GPU node for inference separates concerns and allows independent scaling, but adds network latency. The best approach is to use a single machine with both CPU and GPU to avoid network round-trip, and to adjust the machine type to have enough CPU resources.

Answer

Use Cloud Run for preprocessing and send HTTP requests to a GPU-backed Vertex AI endpoint for inference.

Answer

Use two separate Vertex AI endpoints: one CPU-based for preprocessing, one GPU-based for inference, and chain them with Cloud Tasks.

Answer

Use Dataflow for preprocessing and then invoke the model, but Dataflow is not designed for real-time prediction.

Question 5

You need to serve a large embedding model for similarity search with low latency. The model was trained to generate 256-dimensional embeddings. You plan to use Vertex AI Vector Search. Which index type should you choose to balance accuracy and performance for a dataset with 10 million vectors?

Accepted Answer

Approximate nearest neighbor (ANN) index using ScaNN. Vertex AI Vector Search uses ScaNN (Scalable Nearest Neighbors) as its underlying ANN algorithm, which is specifically designed for high-dimensional embeddings (like 256-d) and large-scale datasets (10M vectors). ScaNN balances accuracy and performance by employing anisotropic quantization and tree-based partitioning, making it the optimal choice for low-latency similarity search without requiring exhaustive comparison.

Answer

Tree-based index

Answer

Brute-force index

Answer

Hash-based index

Question 6

A machine learning engineer needs to run batch predictions on 50 TB of data stored in BigQuery using a Vertex AI model. The model is a custom container. What is the most efficient way to set up the batch prediction job?

Accepted Answer

Create a Vertex AI batch prediction job with BigQuery source and BigQuery destination.. Vertex AI batch prediction supports BigQuery as both input and output source, which is the most direct approach. Dataflow preprocessing is optional only if needed.

Answer

Use Dataflow to process the data and call the model via Vertex AI online prediction.

Answer

Export BigQuery data to CSV in GCS, then create a batch prediction job with GCS source.

Answer

Create a Cloud Function to iterate over BigQuery rows and call the endpoint.

Question 7

You have a Vertex AI endpoint with min_replica_count=2 and max_replica_count=10. You notice that during a traffic spike, the endpoint does not scale up quickly enough, causing increased latency. What should you do to improve autoscaling responsiveness?

Accepted Answer

Reduce the target CPU utilization percentage from default to a lower value.. Reducing the target CPU utilization percentage (e.g., from the default 60% to a lower value like 40%) causes the autoscaler to trigger scale-up actions sooner, as the threshold for adding replicas is reached at a lower CPU load. This improves responsiveness during traffic spikes by initiating scaling earlier, reducing latency. The endpoint's min_replica_count=2 and max_replica_count=10 remain unchanged, so the scaling range is preserved.

Answer

Increase max_replica_count to 20.

Answer

Disable autoscaling and manually manage replicas.

Answer

Increase min_replica_count to 10.

Question 8

You are deploying a PyTorch model on Vertex AI using a custom container with NVIDIA Triton Inference Server. The model is a large transformer that requires GPU. You want to optimize GPU utilization and reduce memory footprint. Which technique should you apply?

Accepted Answer

Apply model quantization using TensorRT.. Model quantization using TensorRT reduces the precision of model weights (e.g., from FP32 to FP16 or INT8), which directly decreases GPU memory usage and can improve throughput by enabling faster arithmetic operations on compatible NVIDIA GPUs. This technique is specifically designed to optimize GPU utilization and memory footprint for large transformer models deployed with Triton Inference Server.

Answer

Enable dynamic batching in Triton.

Answer

Use CPU-only instances to avoid GPU memory issues.

Answer

Increase the number of GPU replicas.

Question 9

A company wants to cache predictions for identical requests to reduce latency and cost. They use Vertex AI Prediction with a custom container. Which GCP service should they use to implement prediction caching?

Accepted Answer

Cloud Memorystore for Redis. Cloud Memorystore (Redis) is ideal for caching because it provides low-latency key-value storage. The prediction request can be hashed to create a key.

Answer

Cloud Bigtable

Answer

Cloud Storage

Answer

Cloud Firestore

Question 10

You have a Vertex AI endpoint that serves a model for real-time predictions. You want to update the model to a new version with zero downtime. Which approach should you take?

Accepted Answer

Deploy the new model version to the same endpoint and then set traffic to 100% for the new version.. Vertex AI endpoints support canary deployments by allowing you to deploy a new model version to the same endpoint and then gradually shift traffic to it using the `traffic_split` parameter. Setting traffic to 100% for the new version after deployment ensures zero downtime, as the endpoint remains active and serves requests from the old version until the switch is complete.

Answer

Delete the endpoint and recreate it with the new model.

Answer

Use Cloud Load Balancing to switch traffic between two endpoints.

Answer

Create a new endpoint and update the client application to point to the new endpoint.

Question 11

You are using Vertex AI Vector Search with an approximate nearest neighbor index. You need to update the index with new data every hour. The updates must be available for queries immediately. Which update method should you use?

Accepted Answer

Streaming updates using the streaming API.. Vertex AI Vector Search supports streaming updates via its streaming API, which allows you to insert, update, or delete vectors in real time. This ensures that new data is immediately available for approximate nearest neighbor (ANN) queries without requiring index recreation or redeployment, meeting the requirement for hourly updates with instant query availability.

Answer

Recreate the index every hour using a scheduled job.

Answer

Batch update by creating a new index and deploying it.

Answer

Use a brute-force index that supports real-time updates.

Question 12

An ML team wants to deploy multiple models (e.g., a recommender and a classifier) behind a single Vertex AI endpoint. The models have different resource requirements: the recommender needs GPU, the classifier needs high memory. How should they configure the endpoint?

Accepted Answer

Deploy both models to the same endpoint with different machine types per deployed model.. Vertex AI allows deploying multiple models on the same endpoint, each with its own machine type and resources. Traffic splitting routes requests to the correct model.

Answer

Use Cloud Run for one model and Vertex AI for the other.

Answer

Use a single machine type that meets the highest requirements.

Answer

Create separate endpoints for each model.

Question 13

You need to run a batch prediction job on Vertex AI using a model that requires custom preprocessing using a Python script. The preprocessing must be applied before inference. Which approach should you use?

Accepted Answer

Use Dataflow to preprocess the data and write the results to BigQuery or GCS, then launch a batch prediction job on the preprocessed data.. Dataflow (Apache Beam) is the recommended serverless service for distributed, scalable preprocessing of large datasets on Google Cloud. It can read raw data from GCS, apply custom Python preprocessing logic, and write the preprocessed results to GCS or BigQuery. The batch prediction job then reads the preprocessed data directly, avoiding the need to embed preprocessing in the prediction container or handle data on a single VM.

Answer

Use Cloud Functions to preprocess each record individually.

Answer

Preprocess the data on a single VM and then upload to GCS.

Answer

Include the preprocessing logic in the custom container used for batch prediction.

Question 14

A company is deploying a model on Vertex AI for online predictions with strict latency SLOs. The model requires GPU acceleration. Which TWO configurations should they consider to meet the SLOs while optimizing cost?

Accepted Answer

Set min_replica_count to handle base traffic and max_replica_count to handle spikes. Enabling autoscaling with min_replica_count to handle base load and max_replica_count for spikes, and using scale-to-zero for non-production (but for production, scale-to-zero may not meet SLOs due to cold starts; however, the question says 'optimizing cost', so scale-to-zero is not appropriate for low latency. The correct answers are: set appropriate min and max replicas, and use GPU-enabled machine types. The other options are either irrelevant or counterproductive.

Answer

Use n1-highmem-32 machine types without GPU

Answer

Enable autoscaling with min_replica_count=0 and max_replica_count=10

Answer

Disable autoscaling and set a fixed number of replicas equal to peak load

Question 15

You are designing a batch prediction pipeline using Vertex AI. The input data is 100 TB of images stored in Cloud Storage. The model is a custom TensorFlow model that expects TFRecord format. The pipeline must be cost-effective and run within a time window of 2 hours. Which THREE steps should you include?

Accepted Answer

Create a Vertex AI batch prediction job with input from GCS (TFRecord files).. Vertex AI batch prediction jobs natively accept TFRecord files stored in Cloud Storage as input, which aligns with the requirement for a custom TensorFlow model. This approach is cost-effective and can complete within 2 hours by leveraging Vertex AI's managed infrastructure, avoiding the need to spin up and manage compute resources manually.

Answer

Store batch prediction results in BigQuery.

Answer

Use Cloud Functions to convert images to TFRecord.

Question 16

An organization wants to deploy a model on edge devices (e.g., Android phones) for offline inference. They trained a model using TensorFlow. Which THREE steps should they take to prepare and deploy the model?

Accepted Answer

Convert the model to TensorFlow Lite format.. For edge deployment on Android, you need to convert the model to TensorFlow Lite, use Vertex AI Edge Manager to manage the deployment, and then deploy the TFLite model to the devices. Exporting to ONNX is for other platforms, and creating a REST endpoint is for online serving.

Answer

Deploy the model to a Vertex AI endpoint for online inference.

Answer

Export the model to ONNX format.

Question 17

You deployed a model to a Vertex AI endpoint with minReplicas=0 and maxReplicas=5. After sending prediction requests, you notice the endpoint takes about 30 seconds to respond initially, but subsequent requests are fast. What is the most likely cause?

Accepted Answer

Cold start occurs because the endpoint scaled down to zero.. Vertex AI endpoints with minReplicas=0 scale down to zero when idle. The first request after a period of inactivity triggers a cold start, where the endpoint must provision a new VM instance and load the model, causing a ~30-second delay. Subsequent requests are fast because the instance remains warm and handles them without provisioning overhead.

Answer

The model is too large for the machine type.

Answer

The VPC Service Controls are blocking the initial request.

Answer

The endpoint's autoscaling is misconfigured.

Question 18

You have a champion model serving 100% traffic on a Vertex AI endpoint. You want to deploy a challenger model and gradually shift 10% of traffic to it for A/B testing. What is the correct approach?

Accepted Answer

Deploy the challenger on the same endpoint and use the traffic split parameter to allocate 10% traffic to it.. Vertex AI endpoints support traffic splitting by deploying multiple model versions and assigning traffic percentages. You deploy the challenger as a new deployed model on the same endpoint and set traffic split: champion 90%, challenger 10%.

Answer

Use Cloud Run to deploy both models and use Cloud Endpoints for traffic splitting.

Answer

Deploy the challenger on a separate endpoint and use Cloud Armor to split traffic.

Answer

Create a new endpoint for the challenger and route 10% of requests via a load balancer.

Question 19

You need to run batch predictions on 10 TB of text data stored in BigQuery using a custom container model hosted in Vertex AI. What is the most cost-effective and simple approach?

Accepted Answer

Use Vertex AI batch prediction with BigQuery source and sink.. Vertex AI batch prediction natively supports BigQuery as both source and sink, allowing you to run predictions on 10 TB of text data without any data movement or intermediate storage. This is the most cost-effective and simple approach because it eliminates the need for exporting data, managing infrastructure, or calling online endpoints, and it leverages Vertex AI's optimized batch inference infrastructure that scales automatically.

Answer

Use Cloud Run jobs to read from BigQuery and write results back.

Answer

Export BigQuery data to GCS, then run a Dataflow pipeline to call the model's online prediction endpoint for each row.

Answer

Use Cloud Dataproc to spin up a Spark cluster and run the model inference in parallel.

Question 20

Your team is deploying a large recommendation model on Vertex AI endpoints using GPUs. You need to minimise latency while optimising cost. The model serves many similar requests from the same users within short time windows. Which additional service would best reduce latency and cost?

Accepted Answer

Use Cloud Memorystore to cache prediction results.. Caching identical prediction requests can reduce load on the model and improve latency. Cloud Memorystore (Redis) can be used to cache responses based on a hash of the request, and the endpoint can check cache before invoking the model.

Answer

Switch to CPU-only instances to reduce cost.

Answer

Increase maxReplicas to handle the load without caching.

Answer

Set up a Cloud CDN in front of the endpoint.