Serving and Scaling Models practice questions

Deploy the challenger to the same endpoint with initial traffic split, e.g., champion 90%, challenger 10%, and gradually adjust.

Deploying the challenger alongside the champion on one endpoint enables Vertex AI traffic splitting, so a small percentage of requests reach the new model while the champion serves the rest, allowing gradual, reversible rollout with error monitoring before full cutover.

Use a single GPU machine (e.g., n1-standard-4 with T4) and perform both preprocessing and inference on the same instance.

Colocating preprocessing and inference on one GPU instance avoids network hops and serialisation between separate services, minimising latency. The CPU-intensive preprocessing shares the instance's CPU while the T4 handles inference, satisfying the latency and cost constraints without provisioning two machines.

Reduce the target CPU utilization percentage from default to a lower value.

Lowering the target CPU utilisation percentage makes the autoscaler treat current load as above target sooner, triggering replica additions earlier during spikes. This directly addresses the slow scale-up, since scaling reacts to utilisation relative to the configured target.

Deploy both models to the same endpoint with different machine types per deployed model.

Vertex AI endpoints support multiple deployed models, each with its own machine type, so the recommender can use a GPU node while the classifier uses a high-memory node. This satisfies the differing resource requirements behind a single endpoint.

Use Dataflow to preprocess the data and write the results to BigQuery or GCS, then launch a batch prediction job on the preprocessed data.

Dataflow runs the custom Python preprocessing at scale, writing transformed records to BigQuery or GCS; Vertex AI batch prediction then scores that prepared data. This separates preprocessing from inference, satisfying the requirement that the Python script run before the model executes.

Convert the model to TensorFlow Lite format.

TensorFlow Lite converts the trained TensorFlow model into a compact, optimised format with reduced precision and size, enabling efficient on-device inference. This satisfies the offline Android deployment constraint, since full TensorFlow runtimes are unsuitable for mobile hardware.

Use Vertex AI Edge Manager to package and deploy the model.

Vertex AI Edge Manager provides managed packaging, versioning and over-the-air deployment of models to fleets of Android and IoT devices, satisfying the offline edge deployment requirement. It handles device targeting and monitoring, which manual sideloading cannot scale to.

Deploy the TFLite model to the edge devices.

Deploying the converted TFLite model satisfies the offline inference constraint: TensorFlow Lite's interpreter runs entirely on-device, requiring no network round-trip to a cloud endpoint. This delivers the low-latency, connectivity-independent execution that Android phones demand, completing the preparation-to-deployment pipeline after conversion and optimisation.

a2-highgpu-1g (with A100 GPU)

GPU inference requires an accelerator-attached machine type, and a2-highgpu-1g pairs one NVIDIA A100 with sufficient vCPU and memory to host the TensorFlow model. General-purpose types such as n1 lack GPUs, so they cannot satisfy the GPU constraint.

The maxReplicas limit is set to 10, but the cooldown period is preventing rapid scaling.

Correct. Cooldown periods can delay scaling decisions, especially for short spikes.

Create a new index version, deploy it to the same endpoint, and then update the endpoint to use the new index version.

Deploying a new index version to the existing endpoint, then switching the endpoint's traffic to it, avoids rebuilding the endpoint. This satisfies the no-downtime constraint because the old version keeps serving queries until the switch completes.

Export the model to TensorFlow Lite and use Vertex AI Edge Manager for deployment.

TensorFlow Lite produces a compact, optimised model that runs inference locally on the device without network calls, meeting the offline and low-latency constraints. Vertex AI Edge Manager handles deployment and monitoring across the edge fleet.

Use the gcloud ai endpoints update command to change traffic split.

The gcloud ai endpoints update command lets you modify the deployed models' traffic split percentages directly, enabling gradual shifts from 90:10 towards 50:50 without redeploying. This satisfies the requirement to incrementally adjust traffic over the week.

Configure the endpoint to use custom metrics from Cloud Monitoring.

Cloud Monitoring custom metrics let the endpoint autoscale on signals such as request queue depth or inference latency, satisfying the stem's sub-100ms P99 constraint. Vertex AI online prediction endpoints natively integrate with Cloud Monitoring, so scaling reacts to your own telemetry rather than coarse CPU utilisation, which lags real-time demand.

Set a target value for the custom metric in the autoscaling policy.

An autoscaling policy needs a target threshold to act on; setting one for the custom metric tells Vertex AI when to add or remove replicas. This satisfies the requirement to autoscale against your own latency signal rather than default CPU utilisation.

Deploy the model with a GPU machine type and use TensorRT optimisation.

GPUs accelerate the PyTorch model's matrix operations, and TensorRT fuses layers and applies precision calibration to cut latency. This addresses both the cost and performance constraints for the custom-op NLP workload currently bottlenecked on CPU.

Use Vertex AI Model Optimisation to automatically quantise and compile the model.

Vertex AI Model Optimisation applies post-training quantisation and XLA compilation, shrinking the PyTorch model's memory footprint and cutting inference latency on Vertex AI Prediction. This directly addresses the cost and performance constraints for the large-scale NLP workload, including its custom ops, without manual re-engineering.

Integrate the model with NVIDIA Triton Inference Server for dynamic batching and model ensembles.

NVIDIA Triton Inference Server provides dynamic batching, which coalesces concurrent inference requests into larger batches to raise GPU utilisation, plus model ensembles that chain preprocessing and postprocessing steps. This directly addresses the cost and performance optimisation goal for the PyTorch model with custom ops, without requiring model conversion.

Set minReplicas to 1.

Setting minReplicas to 1 keeps at least one replica permanently provisioned, so the endpoint never scales to zero. That directly satisfies the no-cold-start constraint: burst traffic hits a warm, already-loaded container instead of waiting for a new replica to initialise, preserving the low-latency requirement for online predictions.

Use a machine type with sufficient memory for the model.

Sufficient memory prevents out-of-memory failures and swapping, which would otherwise inflate inference latency during bursts. However, memory alone does not eliminate cold start; that requires minimum instance count and scale settings. This option satisfies the stem's low-latency constraint only partially, so it is one of the two correct configurations.

Implement a caching layer using Cloud Memorystore with request hashing.

Hashing each request and checking Cloud Memorystore before invoking the endpoint returns cached predictions for identical inputs, bypassing model execution entirely. This directly minimises latency for repeated requests while reducing serving cost, satisfying the stem's real-time constraint.

Use Vertex AI Model Optimization with TensorRT.

TensorRT applies layer fusion, kernel auto-tuning and FP16/INT8 quantisation, shrinking the model and cutting inference latency on NVIDIA GPUs with minimal accuracy loss. Vertex AI Model Optimization packages these transformations directly into the deployment pipeline, satisfying the GPU acceleration and latency constraints without manual conversion work.

Use a single endpoint with multiple deployed models and traffic allocation.

A single endpoint hosting multiple deployed models with traffic allocation lets each model receive a defined percentage of requests, sharing the underlying endpoint infrastructure. This satisfies the cost-reduction goal while accommodating differing traffic patterns across models without provisioning separate endpoints.

Maintain two separate indexes: one for building and one for serving, and swap them after building.

Vector Search indexes are immutable once built, so updates require rebuilding. Running a second index alongside the serving one lets the company build the refreshed index and swap it in, avoiding downtime during frequent embedding updates.

Use streaming updates to add new embeddings incrementally.

Streaming updates let Vector Search ingest new embeddings incrementally, so the index reflects fresh product data without a full rebuild. This directly satisfies the frequent-update requirement while avoiding the downtime that batch rebuilds cause, since existing serving capacity stays online throughout.

Change min replicas to 0 and max replicas to 10.

Setting min replicas to 0 enables scale-to-zero, so the endpoint provisions no replicas during idle periods and incurs no compute cost. Max replicas of 10 preserves burst capacity, directly satisfying the requirement to reduce costs to zero when there is no traffic.

Increase the minimum number of replicas to maintain a larger buffer.

Raising the minimum replica count keeps spare capacity permanently provisioned, so traffic spikes are absorbed before autoscaling reacts. This directly addresses the slow scale-up constraint, since additional replicas are already serving rather than waiting to be created. It reduces prediction errors during bursts, though at higher baseline cost.