You are an ML engineer at a large e-commerce company. Your team has developed a product recommendation model using TensorFlow and deployed it on Vertex AI Endpoints for real-time inference. The model is retrained weekly using a Vertex AI Pipeline that reads new user interaction data from BigQuery, trains the model, evaluates it, and deploys the new version to the endpoint with a traffic split: 10% to the new model and 90% to the previous champion model. Recently, the team noticed that the new model's online prediction latency has increased significantly (from 50ms to 200ms) after deployment, causing timeouts for some requests. The training code has not changed, and the model size is similar. The pipeline uses a custom container with the same TensorFlow Serving image as before. The deployment step uses the same machine type (n1-standard-4) for the endpoint. What is the most likely cause of the latency increase?
A data validation step might have inadvertently added preprocessing ops, increasing latency.
Why this answer
The pipeline now includes a data validation step that modifies the SavedModel's serving signature, adding an extra preprocessing operation. This additional operation runs during inference on Vertex AI Endpoints, increasing the per-request latency from 50ms to 200ms, even though the model architecture and size remain unchanged. The custom container and machine type are identical, so the latency increase must stem from a change in the serving graph itself.
Exam trap
Google Cloud often tests the concept that changes in the ML pipeline (like adding a data validation step) can alter the serving signature and increase latency, even when the model architecture and infrastructure remain unchanged, tricking candidates into focusing on hardware or data distribution instead.
How to eliminate wrong answers
Option A is wrong because the endpoint uses the same machine type (n1-standard-4) as before, so the machine is not the cause of the latency increase. Option B is wrong because the training code has not changed and the model size is similar, indicating the architecture is not significantly different. Option D is wrong because data skew affects prediction accuracy, not latency; it does not explain a 4x increase in inference time.