How do you take a machine learning model that works perfectly on your laptop and make it answer millions of user requests a day without crashing? This is the core problem of deploying models to production, a crucial skill for the Google Professional Machine Learning Engineer exam and any real-world ML system. Without proper deployment and scaling, even the most accurate model is useless, because nobody can actually use it.
Jump to a section
A simple way to picture Deploying Models for Serving and Scaling
When a restaurant owner decides to open for business, a great menu is just the start. First, the chef perfects every recipe in a test kitchen, which is like training a model. But that perfect recipe sitting on a shelf doesn't feed anyone. The real challenge begins when the doors open and customers flood in. This is the moment the kitchen must be deployed for serving, meaning the trained model is put to work answering live requests from users.
If only two parties arrive per hour, a single chef can fry each dish from scratch. But when lunch rush hits, hundreds of orders arrive every minute. The chef cannot cook each dish sequentially; that would cause a disaster (or, in tech terms, high latency and timeouts). Instead, the kitchen must scale. This means the restaurant needs multiple stoves, more prep cooks, and a system to distribute orders evenly. In machine learning, this is an endpoint, the address where users send their requests, paired with a load balancer that spreads traffic across many copies of the model.
The restaurant also needs a plan for scaling. A static kitchen with two stoves will fail on a Saturday night. A kitchen that can automatically add more stoves and cooks as the queue grows, and remove them when things quiet down, uses elastic scaling. Finally, to protect the kitchen, the owner sets a maximum capacity and a queue. If too many orders arrive, new customers get a busy signal (a 429 error) instead of having their order lost or the kitchen set on fire. This is exactly how Vertex AI Endpoints manage production traffic: train in a lab, deploy to a live address, and continuously adjust resources to match demand.
When you train a machine learning model on your computer, it's like having a brilliant but very shy genius living in your basement. They can answer your questions one at a time, but they cannot handle a crowd. To make that genius useful to the world, you need to move them to a public office with a phone number (the endpoint) and hire a team of assistants (scaling). This process is called deploying a model for serving.
Let's break down the key components. First, the model itself is just a file containing all the patterns it learned from data. Think of it as a complex set of instructions. To serve the model, you need to host it somewhere accessible on the internet. In Google Cloud, this place is called Vertex AI. You upload your model to Vertex AI's model registry (a catalogue of your models). Then, you create an endpoint. An endpoint is a specific URL, or web address, that applications can send requests to. When a user's app wants a prediction, it sends data to that URL.
Now, what happens when thousands of users send requests at the same time? If the model is running on just one server (a single powerful computer), it can only handle one request at a time. The rest have to wait. This creates latency (delay) and eventually causes the server to crash. This is where scaling comes in. Scaling means running multiple copies of your model simultaneously. Each copy, called a replica, lives on its own virtual machine (a simulated computer in the cloud). If you have five replicas, five requests can be handled at the same time, with each request getting its own dedicated model copy.
In Vertex AI Endpoints, you configure how your model scales using autoscaling. Autoscaling is an automatic system that monitors the traffic coming to your endpoint. When traffic goes up, it adds more replicas. When traffic goes down, it removes them. This saves you money and keeps your service fast. You set a minimum number of replicas (your base capacity) and a maximum (the ceiling you are willing to pay for). Within that range, the system decides exactly how many are needed.
Why does this matter instead of a simpler approach? In the past, companies would buy and maintain their own physical servers, guessing how much capacity they would need. They often overpaid for idle machines or under-provisioned during traffic spikes, crashing their websites. Cloud platforms like Google Cloud replaced this with infrastructure as a service (IaaS). You now lease compute power on demand. Vertex AI Endpoints goes a step further: it is a fully managed service. You do not manage the underlying servers, network, or operating system. You simply provide the model and the scaling rules, and Google handles the rest.
Another critical concept is traffic splitting. When you update your model with new data, you do not just swap it out and hope for the best. Instead, you can deploy a new version alongside the old one and send only a small percentage of traffic to it. This lets you test the new model in real-world conditions without risking the entire service. If the new version performs well, you increase the traffic percentage until it handles 100%. This is called canary deployment.
Finally, you must consider prediction types. Online predictions are real-time, low-latency requests like 'is this email spam?' that need an answer in milliseconds. Batch predictions are where you send a large file of data for processing later, like predicting next month's sales for every store. Vertex AI handles both, but online predictions specifically require endpoints and scaling. For the PMLE exam, remember that online serving demands low latency, elastic scaling, and high availability, while batch jobs run asynchronously.
Step 1: Upload the Model to the Model Registry
You take your trained model file (saved in a format like TensorFlow SavedModel or ONNX) and upload it to Vertex AI's central catalogue. This creates a versioned entry that the system can later reference when deploying. Think of this as adding the blueprint to the library.
Step 2: Create an Endpoint
You create a new endpoint resource in Vertex AI. This endpoint will have a unique URL. You must choose a region (e.g., us-central1) that is close to your users to minimise latency. The endpoint is just the door; it does not contain the model yet.
Step 3: Deploy the Model to the Endpoint
You deploy the specific model version from the registry to the endpoint. During this step, you configure the machine type (CPU/GPU), the minimum and maximum number of replicas, and the scaling policy. This is where you tell the system how many copies to run and how to handle traffic.
Step 4: Configure Traffic Splitting (Optional)
If you are deploying a new version alongside an existing one, you set the percentage of traffic each version receives. For example, send 10% to the new version and 90% to the old one. This allows safe testing without impacting all users.
Step 5: Monitor and Adjust
After deployment, use Cloud Monitoring to watch metrics like latency, request count, and error rates. If traffic increases unexpectedly, check if autoscaling is hitting the max replica limit. Adjust min/max values or machine type based on observed performance to optimise cost and speed.
Imagine you work as a machine learning engineer for a global e-commerce company that sells everything from books to garden furniture. Your team has just finished training a recommendation model. When a user visits the homepage, the model should instantly suggest five products they are likely to buy. This is a classic online serving scenario. Now, what do you actually do?
First, you export the trained model and upload it to the Vertex AI Model Registry. This is the central repository where all company models live, each with a version number. You give it a name like 'homepage-recommender-v2'. Next, you create a new endpoint in the same Google Cloud region as your website's servers. If your users are mostly in Europe, you deploy the endpoint to the 'europe-west1' region to minimise network travel time and reduce latency.
You then deploy your model to that endpoint. During deployment, you configure the machine type. The machine type specifies the power of the virtual machine your model will run on, such as the number of CPU cores and the amount of RAM. For a simpler model, a small machine is fine. For a deep neural network, you might need a machine with a GPU (graphics processing unit, a specialised chip for heavy calculations). You also set the minimum number of replicas to 3, ensuring there are always at least three copies of your model running to handle base traffic. You set the maximum to 30, so the system can scale up if a big sale or holiday creates a traffic surge.
Now, what happens step by step in production? \ - A user opens the website. \ - The website's backend server sends a request containing the user's ID and browsing history to your Vertex AI endpoint URL. \ - Vertex AI's load balancer receives the request and forwards it to one of the available replicas. A load balancer is a device that distributes requests evenly to prevent any single replica from being overwhelmed. \ - The replica runs the model, generates five product recommendations, and sends the prediction back. \ - The website displays these recommendations to the user, all within half a second. \
As an IT professional, you must also monitor this setup. You use Cloud Monitoring to set up alerts. If the average latency goes above 500 milliseconds, you get a notification. If the endpoint starts returning errors because all replicas are busy, you check the scaling settings. You might need to increase the maximum number of replicas or optimise the model to run faster. You also practise traffic splitting. Before launching 'v2' to all users, you send 5% of traffic to it. After a day of confirming that it does not cause errors and improves click-through rates, you increase to 50%, then 100%.
The PMLE exam tests the 'Deploying Models for Serving and Scaling' objective (5.1) in very specific ways. You will not be asked to write code or configure a real endpoint from scratch. Instead, the exam presents situational scenarios where you must choose the best deployment strategy or identify the correct setting. The questions are designed to test your understanding of core concepts and your ability to distinguish between similar services.
First, you must know the difference between two key services: Vertex AI Endpoints for online predictions and Cloud Run or Compute Engine for other online workloads. The exam may try to confuse you by offering Cloud Run for deploying a model. Remember: Cloud Run can deploy containers, but for a managed ML model serving, Vertex AI Endpoints is the correct answer because it offers built-in model versioning, autoscaling, and monitoring specifically for ML.
Second, they love testing autoscaling parameters. You must understand the difference between 'min replicas' and 'max replicas'. A typical question: 'Your model serves low traffic at night but spikes during the day. What is the most cost-effective and responsive scaling strategy?' The correct answer is to set min replicas to 1 (or a low number) and max replicas to a higher number. They will try to trap you with 'set max replicas to 1' or 'set min replicas to a high number', which waste money or cause latency.
Third, traffic splitting is a guaranteed topic. The exam will present a scenario where you have a new model version and need to test it safely. The answer will involve deploying both versions to the same endpoint and using a small percentage for the new version. Traps include: 'deploy the new model to a separate endpoint and switch DNS' (too slow and risky) or 'delete the old model first' (dangerous).
Fourth, you need to know about machine types. Questions will test when to use a CPU machine versus a GPU machine, or when to use a high-memory machine. Remember: CPU is fine for simple models (linear regression, small decision trees). GPU is for large neural networks that benefit from parallel processing. High-memory machines are for models that store large embedding tables, like recommendation systems.
Finally, the exam tests your understanding of the difference between online and batch predictions. A typical question: 'You need to generate predictions for a one-time million-row dataset. Which service should you use?' The answer is batch prediction, not an endpoint. They will list Vertex AI Batch Prediction as a separate option. Do not choose the online endpoint for batch work, and vice versa.
Key definitions to memorise: \ - Endpoint: A URL that accepts prediction requests. \ - Replica: A single instance of your model running on a virtual machine. \ - Autoscaling: Automatic adjustment of replica count based on traffic. \ - Traffic splitting: Routing a percentage of requests to different model versions. \ - Canary deployment: Gradually rolling out a new version to a subset of users. \ \ The most common traps involve confusing these terms with general Google Cloud compute services or selecting a manual process when an automated one exists.
Vertex AI Endpoints provide a managed URL to serve online predictions from a trained model.
Autoscaling automatically adjusts the number of model replicas based on traffic, but you must set both a minimum and maximum replica count.
Traffic splitting allows you to safely roll out a new model version by sending a small percentage of requests to it first.
Online predictions require low latency and are used for real-time applications, whereas batch predictions process large files asynchronously.
GPU machines are necessary for large deep learning models, while CPU machines suffice for simpler models like linear regression.
A model must be uploaded to the Vertex AI Model Registry before it can be deployed to an endpoint.
These come up on the exam all the time. Here's how to tell them apart.
Vertex AI Endpoints
Built for ML model serving with model registry integration.
Supports traffic splitting between model versions natively.
Autoscaling based on request concurrency targeting model inference.
Cloud Run
General-purpose container execution, not ML-specific.
No built-in model versioning; relies on container tags.
Scales based on HTTP request concurrency, not inference metrics.
Online Prediction
Requires a live endpoint with replicas.
Returns predictions in real-time (milliseconds).
Ideal for interactive applications like chatbots or recommendation systems.
Batch Prediction
Uses an asynchronous job; no endpoint needed.
Processes large files stored in Cloud Storage.
Ideal for periodic tasks like generating monthly reports.
Manual Scaling
You manually set a fixed number of replicas.
Costs are predictable regardless of traffic changes.
Requires constant monitoring to adjust for traffic spikes.
Autoscaling
System automatically adds/removes replicas based on load.
Can be more cost-effective for variable traffic patterns.
Must configure min and max limits to avoid runaway costs.
Canary Deployment
Routes small percentage of traffic to new version gradually.
Allows monitoring of new version on live traffic.
Easy to roll back by reducing traffic to new version.
Blue/Green Deployment
Runs two full environments (blue and green) simultaneously.
Switches all traffic at once from old to new version.
More costly because it requires duplicate infrastructure.
Mistake
Deploying a model is the same as training it, just on a bigger computer.
Correct
Deploying for serving requires setting up an endpoint, configuring scaling, and handling live traffic. Training is a separate process that happens offline before deployment.
Many beginners think once the model is trained, you just 'run it' on a server. They do not understand that serving requires a system to accept requests, manage multiple copies, and handle network communication.
Mistake
If I deploy my model to Vertex AI Endpoints with one replica, it will automatically scale up to handle any traffic.
Correct
Autoscaling only works if you set a maximum replica count greater than your current replicas. With only one replica set as both min and max, the system cannot add more copies, and traffic will fail.
People see 'autoscaling' in the name and assume it happens magically without configuration. They miss the crucial step of setting the maximum limit.
Mistake
The model runs on the endpoint itself.
Correct
An endpoint is just a URL or entry point. The model runs on one or more replicas (virtual machines) behind the endpoint. The endpoint routes traffic to those replicas.
This is a common mental model error. Beginners picture the model living at the URL, when in reality the URL is a door, not the room itself.
Mistake
Batch predictions use endpoints, just like online predictions.
Correct
Batch predictions run as asynchronous jobs and do not require an endpoint. They read input from a file in Cloud Storage and write output back. Online predictions use endpoints for real-time requests.
Confusion arises because both involve getting predictions from a model. Beginners think endpoints are always needed, but batch jobs bypass the need for a live, always-on endpoint.
Mistake
Scaling up replicas always improves latency because there are more machines.
Correct
Scaling reduces latency only up to a point. If requests are very small and frequent, the overhead of distributing requests across many replicas can actually increase latency. Additionally, if the model is slow, more replicas help, but each request still takes the same time to run.
This misconception comes from thinking of servers as infinitely parallelisable. In reality, network overhead and the model's own processing time set a floor on how fast a single request can be served.
Reveal each answer, then mark whether you got it right. Score 60%+ to unlock the next chapter.
Vertex AI Endpoints is specifically designed for serving ML models with built-in model registry, autoscaling for predictions, and traffic splitting. Cloud Run is a general-purpose compute service for running stateless containers, but it lacks ML-specific features like model versioning.
For the PMLE exam, you do not need to write code. You need to understand the concepts and configuration choices. In practice, you can deploy using the Google Cloud Console, gcloud CLI commands, or the Python SDK.
Setting min replicas to 0 means the endpoint can scale down to zero when there is no traffic. This saves costs but causes a cold start delay (the first request may be slow) because a new replica must be spun up. For production services that need fast responses, a min of 1 or more is typical.
Yes, you can deploy a new model version to the same endpoint and use traffic splitting to gradually shift traffic. The old version stays online until you manually undeploy it, ensuring zero downtime.
A 429 error occurs when the endpoint receives more requests than it can handle, and the configured maximum replicas are already fully utilised. You can avoid this by increasing the maximum replica limit or by using a queueing system.
Vertex AI Endpoints primarily scales based on request traffic (number of concurrent requests), not CPU usage. This is different from general compute services. You define the target utilization, and the system scales to keep requests flowing.
You've finished Deploying Models for Serving and Scaling. Continue through the PMLE study guide to build a complete picture of the exam.
Done with this chapter?