Model deployment is the act of taking a trained machine learning model and making it available to serve predictions to real users or systems. In the AWS Certified Machine Learning Specialty exam, understanding how to deploy models and manage inference endpoints on SageMaker is crucial because it represents the final, production-ready stage of any ML project — where your model actually delivers value.
Jump to a section
A simple way to picture Model Deployment and Inference on SageMaker
Your kitchen at home, late on a Friday night. You’ve spent months perfecting a single flavour of ice cream — let’s call it “Choco-Vanilla Swirl.” It’s your masterpiece. You’ve tested it, tweaked it, and it’s ready for the world. But your home freezer can only serve one cone at a time. If fifty neighbours show up at your door, each asking for a scoop, your tiny freezer can’t handle the queue. The ice cream melts, the cones droop, and everyone leaves hangry.
So you rent a commercial catering kitchen. This kitchen has multiple industrial freezers, each kept at the perfect temperature. You hand over your one tub of Choco-Vanilla Swirl, and the kitchen staff copy it into every freezer so each unit has its own ready supply. Now when fifty neighbours arrive, the catering manager instantly directs each person to the next available freezer station. No queues, no melting, just a perfect scoop every time. When the weekend rush ends, you close some freezers to save money until next Saturday’s demand spikes again. This renting, copying, directing, and scaling is exactly what SageMaker does when you deploy a machine learning model. The model is your ice cream recipe, the freezers are compute instances, and the catering manager is the inference endpoint that routes each request to an available copy of your model.
Model deployment on SageMaker means taking a trained machine learning model — a mathematical representation that has learned patterns from data — and placing it onto a persistent, scalable server infrastructure so that other applications can send data to it and receive predictions back. This process is called “inference” when the model is used to make predictions on new, unseen data, as opposed to “training” where the model learns from labelled examples.
Before cloud platforms like AWS, deploying a model required an IT team to buy physical servers, install operating systems, set up networking, configure load balancers (which distribute incoming traffic across multiple servers), and write custom API code (application programming interface — the rules that allow two pieces of software to talk to each other) to accept requests and return predictions. It was slow, expensive, and brittle. SageMaker automates almost all of that.
The core component for deployment on SageMaker is the “endpoint.” An endpoint is a fully managed HTTPS web address (a secure URL that applications call over the internet) that your model lives behind. You configure the endpoint with three things:
The model itself, which you package into a “model artifact” — typically a single file or compressed folder containing the trained model’s parameters and the code needed to run it.
The “inference code” — a Python script that tells SageMaker how to load your model, preprocess incoming data (clean and format it before the model sees it), and postprocess the model’s output (turn raw numbers into a human-readable prediction).
The “instance type” and “instance count” — the hardware specifications of the virtual servers (called EC2 instances) that will run your model. For example, you might choose a ml.m5.large instance with 2 virtual CPUs and 8 GB of RAM, and set the count to 3 so three identical copies of your model run simultaneously.
When you create the endpoint, SageMaker automatically provisions the instances, copies your model artifact onto each one, starts the inference code, and sets up a load balancer across all instances. Once the endpoint status shows “InService,” it is ready to accept inference requests.
“Real-time inference” is the default mode. You send a single request — for example, a JSON object containing a customer’s features — and within milliseconds the endpoint returns a prediction. This is ideal for interactive applications like a loan approval system where a user waits for an answer.
“Batch transform” is a different approach used when you have a large dataset of records that all need predictions at once, but the results are not needed instantly. You point SageMaker at an S3 bucket (AWS’s storage service) containing your input file, SageMaker spins up temporary instances, runs all records through the model, and writes the predictions to another S3 bucket. You pay only for the time the instances are running, and you do not keep an endpoint live.
“Serverless inference” is a newer option where you do not choose any instance type or count at all. SageMaker manages the underlying compute automatically, scaling to zero when there are no requests and scaling up instantly when requests arrive. You pay per request and per amount of memory used. This is great for sporadic workloads where you cannot predict traffic patterns.
“Multi-model endpoints” allow you to host multiple models behind a single endpoint. SageMaker loads each model into memory only when it is requested, saving money when you have many models but each is used infrequently. “Multi-container endpoints” let you run multiple containers — each containing different code or dependencies — behind one endpoint, useful when you need to chain a preprocessor and a model together.
All these options share the same deployment workflow: you register a model in the SageMaker Model Registry (a central catalogue of your trained models), you create an endpoint configuration specifying hardware and scaling settings, and you create the endpoint. AWS handles the rest.
Key concepts to remember for the exam:
“Endpoint” is the live, running URL for inference.
“Model artifact” is the saved model file (often a .tar.gz file) stored in S3.
“Inference code” is the Python script that contains a function called model_fn (loads the model) and predict_fn or input_fn and output_fn (handles data transformation).
“Auto Scaling” adjusts the number of instances behind an endpoint based on traffic, defined via a “scaling policy” that monitors metrics like latency or request count.
“Variant” refers to a specific combination of instance type and count within a production endpoint. You can use multiple variants for A/B testing — sending a percentage of traffic to a new model variant while the rest goes to the old one.
Upload Model Artifact to S3
Take the trained model file (usually a .tar.gz file) and upload it to an S3 bucket. This is the raw model data that SageMaker will copy onto each inference instance. Without this step, SageMaker has nothing to deploy.
Write Inference Code
Create a Python script (e.g., inference.py) that contains model_fn to load the model from disk, and input_fn/ predict_fn/ output_fn to handle data transformation and prediction logic. This code defines how your model interacts with the outside world.
Create a SageMaker Model Resource
In the AWS console or SDK, specify the S3 location of the model artifact and the container image (either a built-in framework like XGBoost or a custom container). This registers the model in SageMaker's catalog so it can be referenced by endpoints.
Create an Endpoint Configuration
Define the instance type (e.g., ml.m5.large), instance count, auto scaling settings, and any production variants for A/B testing. This tells SageMaker what hardware to provision and how to route traffic.
Create the Endpoint
SageMaker takes the endpoint configuration and model resource, provisions the specified instances, copies the model and inference code onto each, and starts the inference server. When the endpoint status shows 'InService', you can send HTTPS requests to it.
Test and Monitor the Endpoint
Send sample requests to the endpoint URL using curl or the AWS SDK. Check CloudWatch logs for errors. Enable data capture to log all requests and responses for later analysis. This step validates that the deployment is working correctly in production.
Imagine you work as a machine learning engineer for a UK-based online retailer called “BookWorm.” The company has trained a model that predicts whether a customer will buy a recommended book based on their browsing history. The model is a gradient-boosted tree that achieved 92% accuracy during testing. The business wants to integrate this model into their main website so that when a customer views a book page, the site instantly shows a “You might also like” section.
Here is what you actually do, step by step:
First, you take the trained model file (let’s call it model.tar.gz) and upload it to an S3 bucket in the same AWS region as your application. This is the model artifact.
Next, you write a small Python file called inference.py. It contains two essential functions: model_fn(model_dir) which loads the model from the local directory SageMaker provides, and predict_fn(input_data, model) which takes the incoming JSON (the customer’s browsed book IDs and time spent on each page), transforms them into the numeric features the model expects, runs the prediction, and returns a list of recommended book IDs.
You package inference.py together with any dependencies (like a list of book titles) into a separate archive, but SageMaker’s built-in frameworks (like XGBoost or Scikit-learn) often handle this automatically if you use their containers.
You then use the SageMaker console (or SDK) to create a model resource, pointing to the S3 location of model.tar.gz and to the container image for XGBoost.
You create an endpoint configuration. For BookWorm’s traffic — about 10,000 requests per minute during peak hours — you decide on two “production variants.” The first variant uses two ml.c5.large instances running the current model. The second variant uses one ml.c5.large instance running a candidate model you want to test. You set the traffic weight to 90% for the first variant and 10% for the second. This is called A/B testing.
You create the endpoint. SageMaker spins up the instances, loads the model, and within about 10 minutes the endpoint is “InService.” You use Amazon Route 53 (AWS’s DNS service) to point your website’s recommendation API call to the endpoint’s URL.
As traffic grows, you set up “auto scaling.” You create a scaling policy that tracks the “InvocationsPerInstance” metric. If each instance receives more than 100 requests per second for 5 consecutive minutes, SageMaker launches an additional instance automatically. When traffic drops, it removes instances to save cost.
You also enable “data capture” on the endpoint, which logs every request and response to S3. This lets you audit predictions later, debug when the model makes mistakes, and gather data for retraining.
When the A/B test shows that the candidate model improves click-through rate by 5%, you shift all traffic to the new variant and delete the old one. You update the model registry to mark the old model as “deprecated” and the new one as “approved.”
In practice, you rarely touch the infrastructure directly. You write configuration files (often in YAML or using AWS CloudFormation) that declare the endpoint and its properties, and let SageMaker’s orchestration handle the rest. The most common struggles are: choosing the right instance type (too small leads to timeouts, too large wastes money), setting sane auto scaling limits, and ensuring the inference code handles malformed input gracefully (so a bad request doesn’t crash the endpoint).
The MLS-C01 exam dedicates a significant portion of objective 4.1 to testing your knowledge of how SageMaker performs model deployment and inference. The questions are scenario-based, meaning you get a business requirement (e.g., “lowest cost for sporadic traffic”, “must handle 5000 requests per second”, “need to compare two models in production”) and you must pick the correct AWS service or configuration.
Question types you will see:
“A company needs to serve predictions with sub-100 millisecond latency from a trained model. Which deployment option should they use?” Correct answer: Real-time inference endpoint, because batch transform has startup overhead and serverless can have cold starts exceeding that latency.
“A data scientist wants to run predictions on a 10 GB CSV file once per month. What is the most cost-effective approach?” Correct answer: Batch transform, because you don’t need to keep an endpoint running 24/7.
“You need to deploy 50 different models, each used by a different team only a few times a day. What is the cheapest option?” Correct answer: Multi-model endpoint, because models are loaded on demand, avoiding the cost of 50 separate endpoints.
Concepts the exam loves to test (and where they set traps):
The difference between “real-time inference” and “batch transform”. Trap: They might describe a scenario where results are needed quickly but not instantly (e.g., “within 5 minutes”) and suggest batch transform is fine — but batch transform can take minutes to spin up instances, so real-time may still be needed if the wait is unacceptable.
“Serverless inference” vs “real-time inference with auto scaling”. Trap: Serverless is only suitable for workloads with a max concurrency of 200 per endpoint and a maximum memory of 6 GB. If the question mentions high throughput or large models (like a BERT language model needing 8 GB GPU memory), serverless is not an option — you need a real-time endpoint on GPU instances.
“Multi-container endpoints” vs “multi-model endpoints”. Trap: Multi-container is for chaining different processing steps (e.g., a data preprocessor container and a model container) in a pipeline. Multi-model is for hosting multiple separate models behind one endpoint and SageMaker swaps them in memory based on request headers. They are not interchangeable.
“Elastic Inference” (EI) — an older feature that accelerated deep learning inference using a dedicated accrual hardware. Trap: Elastic Inference is now deprecated and not supported on new instance types. If the question implies EI is a current option, the answer is probably to use a GPU instance directly or use serverless if possible.
“Endpoint configuration” vs “model” vs “endpoint”. Trap: The exam will give you the correct order — you first create a model (pointing to the artifact and container), then an endpoint configuration (specifying instances and scaling), then the endpoint. A wrong answer might say you create the endpoint first or that the configuration is optional.
“Auto scaling” based on “SageMakerVariantInvocationsPerInstance” vs custom CloudWatch metrics. Trap: SageMaker only scales on the “InvocationsPerInstance” metric or a custom metric you explicitly publish. You cannot use generic EC2 CPU utilisation out of the box — you must set up a custom scaling policy.
Memorise these definitions verbatim: - “Model artifact”: the output of training, stored in S3. - “Endpoint”: the HTTPS URL for live predictions. - “Variant”: a specific instance type and count behind an endpoint, used for A/B testing. - “Data capture”: logging of request/response payloads to S3 for monitoring or retraining. - “Inference pipeline”: a series of containers that pass data through preprocessing, prediction, and postprocessing steps.
The exam also tests the “Model Registry” (part of SageMaker Projects) where models are versioned and promoted from “Pending” to “Approved” to “Deployed”. Questions will ask which status a model must have before it can be deployed to production — answer: “Approved”.
A common trap question: “You want to deploy a PyTorch model. Which SageMaker built-in container should you use?” Answer: “PyTorch container version 1.13 or higher” — but if the exam mentions a custom pre-processing script, you might need to use the “SageMaker Inference Toolkit” with your own container. They love testing when to use a custom container versus a built-in framework container.
A real-time inference endpoint provides a persistent HTTPS URL for low-latency predictions, ideal for interactive applications.
Batch transform processes entire datasets at once and automatically shuts down after completion, saving money for non-real-time workloads.
Serverless inference scales to zero and charges per request, but is limited to 200 concurrent calls and 6 GB of memory.
You must always provide inference code (input_fn, predict_fn, output_fn) alongside your model artifact to SageMaker.
Auto scaling for endpoints can only be configured using the 'SageMakerVariantInvocationsPerInstance' metric or a custom CloudWatch metric.
Multi-model endpoints reduce costs by loading infrequently used models on demand, but all models must share the same container and fit in memory.
Data capture on an endpoint logs request/response pairs to S3, which is essential for monitoring, debugging, and retraining.
These come up on the exam all the time. Here's how to tell them apart.
Real-time Inference Endpoint
Persistent HTTPS URL always available for requests
Requires manual or scheduled trigger to run a job
Charges by instance hour even when idle
Batch Transform
No persistent endpoint; spins up instances per job
Auto shuts down after job completes
Charges only for the duration of the job
Serverless Inference
No instance management or capacity planning
Max 200 concurrent requests per endpoint
Max 6 GB memory per request
Real-time Inference (auto scaled)
You choose instance type and count
Can handle thousands of concurrent requests
Supports large models up to instance memory limits
Multi-model Endpoint
Hosts multiple models behind one endpoint
Models loaded on demand from S3
Lower cost if models are used infrequently
Single-model Endpoint
Hosts one model per endpoint
Model always in memory for instant requests
Higher cost per model if many models are needed
Inference Pipeline (multi-container)
Chains multiple containers (preprocessing, model, postprocessing)
Each container runs in sequence
Requires custom container images for each step
Single Container Endpoint
Single container runs all logic (model + preprocessing)
Simpler to deploy and debug
Uses built-in framework containers when possible
Mistake
Deploying a model means you just upload the model file and it instantly works without any additional code.
Correct
You must provide inference code that tells SageMaker how to load the model, preprocess input, and postprocess output. The model file alone is not executable.
Many beginners come from environments like Jupyter notebooks where they run code interactively, and they assume a serialised model file contains the runtime logic. SageMaker needs explicit instructions because the production environment is stateless.
Mistake
Batch transform is always cheaper than a real-time endpoint for any workload.
Correct
Batch transform is cheaper only when you have occasional large jobs and don't need immediate results. For steady, low-latency traffic, a real-time endpoint with auto scaling can be cheaper because you don't pay for the spin-up time of batch instances.
People see that batch transform charges only for the duration of the job and think that's always optimal. They ignore that if you run batch jobs every hour, the cumulative cost of spinning up instances frequently can exceed a small real-time endpoint.
Mistake
Serverless inference is the best choice for all simple models because it requires no instance management.
Correct
Serverless inference has limits: up to 200 concurrent requests, max 6 GB memory, and no GPU support. It is only suitable for small models with sporadic, low-concurrency traffic.
AWS promotes serverless as simple, so beginners assume it's universally applicable. They don't read the quotas page or test with models that exceed memory limits, leading to deployment failures.
Mistake
You can host any number of models on a single endpoint without any configuration changes.
Correct
Multi-model endpoints require that each model has the same memory and inference code interface, and the client sends a header specifying which model to use. Not all models are compatible.
The name “multi-model” sounds like a magical bucket where you just dump models. In reality, SageMaker uses a shared memory cache and swaps model files in and out, so models must fit individually and must share the same container image.
Reveal each answer, then mark whether you got it right. Score 60%+ to unlock the next chapter.
Create a new endpoint configuration with the updated model, then use SageMaker's 'UpdateEndpoint' API to switch traffic to the new variant. You can also use blue/green deployment by creating a second endpoint and swapping DNS.
Yes, you can select GPU instance types like ml.p3.2xlarge or ml.g4dn.xlarge in your endpoint configuration. This is common for deep learning models that require GPU acceleration for low latency.
This usually means the model artifact or container image is not accessible. Check that the S3 bucket exists, the model file is correct, and the IAM role has permissions to read from S3 and use the container image from Amazon ECR.
The maximum payload size for a real-time endpoint is 6 MB for the request body and 6 MB for the response. Larger payloads should use batch transform or preprocess data to fit.
You can use the SageMaker Python SDK's LocalMode to simulate an endpoint on your local machine. Alternatively, run your inference code in a local Docker container that matches the SageMaker container image.
If your account has insufficient funds or is suspended, AWS will stop the endpoint after a grace period. Always set up billing alerts and consider using auto scaling to minimise costs when traffic is low.
You've finished Model Deployment and Inference on SageMaker. Continue through the MLS-C01 study guide to build a complete picture of the exam.
Done with this chapter?