Cost optimisation and scaling for machine learning (ML) workloads is the practice of spending exactly the right amount of money on computers and storage to train and run ML models — no more, no less. For the MLA-C01 exam, this matters because Amazon Web Services (AWS) charges you per second for the resources you use, and the difference between a well-optimised and a wasteful setup can be thousands of pounds per month. You need to know how to choose the cheapest option that still gets your ML job done on time.
Jump to a section
A simple way to picture Cost Optimization and Scaling for ML Workloads
You run a 5-star restaurant that serves 10,000 meals a day. Your kitchen has 4 stoves, each costing £100 an hour to run. During the lunch rush, all 4 stoves are blasting — you're cooking full capacity. But from 2 PM to 5 PM, only 1 stove is simmering a single stockpot. You're still paying for 4 stoves. That's wasted cost — you're over-provisioned.
Now imagine a smart kitchen manager who switches off unused stoves the second the lunch rush ends. When dinner begins, they fire stoves back up instantly, one by one, as orders arrive. They also keep an eye on the prep counter — if too many orders pile up, they add a portable induction hob for just 30 minutes, then store it away. They never buy a permanent 5th stove for a temporary spike.
Finally, they compare takeaway orders vs dine-in orders. Takeaway meals use less oven time, so they shift some dishes to cheaper countertop grills during takeaway peaks. Every decision — when to turn burners on, which appliance to use, whether to buy permanent gear or rent portable hobs — directly affects the nightly profit. This is exactly how cost optimisation and scaling works for machine learning workloads: you match compute resources precisely to demand, use the right tool for each job, and never pay for idle capacity.
To understand cost optimisation and scaling for ML workloads, you first need to know what a "workload" is. A workload is any task a computer performs — in ML, that includes training (teaching a model using data) and inference (using a trained model to make predictions on new data). Each workload consumes compute resources: central processing units (CPUs), which are general-purpose brains; graphics processing units (GPUs), which are specialised brains great at maths; and memory (RAM), which is short-term storage for active work. AWS lets you rent these resources by the second through services like Amazon Elastic Compute Cloud (Amazon EC2), which gives you virtual servers called instances.
The core principle of cost optimisation is this: only use as much resource as you need, for only as long as you need it. AWS offers four main levers to achieve this. First is right-sizing: picking an instance type (a specific combination of CPU, GPU, and memory) that fits your job. If you buy a massive GPU instance for a tiny text-processing job, you are paying for capacity you never use. Second is elastic scaling: automatically adding or removing instances as demand changes. AWS services like Amazon SageMaker can launch 100 training instances at midnight and shrink back to zero at 6 AM — you only pay for those six hours.
Third is spot instances. Spot instances are spare compute capacity that AWS sells at a huge discount — up to 90% off — because the cloud provider has more servers than customers need at that moment. The trade-off is that AWS can reclaim (interrupt) a spot instance with only two minutes' warning if another customer needs it. For ML training, which can often be paused and resumed (fault-tolerant), spot instances are perfect. Fourth is managed services. Instead of setting up and managing your own servers, you use a service like Amazon SageMaker, which handles all the infrastructure for you. AWS charges a small premium for this convenience, but you save money on the staff time needed to manage servers yourself.
Scaling is about adjusting capacity to match demand. There are two flavours: vertical scaling (making a single instance bigger — like upgrading from a 2-core CPU to a 32-core CPU) and horizontal scaling (adding more instances side-by-side — like going from 1 server to 10 servers). For ML, horizontal scaling is more common because training can be split across many machines using a technique called distributed training.
Why does all this exist? In the old days (and still in many on-premises data centres), companies bought fixed hardware that could handle their peak workload. If the peak lasted one hour a day, the other 23 hours of equipment was idle. The cloud changed this by letting you treat compute as a utility — like electricity — where you pay only for what you use. AWS's pay-as-you-go model means you can scale up to handle a huge training job, then scale down to zero when done.
A real example: Suppose you need to train a model on 100 GB of images. A single GPU instance might take 20 hours at $10 per hour = $200. But you could split the data across 10 spot instances, each costing $1 per hour (90% discount), and finish in 2 hours = $20 total. That's a 90% saving. The key is that the training job must be designed to handle interruptions — for example, by saving progress every few minutes (checkpointing) so that if a spot instance is reclaimed, you restart from the last save, not from scratch.
AWS provides a suite of services to help: Amazon SageMaker for managed training and inference, AWS Auto Scaling to automatically adjust the number of instances, AWS Cost Explorer to track spending, and AWS Budgets to set alerts when costs exceed a threshold. For inference, you can also use Amazon Elastic Inference, which lets you attach a small GPU accelerator to a cheap CPU instance rather than buying a full GPU instance — cutting cost when the model doesn't need full GPU power.
In summary, cost optimisation and scaling are about matching resources to demand at the lowest possible price. The exam expects you to know the trade-offs: spot vs on-demand, managed vs unmanaged, vertical vs horizontal, and when each is appropriate. You must also understand that cost optimisation is a continuous process — you monitor, analyse, and adjust because workloads and pricing change over time.
Analyse the workload profile
Determine whether your ML task is training or inference, whether it can be interrupted (fault-tolerant), how long it takes, and whether demand fluctuates. This decides whether spot instances, on-demand, or reserved capacity is appropriate.
Right-size the instance
Select an Amazon EC2 or SageMaker instance type that matches your job's compute, memory, and GPU requirements. Use AWS Compute Optimizer to get recommendations based on past usage, or start small and scale up gradually (vertical scaling) to find the sweet spot.
Choose the pricing model
For fault-tolerant training, enable managed spot training in SageMaker or use spot instances in EC2. For steady-state inference, consider Reserved Instances or Savings Plans. For unpredictable workloads, stick with on-demand. Set a maximum budget to prevent runaway costs.
Configure auto scaling
Set up AWS Auto Scaling for inference endpoints to add instances when demand rises and remove them when it falls. Define minimum and maximum instance counts and a target metric (e.g., average CPU utilisation at 60%). Test scaling behaviour to ensure it responds fast enough for your application's latency requirements.
Monitor and optimise continuously
Use AWS Cost Explorer to view daily and weekly ML spending trends. Set AWS Budgets alerts to notify you if costs exceed a threshold. Review SageMaker notebooks and endpoints for idle resources. Adjust instance types or scaling policies based on observed usage patterns every few weeks.
An IT professional — let's call her Priya — is a machine learning engineer at an e-commerce company that recommends products to 2 million daily visitors. Priya's team has just built a new recommendation model that needs to be trained on 5 terabytes of purchase history data every week. She also needs to serve predictions to users in real time — every time a shopper loads a webpage, the model must return personalised product suggestions within 200 milliseconds.
Priya starts with training. She uses Amazon SageMaker, a managed service, to avoid manually setting up servers. She creates a training job and configures it to use ml.p3.2xlarge instances — these have one NVIDIA V100 GPU each. She estimates the job will take 10 hours on 20 instances. The cost would be high if she used on-demand instances ($3.06 per instance per hour). Instead, she enables managed spot training inside SageMaker. SageMaker automatically requests spot instances and handles interruptions by checkpointing the model every 15 minutes. This brings the cost down to about $0.30 per instance per hour — a 90% saving.
Next, inference. Priya's model needs to handle 2,000 requests per second during peak hours (evenings and weekends) but only 200 requests per second during off-peak hours (early mornings). If she ran 50 full GPU instances all day, she would waste money. Instead, she uses SageMaker real-time inference endpoints with automatic scaling. She configures a minimum of 5 instances and a maximum of 50. She also switches the endpoint to use ml.c5.large instances — CPU-only — because her model is a gradient-boosted tree, not a deep neural network, so it doesn't need a GPU. This swap alone cuts per-instance cost from $3.06 to $0.085 per hour.
To further optimise, Priya sets up a lifecycle configuration: from midnight to 6 AM, the endpoint scales down to the minimum of 5 instances because traffic is low. She also creates an AWS Budget alert that emails her if daily inference costs exceed $50. Once a week, she reviews AWS Cost Explorer reports to see if any unused resources remain — for example, a development environment accidentally left running over the weekend.
Priya's actions step-by-step:
Right-sizing: choosing CPU instances for a non-GPU model.
Using spot instances for fault-tolerant training.
Auto scaling inference endpoints to match demand.
Using managed services (SageMaker) to reduce operational overhead.
Monitoring with Cost Explorer and Budgets to catch waste.
The result: Priya's team slashes monthly ML costs from $12,000 to $2,500 — a 79% reduction — while still delivering predictions under 200 milliseconds. She documents this setup in a runbook (a set of instructions for repeatable tasks) so new team members can replicate it. This is exactly what the MLA-C01 exam expects you to describe: a practical, cost-aware approach to ML workloads on AWS.
The MLA-C01 exam tests your ability to choose the most cost-effective compute option for a given ML scenario while meeting performance requirements. You will see scenario-based multiple-choice questions that describe a workload — for example, "a data science team needs to train a model on 10 TB of data every night. The training can be interrupted and resumed. What is the most cost-effective compute option?" The correct answer will involve spot instances. A common trap is picking on-demand instances because they are "reliable" — but the question specifies the training can be interrupted, making spot the cheaper choice.
Key exam topics:
Spot instances: when to use them (fault-tolerant workloads like training, batch inference, hyperparameter tuning) and when NOT to use them (real-time inference, where a sudden interruption would cause a user-facing error).
On-demand vs Reserved vs Spot vs Dedicated Hosts pricing models. Reserved instances give a discount (up to 72%) in exchange for a 1- or 3-year commitment — use for steady-state workloads. Dedicated Hosts are for compliance scenarios (e.g., licensing restrictions) and are the most expensive.
Amazon SageMaker managed spot training: how it automatically checkpoints and restarts.
Elastic Inference: attaching a fractional GPU to a CPU instance for inference, cheaper than a full GPU instance.
AWS Auto Scaling: setting target tracking (e.g., keep CPU utilisation at 60%) to automatically adjust instance count.
AWS Compute Optimizer: a service that analyses your usage and recommends right-sized instance types.
Savings Plans: commit to a dollar amount of spend per hour for 1 or 3 years to get lower prices (similar to Reserved Instances but more flexible).
Trap patterns to watch for:
The exam will offer a shiny managed service (like SageMaker) even when a cheaper unmanaged option (like setting up your own EC2 instance) is available. But the question might specify "the team has no DevOps engineers" — that makes the managed service correct despite higher compute cost, because staff time is also a cost.
They might describe a workload that is "very sporadic" — every few minutes, one request comes in. The cheapest option is AWS Lambda (serverless functions) not a constantly running instance.
They might ask about inference latency requirements. If the requirement is "under 10 milliseconds", a cheap CPU instance might not satisfy it, and a GPU instance becomes necessary — even if cheaper options exist.
They might mix up vertical and horizontal scaling: horizontal scaling always involves adding more instances, not making a single instance bigger.
Key definitions to memorise:
Spot instance: spare compute capacity sold at up to 90% discount, can be reclaimed with 2-minute notice.
On-demand instance: pay per second with no commitment, highest cost.
Reserved instance: commit to 1 or 3 years, up to 72% discount.
Savings Plan: commit to a consistent amount of compute usage (measured in $/hour) for 1 or 3 years.
Auto Scaling: automatically adjusts compute capacity based on demand.
Elastic Inference: GPU acceleration for inference without a full GPU instance.
Finally, expect at least one question about cost monitoring tools: AWS Cost Explorer (for historical analysis), AWS Budgets (for alerts), and Trusted Advisor (for cost optimisation recommendations). You must know which tool does what.
Spot instances can reduce ML training costs by up to 90%, but should only be used for workloads that can handle interruptions through checkpointing.
Right-sizing your instance means matching CPU, GPU, and memory to your specific job — never pay for unused capacity.
Horizontal scaling with many small spot instances is typically cheaper for training than vertical scaling with one very large instance.
Amazon SageMaker managed spot training automatically handles checkpointing and restarts, making it the easiest way to use spot instances for training.
For inference, always consider whether GPU acceleration is necessary — many models run just fine on cheaper CPU instances, and Elastic Inference offers a middle ground.
AWS Cost Explorer, Budgets, and Trusted Advisor are the three main tools for monitoring and controlling ML workload costs.
Reserved Instances and Savings Plans provide discounts in exchange for commitment, but differ in flexibility — Savings Plans are more versatile.
These come up on the exam all the time. Here's how to tell them apart.
Spot Instances
Up to 90% discount compared to on-demand
Can be interrupted with 2-minute notice
Best for fault-tolerant training and batch jobs
On-Demand Instances
Full price, no discount
Guaranteed availability — never interrupted
Best for real-time inference and critical workloads
Vertical Scaling
Upgrade to a larger single instance (more CPU/GPU/RAM)
Limited by maximum instance size available
Simpler to set up — no distributed computing needed
Horizontal Scaling
Add more instances to work in parallel
Limited only by budget and workload parallelism
Requires distributed training or load balancing
Reserved Instances
Locks you to a specific instance family and region
1 or 3 year commitment with up to 72% discount
Best for predictable, steady-state workloads
Savings Plans
Flexible — applies to any instance within a region (Compute) or family (EC2 Instance)
1 or 3 year commitment with similar discounts
Best for workloads with some variability in instance types
Managed Service (SageMaker)
AWS handles provisioning, patching, and scaling
Higher per-hour compute cost but lower operational cost
Built-in spot training and auto scaling
Unmanaged Service (EC2)
You manage everything — OS, drivers, scaling script
Lower per-hour compute cost but higher staff time
More control over custom configurations
Mistake
Using spot instances always saves the most money, so they should be used for every ML workload.
Correct
Spot instances are only suitable for workloads that can tolerate interruptions, like training with checkpointing. For real-time inference where a drop in service would affect users, on-demand or reserved instances are required.
Beginners see the huge discount (up to 90%) and assume it is universally better, not realising the trade-off of potential interruption.
Mistake
Vertical scaling (making a single instance bigger) is cheaper than horizontal scaling (adding more instances) for ML training.
Correct
Horizontal scaling is often cheaper because you can use smaller, cheaper spot instances and parallelise the work. Very large instances have fewer availability options and can be cost-prohibitive.
People from non-tech backgrounds think big machines are always more efficient, but cloud pricing penalises oversized instances disproportionately.
Mistake
If you use a managed service like Amazon SageMaker, you don't need to think about cost optimisation at all — AWS handles it for you.
Correct
Managed services help with operational overhead, but you still choose instance types, scaling policies, and whether to use spot instances. A poorly configured SageMaker job can still waste money.
The name 'managed' implies hands-off, but cost optimisation remains the customer's responsibility under the AWS Shared Responsibility Model.
Mistake
Reserved Instances and Savings Plans are the same thing — both are just prepaying for compute.
Correct
Reserved Instances lock you into a specific instance family (e.g., m5.large) and region, while Savings Plans give you flexibility across any instance in a region (Compute Savings Plan) or within a family (EC2 Instance Savings Plan).
The similar discount percentages cause confusion, but the flexibility difference is crucial for exam questions that ask about changing instance types.
Mistake
You should always run your inference endpoint at maximum capacity to handle the highest possible traffic spike.
Correct
You should use Auto Scaling to match capacity to actual demand, keeping a buffer for spikes but scaling down when traffic is low. Running at peak capacity permanently wastes money.
Beginners over-provision out of fear of performance issues, not understanding that Auto Scaling can react in minutes.
Reveal each answer, then mark whether you got it right. Score 60%+ to unlock the next chapter.
A spot instance is spare AWS compute capacity sold at a discount of up to 90%. You request it like an on-demand instance, but AWS can reclaim it with a two-minute warning if demand returns. It saves money because you only pay the low, dynamically set spot price.
No — spot instances can be interrupted at any time, which would cause your inference endpoint to fail for users. For real-time inference, use on-demand or reserved instances. Spot instances are ideal for training, batch processing, and hyperparameter tuning.
Vertical scaling (scaling up) means upgrading to a more powerful instance with more CPU, GPU, or memory. Horizontal scaling (scaling out) means adding multiple instances to work in parallel. For ML training, horizontal scaling with distributed training is usually cheaper and faster.
If your model is a deep neural network with many layers (e.g., for image or language processing), it likely benefits from a GPU. Simpler models like linear regression or decision trees run fine on CPUs. Test latency on a CPU instance first — only add GPU if the response time is too slow.
A Savings Plan is a flexible discount model where you commit to spending a certain amount per hour (e.g., $10/hour) for 1 or 3 years. In return, AWS gives you lower prices on any eligible compute usage within a region (Compute Savings Plan) or within a specific instance family (EC2 Instance Savings Plan).
SageMaker automates infrastructure management, including auto scaling of inference endpoints, managed spot training (with automatic checkpointing), and lifecycle configurations that stop idle notebooks. This reduces both direct compute waste and the staff time needed to manage servers.
AWS is responsible for the cost of the physical infrastructure (data centres, networking). The customer (you) is responsible for choosing which resources to use, configuring scaling, and turning off idle resources. AWS provides tools to help, but cost control is ultimately your job.
You've finished Cost Optimization and Scaling for ML Workloads. Continue through the MLA-C01 study guide to build a complete picture of the exam.
Done with this chapter?