If you ignore cost optimisation and performance tuning, your machine learning project will either run out of budget before it finishes or take so long it becomes irrelevant. This chapter will teach you how to find the sweet spot where your model trains fast enough without spending a fortune on cloud resources.
Jump to a section
A simple way to picture Cost Optimization and Performance Tuning for ML
A master sushi chef running a popular restaurant.
They face a constant puzzle: how to serve the freshest, most exquisite sushi to dozens of customers without wasting a single piece of fish, running out of ingredients mid-service, or spending too much on fish that spoils. This is the daily reality of cost optimisation and performance tuning in their world, just as it is in the world of machine learning (ML).
The chef has a fixed budget for sourcing fish. They must decide which fish to buy – tuna, salmon, sea bream – how much of each, and how often. If they over-order tuna and fewer customers want it, the fish goes bad and money is wasted. This is like an ML engineer choosing a compute resource (a powerful virtual machine, or instance, to run training). If they pick an oversized instance that sits idle, they pay for unused capacity. If they under-order, they run out of tuna during a rush, disappointing customers. That is like choosing an instance too small for the ML model, causing training to take forever or fail entirely.
The chef also adjusts cooking techniques. For a simple staple like rice, they use a reliable, standard method. But for a complex new dish like a toro tartare, they might experiment with different knives, temperatures, and timing – testing variations on a small batch first before making it for the whole menu. This mirrors a data scientist conducting small experiments (a hyperparameter sweep) on a cheap, small subset of data before committing to a full, expensive training run.
Finally, the chef tracks every ingredient's cost, every dish's popularity, and every customer's wait time. They use this data to refine the menu, discard unpopular items, and streamlines the prep workflow. This constant measurement and adjustment is the very heart of cost optimisation and performance tuning in ML – you must monitor, measure, and iterate to get the best results for the lowest cost.
Machine learning models are not just algorithms; they are programs that need serious computing power to train and run. Think of it like building a car: you need engines (computing power), fuel (data), and a factory (software infrastructure). Cost optimisation and performance tuning is the process of making sure you use the right engine size for each job, buy just enough fuel, and organise your factory so nothing slows you down.
To understand this, we first need to define a few key terms.
*Training* is the phase where the model learns from your data. It is the most expensive phase because it requires lots of computation, often using powerful graphics processing units (GPUs). A GPU is a specialised chip originally designed for video games that is also excellent at performing the many parallel calculations that ML needs.
*Inference* is the phase where the trained model makes predictions on new, unseen data. This can be less compute-intensive than training, but if you have millions of users, the cost of each tiny prediction adds up.
*Compute resources* are the virtual machines (VMs) you rent from a cloud provider like Amazon Web Services (AWS). A VM is an emulated computer running in a data centre. You pay by the hour, so stopping a VM when not in use saves money.
*Storage* is where you keep your data and your trained model files. Different storage types have different speeds and costs. For example, an Amazon Simple Storage Service (S3) standard bucket is cheap for storing large datasets, while Amazon Elastic Block Store (EBS) is faster but costs more, and is attached directly to your compute instance.
Now, how do you actually optimise cost and performance? The first big idea is *right-sizing*. This means choosing compute resources that are just powerful enough for the job. If your model training fits easily into a smaller instance (like a t3.medium with 2 virtual CPUs and 4GB of RAM), using a massive GPU instance (like a p3.2xlarge with 8 GPUs) wastes money and time. Many beginners assume bigger is always better. It is not. Start small, test, then scale up only if you must.
The second idea is *elasticity*. Cloud resources can be turned on and off on demand. During active training, you need the power. During experimentation and data preparation, you likely do not. If you leave training instances running overnight or over the weekend when no one is using them, you burn cash. So you *must* set a schedule to stop them automatically, or use services like SageMaker that stop on completion.
A third technique is using *spot instances*. These are spare compute capacity that the cloud provider sells at a deep discount (up to 90% off the on-demand price). The catch is the provider can reclaim that resource with just a two-minute warning. Therefore, you should only use spot instances for *fault-tolerant* workloads. For example, training a model that regularly saves its progress (checkpoints) can survive being interrupted: when the spot instance is taken back, you simply restart from the last checkpoint on another spot instance. Do not use spot instances for critical, long-running, or non-interruptible tasks.
Fourth, *storage tiering* matters. Data that you access frequently should be on fast, more expensive storage (like S3 Standard). Data that you access rarely (like old training data) can be moved to cheaper tiers (like S3 Glacier) where retrieval takes minutes to hours. This saves significant money.
Finally, *performance tuning* involves optimising the data pipeline. If the model is waiting for data to load (I/O bound), it is wasting expensive compute time. You can pre-process data before training or use a faster data format like Parquet, which is a columnar storage format that allows your system to read only the columns it needs. You can also use multi-threading to load data in parallel.
All of these techniques work together. You cannot just optimise one piece and ignore the others. The cloud bill is the sum of all your choices: instance type, running hours, storage tier, and how efficiently your code uses the resources. The goal is to deliver a model that meets performance targets (e.g., under 1 second inference time, or 95% accuracy) while spending the least possible amount of money.
1. Estimate and Plan
Before spending any money, estimate the cost of a single training run using a calculator (like AWS Pricing Calculator). Determine the smallest feasible instance, estimate runtime, and set a budget for the project. This step prevents surprises.
2. Start with a Small Experiment
Test the model on a tiny subset of data (e.g., 1%) using a cheap non-GPU instance. This verifies the code and hyperparameters without burning GPU money. It is much cheaper to debug a small job.
3. Right-size and Choose Instance Type
Select an instance that matches the workload – GPU for heavy deep learning, compute-optimised for CPU-bound tasks. Use monitoring to ensure utilisation is between 60-80% to avoid overpaying.
4. Enable Spot Instances and Checkpointing
Configure the training job to use spot instances (for fault tolerance). Implement checkpointing to save the model state every few minutes to S3. This allows the job to resume if the spot instance is reclaimed.
5. Optimise the Data Pipeline
Convert raw data to a fast, efficient format like Parquet. Use batch reading and multi-threaded data loading to prevent the compute from waiting idle on data. This reduces overall training time without increasing instance cost.
6. Monitor and Automate
Set up CloudWatch alerts for instance utilisation and budget limits. Automate stopping idle instances using a cron job or SageMaker's auto-stop feature. Review the cloud bill weekly to catch waste early.
7. Plan for Deployment and Inference Costs
Once the model is trained, evaluate inference costs. For low-traffic inference, use cheaper options like serverless inference (e.g., SageMaker Serverless) or Elastic Inference. For high-traffic, use a well-sized endpoint with auto-scaling to handle traffic peaks without over-provisioning.
A data scientist at an e-commerce company is building a product recommendation model. The dataset has 10 million user-item interactions, and the training job will take about 8 hours on a baseline GPU instance. The company has a strict monthly cloud budget.
The first step the data scientist takes is to *estimate the cost* using the AWS Pricing Calculator. They know that on-demand GPU instances can cost over $3 per hour. Eight hours of a single p2.xlarge would run about $24 per training run. If they run this job 10 times during the development phase, that is $240 – manageable, but they want to reduce it further.
Then they apply *right-sizing*. They start by training a small version of the model (with 1% of the data) on a cheaper, non-GPU instance like a c5.2xlarge to validate the code and hyperparameters. This initial step costs only $0.34 per hour and takes 20 minutes. Once the code is solid, they scale up to the GPU instance for the final, full training.
Next, they enable *spot instances* on the training job. Because they have trained the model with checkpointing (saving the model every 15 minutes), if the spot instance is interrupted, they can resume automatically from the last checkpoint. This reduces the compute cost from $3/hour to about $0.45/hour – a 85% saving.
For storage, they keep the raw data in S3 Standard (frequent access) but move the older, processed versions of the data to S3 Glacier Deep Archive after the final model is deployed. This saves about 90% on storage costs for historical data.
Finally, they tune the data pipeline. They convert the raw CSV files into the Parquet format. This reduces the file size by about 70% and allows the training job to read 30% faster. This means the model spends less time idle waiting for data, effectively reducing the overall training time from 8 hours to 7 hours.
In the real world, the IT professional does not just set and forget. They set up *cloud monitoring* (like AWS CloudWatch) to track instance CPU, memory, and GPU utilisation. If utilisation is below 50%, they right-size down to a smaller instance. They also set *budget alerts* so that if the monthly spend is projected to exceed the limit, they get a warning email.
They also practise *experimentation with managed services*. Instead of manually managing EC2 instances, they might use Amazon SageMaker, which automatically stops training instances after the job is finished and can automatically use spot instances. This reduces the chance of leaving resources running by accident.
The result: the model is trained and costs only 30% of the original budget, while performance meets the business requirement of making recommendations in under 500 milliseconds.
MLS-C01 tests your understanding of practical cost and performance optimisation strategies, not just theory. Expect about 5-8 questions on this topic across the exam. They love to test your ability to choose the *most cost-effective* solution that still meets performance requirements.
You will see scenario-based questions where you are given a situation (a data scientist training a model with a certain dataset size) and four options. The trap is usually an option that suggests a very powerful instance (like a p3dn.24xlarge) when a smaller instance would work, or an option that saves cost but breaks the model's requirements (e.g., using spot instances for a job that cannot tolerate interruption).
Key concepts they test:
*Spot Instances*: You must know that they are suitable for fault-tolerant, non-time-sensitive training jobs, but not for inference or production workloads that need reliable uptime.
*Right-sizing*: They ask why you should start with a smaller instance (cost and faster debugging) before scaling up.
*Managed Services*: They test the benefits of using Amazon SageMaker (automatic scaling, managed spot training, built-in model optimisation) versus managing EC2 instances directly. SageMaker saves time and reduces risk of errors.
*Data Pipeline Optimisation*: They test whether you know to use S3 for storage, and to use faster data formats (Parquet) and batch your data to reduce I/O wait.
*Storage Tiers*: They will ask about the cost of storing data on S3 Standard vs S3 Infrequent Access vs S3 Glacier. A common trap is suggesting S3 Standard for old, rarely accessed training data (wasteful). The correct answer is to move it to a lower cost tier.
*Checkpointing*: This is key for spot instance recovery. If you do not checkpoint, you cannot resume after interruption. They may test whether you know to save checkpoints to S3 or to EBS. Answer: you should save to S3 (central, durable) because EBS is tied to a single instance and might be lost.
*Elastic Inference*: This is a service (AI) that lets you attach just a small amount of GPU inference acceleration to an instance instead of buying a full GPU instance. It saves money when you have low-volume inference. Know that it is for *inference*, not training.
*Compute Optimised vs GPU Instances*: They test your ability to choose the right instance family. Compute Optimised (C5, C6g) for CPU-bound jobs; GPU (P3, P4) for deep learning training; Memory Optimised (R5) for large in-memory datasets. The wrong choice wastes money.
Trap patterns:
*Choosing a managed service blindly*: Sometimes, for very simple, short running experiments, spinning up a single EC2 instance costs less than the overhead of SageMaker. Learn to recognise when to use each.
*Ignoring data transfer costs*: Moving large datasets between regions costs money. They might ask about this in a multi-region setup. The cheapest way is to process data in the same region where it is stored.
*Over-tuning performance at the cost of dev time*: They might give a scenario where you could manually tune the data pipeline for hours, but the AI service (like SageMaker) already does this well. The correct answer is to use the service.
To memorise: remember the acronym *R.E.C.I.P.E.*: Right-size, Elasticity, Checkpointing, Instance choice (Spot vs On-demand), Pipeline optimisation (format/batch), and Evaluate with monitoring.
Right-sizing your compute instance is the single biggest factor in reducing ML costs: always start small and scale up based on actual utilisation.
Spot instances can save up to 90% on training costs, but only use them with checkpointing and only for non-critical, interruptible workloads.
Storage costs can dwarf compute costs over time; use S3 lifecycle policies to move old data to cheaper tiers like Glacier automatically.
The data pipeline is a common performance bottleneck; use columnar formats (Parquet) and multi-threaded data loading to reduce idle compute time.
Managed services like Amazon SageMaker include automatic cost-saving features (spot training, auto-stop) that you should use rather than managing raw EC2 instances.
Always set budget alerts and monitor resource utilisation with tools like CloudWatch to catch waste early, not at the end of the month.
These come up on the exam all the time. Here's how to tell them apart.
Spot Instances
Up to 90% cheaper than on-demand.
Can be terminated by AWS with a 2-minute notice.
Ideal for fault-tolerant training jobs with checkpointing.
On-Demand Instances
Standard, predictable cost.
No risk of termination; always available.
Good for production, inference, and time-sensitive workloads.
S3 Standard Storage
Low latency (milliseconds) retrieval.
Higher cost per GB per month.
Suitable for actively used training data.
S3 Glacier Deep Archive
Very low cost per GB (up to 90% less than Standard).
Retrieval times are minutes to hours.
Suitable for archived, rarely accessed historical data.
SageMaker Training
Automatically stops instance after training.
Built-in spot instance management.
More expensive per hour due to service overhead for very simple jobs.
Manual EC2 Training
Full control over configuration and security.
Can be cheaper for very simple, short jobs (no service overhead).
Requires manual cleanup; risk of leaving instance running.
GPU Instance (p3.2xlarge)
Has powerful NVIDIA V100 GPU (8GB).
Excellent for deep learning matrix operations.
Cost is high (~$3+/hour).
Compute Optimised Instance (c5.2xlarge)
No GPU; uses Intel Xeon CPUs.
Better for data preprocessing, traditional ML (gradient boosting).
Much cheaper (~$0.34/hour).
Mistake
The biggest GPU instance always trains the fastest and is the best choice for any ML workload.
Correct
Using a massively oversized GPU instance often results in under-utilisation (wasted money) and can even be slower if the bottleneck is data loading, not computation. Right-sizing is crucial.
Beginners see 'GPU is fast' and assume more is better, not realising that other parts of the system (data loading, memory) can become the bottleneck.
Mistake
Spot instances are free money – I should use them for everything including production inference.
Correct
Spot instances are unreliable (can be terminated in 2 minutes). They are only safe for fault-tolerant training with checkpoints, not for production inference requiring 100% uptime.
The deep discount (up to 90%) makes them seem like an obvious choice, but beginners often ignore the 'can be reclaimed at any time' condition.
Mistake
Storing my training data on a fast, attached EBS volume is always the cheapest option because it's attached to the instance.
Correct
EBS volumes cost per GB provisioned whether you use them or not. For large datasets, using S3 (cheaper per GB, with lifecycle policies) and streaming data during training is more cost-effective.
The term 'attached' feels free, but you pay for any provisioned EBS storage even if the instance is off. S3 is pay-what-you-use.
Mistake
If I use Amazon SageMaker, I cannot control costs because it's a managed service.
Correct
SageMaker actually helps control costs by automatically stopping idle instances, supporting spot training, and allowing you to right-size the instance choice before launching the job.
Many beginners equate 'managed' with 'less control', not realising that managed services often include built-in cost-savings features.
Mistake
Performance tuning means buying a faster instance. That's the only way to speed up training.
Correct
Often, tuning the data pipeline (e.g., using Parquet, batching, parallel loading) has a bigger impact on speed than upgrading the instance, and costs nothing extra.
It is easy to think the problem is always 'needs more compute', when in reality the compute is waiting for data most of the time.
Reveal each answer, then mark whether you got it right. Score 60%+ to unlock the next chapter.
The cheapest way is to use spot instances with checkpointing on an appropriately sized GPU instance through SageMaker, and to optimise your data pipeline to reduce training time. Start with a small experiment to validate before full training.
Use GPU instances for deep learning models (neural networks) with large datasets, especially for large matrix operations. Use CPU instances for simpler models like linear regression, decision trees, or when the dataset is small and fits in memory.
On-demand: pay per hour with full control. Spot: up to 90% cheaper but can be stopped instantly. Reserved: commit to 1-3 years for a discount (up to 72%). For ML training, prefer spot if you can checkpoint; use on-demand for experiments; reserved for consistently running production inference endpoints.
Right-size your instance, use spot training with checkpointing, move old data to cheaper storage tiers, optimise your data pipeline (fast format, batch loading), and set budget alerts to prevent accidental runaway spending.
Amazon SageMaker is a managed ML platform that automates many cost-savings: it can automatically stop instances after training, use spot instances for training, and allows you to select instance types per job. Yes, it saves money by reducing human error and idle time.
Use CloudWatch metrics like CPUUtilisation, MemoryUtilisation, and GPUUtilisation. If these metrics stay consistently below 40%, you are paying for capacity you are not using and should right-size down to a smaller instance.
You've finished Cost Optimization and Performance Tuning for ML. Continue through the MLS-C01 study guide to build a complete picture of the exam.
Done with this chapter?