Training, tuning, and evaluating models is the core workflow of machine learning on AWS. Without these three steps, you cannot build a model that reliably solves real problems or pass the MLS-C01 exam because they form the backbone of every ML pipeline you will design.
Jump to a section
A simple way to picture Training, Tuning, and Evaluating Models
A master baker's cake is the result of a precise three-stage process.
First, the baker gathers ingredients—flour, sugar, eggs, butter—and follows a recipe to mix and bake a base cake. This is the training phase: the baker is learning how the ingredients combine to form the cake's structure. The baker does not know yet if the cake will taste good; they are just creating the initial product from the recipe.
Second, the baker tastes the cake and makes adjustments. Too dry? Add more butter or reduce baking time. Too sweet? Cut some sugar. The baker might try a different oven temperature or a different type of flour. This is the tuning phase: they are changing the parameters (ingredients and settings) to make the cake better.
Third, the baker invites a small group of tasters to evaluate the final cake. The tasters score it on texture, flavour, and appearance. The baker uses this feedback to decide if the cake is ready for the market or needs more work. This is the evaluation phase: testing the final product with new data (the tasters' opinions) to see if it truly works.
If the baker skipped any step—if they never tasted the cake, or if they only tasted the same batter they used to bake—the cake might fail. A machine learning model goes through the same disciplined process: train the model on data, tune it by adjusting settings, then evaluate it with fresh data to confirm it performs well.
In machine learning, a model is a mathematical pattern that learns from data. Think of it as a formula that takes input (like a house's size and location) and predicts an output (like its price). But the model does not start knowing the right formula. It must be trained, tuned, and evaluated. These three phases are separate, sequential, and absolutely critical.
Training is the first phase. You give the model a large dataset—called training data—that contains many examples of inputs and their correct outputs (labels). For example, if you want to predict house prices, your training data might have 10,000 rows. Each row has the house size, number of bedrooms, location, and the actual selling price (the label). The model looks at all these examples and adjusts its internal mathematical weights (numbers that define the pattern) to minimise the difference between its predictions and the true labels. The goal is for the model to learn the underlying relationships in the data. AWS services like Amazon SageMaker provide built-in algorithms (pre-written mathematical recipes) that automate this training process. You simply point SageMaker at your training data, choose an algorithm, and specify how long to train. The service runs the heavy computation on powerful machines.
Once the model is trained, you need to tune it. Tuning means adjusting hyperparameters. Hyperparameters are settings that control how the model learns—they are not learned from the data themselves. Examples include the learning rate (how big a step the model takes when adjusting its weights) or the number of trees in a random forest algorithm. Choosing the right hyperparameters can dramatically improve the model's accuracy. The manual trial-and-error method of trying random hyperparameters is slow and inefficient. AWS SageMaker offers Automatic Model Tuning (also called hyperparameter tuning). You tell SageMaker a range of values for each hyperparameter, and it automatically runs many training jobs, each with different combinations, to find the set that gives the best performance on a validation dataset (a separate chunk of data the model has never seen during training).
Evaluation is the final phase. You take the trained and tuned model and test it on data it has never seen before—called test data. This is the only honest way to know if the model will work in the real world. If you test the model on data it already saw, you get an overly optimistic score (this is called overfitting—the model memorised the training data but cannot generalise to new cases). AWS SageMaker provides evaluation tools like confusion matrices (a table showing correct and incorrect predictions) and binary classification metrics (like accuracy, precision, recall, and F1-score). For a regression model (predicting a number, like price), you look at Mean Squared Error (MSE) or Root Mean Squared Error (RMSE)—both are measures of average prediction error. For a classification model (predicting a category, like spam or not spam), you look at accuracy (percentage correct) but also precision (how many of your positive predictions were right) and recall (how many of the actual positives you caught). The exam loves to test these metrics.
Why does this three-phase process exist? Before modern ML, people tried to write explicit rules for every situation (like 'if the house is over 2,000 square feet and has a garage, price is £300,000'). That failed because real-world patterns are too complex to hand-code. Training automates the discovery of those patterns. Tuning optimises the discovery process. Evaluation validates that the discovered pattern is real and not a fluke. AWS makes each phase scalable and repeatable, so you can build models without managing servers.
In summary:
Training = learning patterns from data.
Tuning = adjusting the learning settings for best performance.
Evaluation = testing the final model on new data to confirm it works.
The MLS-C01 exam expects you to know which AWS service or feature handles each phase, how to spot overfitting, and how to interpret evaluation metrics correctly.
Prepare the Data
Clean the data (handle missing values, correct errors) and split it into training (70%), validation (15%), and test (15%) datasets. This split ensures you have separate data for learning, tuning, and honest evaluation.
Choose an Algorithm
Select a machine learning algorithm that fits your problem type—for example, XGBoost for classification, Linear Learner for regression. SageMaker has built-in algorithms you can use without writing code from scratch.
Train the Model
Point SageMaker to your training data in S3, choose the algorithm, and start a training job. SageMaker spins up computing resources, runs the algorithm, and outputs a trained model artifact saved to S3.
Tune Hyperparameters
Use SageMaker Automatic Model Tuning to define a range for each hyperparameter (e.g., learning rate from 0.01 to 0.1). SageMaker runs multiple training jobs with different combinations and picks the best one based on performance on the validation set.
Evaluate the Final Model
Use the test dataset (never seen before) to generate predictions via SageMaker batch transform or a hosted endpoint. Compare predictions to true labels and compute metrics like accuracy, precision, recall, or RMSE to decide if the model is ready for production.
Imagine you work for a retail company that wants to predict which customers will churn (cancel their subscription) next month. Your boss asks you to build a machine learning model. You have a dataset of 50,000 customers with features like number of logins, support tickets, months as a customer, and a label: churned (yes or no).
First, you split the dataset into three parts: 70% for training, 15% for validation (used during tuning), and 15% for test (used only for final evaluation). You upload the training data to an Amazon S3 bucket (a storage service). Then, in SageMaker, you select a built-in algorithm, such as XGBoost (a popular algorithm for classification). You launch a training job, specifying the S3 path, the algorithm, and the instance type (the virtual server that does the computation). SageMaker spins up an EC2 instance, runs the training, and produces a model artifact (the learned pattern) saved to S3.
Next, you tune the model. You set up Automatic Model Tuning. You tell SageMaker to try learning rates between 0.01 and 0.1, and tree counts between 50 and 200. SageMaker runs 15 training jobs in parallel, each with a different combination. It compares their performance on the validation dataset (the 15% you set aside). After a few hours, it tells you the best hyperparameters: learning rate = 0.05, trees = 120, with a validation accuracy of 82%.
Now you evaluate. You take that best model and run a batch transform job (SageMaker's way to make predictions on a big file) using the test dataset (the remaining 15%). The job outputs predictions for each customer. You compare those predictions to the actual churn labels in the test data. You compute the confusion matrix and find:
Accuracy: 83%
Precision: 78% (meaning 78% of customers flagged as churners actually churned)
Recall: 71% (meaning you caught 71% of all actual churners)
Your boss wants to identify churners so the sales team can call them. You explain that the model is decent but not perfect—it will miss some churners (low recall) and waste time on false alarms (moderate precision). You recommend retraining with more data or trying a different algorithm.
The step-by-step actions you (the IT professional) take include:
Preprocessing the data (cleaning missing values, turning categories into numbers).
Splitting data into train/validation/test.
Uploading data to S3.
Creating a SageMaker training job with an algorithm.
Running automatic tuning.
Running evaluation on the test set.
Interpreting metrics and deciding next steps.
This is the day-to-day reality of building ML models on AWS. The exam tests whether you can choose the right service for each step and understand what the metrics mean.
The MLS-C01 exam dedicates a significant portion of questions to training, tuning, and evaluating models. You must know the exact terminology and which AWS service does what.
Key exam concepts:
Overfitting: When a model performs brilliantly on training data but poorly on test data. The exam loves to describe a scenario where training accuracy is 99% but test accuracy is 60%, and ask you to identify overfitting. Correct answer: the model is overfitting. Traps: they might call it 'underfitting' (which is when the model is too simple and performs poorly on both training and test data) or 'bias' (systematic error from wrong assumptions).
Underfitting: The opposite—model is too simple, performs poorly on both training and test data. The exam may show low training accuracy (e.g., 60%) and low test accuracy (58%). Answer: underfitting.
Hyperparameter tuning: You must know that SageMaker's Automatic Model Tuning (also called hyperparameter tuning) is the right feature for this. Traps: they might suggest manually trying different values in a notebook (which is inefficient) or using a different service like AWS Batch (which is for batch computing, not hyperparameter search). The correct answer is always Automatic Model Tuning.
Validation dataset: The exam tests that you should not tune hyperparameters based on test data—that leaks information. The validation set is meant for tuning. The test set is for final evaluation. A common trap question describes a data scientist using the test set to choose hyperparameters. The correct answer: they should only use the validation set for tuning.
Evaluation metrics: You must memorise the definitions:
Accuracy: (True Positives + True Negatives) / Total Predictions. Best when classes are balanced.
Precision: True Positives / (True Positives + False Positives). High precision means fewer false alarms.
Recall: True Positives / (True Positives + False Negatives). High recall means fewer missed positives.
F1-score: Harmonic mean of precision and recall. Best when you need a balance.
ROC AUC: Area under the Receiver Operating Characteristic curve. Measures the model's ability to distinguish between classes.
Exam question patterns:
Scenario-based: 'A data scientist trains a model and gets 98% training accuracy but 65% test accuracy. What is the problem?' Answer: overfitting. 'What should they do?' Answer: add regularisation, reduce model complexity, or get more training data.
Service selection: 'Which AWS service should you use to automatically find the best hyperparameters?' Answer: Amazon SageMaker Automatic Model Tuning.
Data split: 'You have a dataset and want to evaluate your model honestly. How should you split the data?' Answer: training, validation, test. Traps: they might say 'training and testing only' (missing validation for tuning).
Metric interpretation: 'A model has precision of 0.9 and recall of 0.3. What does this mean?' Answer: The model is highly accurate when it predicts positive, but it misses most positive cases.
Key definitions to memorise:
Training dataset: Data used to fit the model's parameters (weights).
Validation dataset: Data used to tune hyperparameters and compare models during development.
Test dataset: Data used for final, unbiased evaluation of the trained and tuned model.
Overfitting: Model memorises training data but fails on new data.
Underfitting: Model is too simple to capture patterns in data.
Bias: Error from oversimplifying the model (e.g., assuming a linear relationship when it is not).
Variance: Error from being too sensitive to small fluctuations in training data.
Traps to avoid:
Confusing validation and test sets. Validation is for tuning, test is for final evaluation.
Thinking accuracy is always the best metric. For imbalanced datasets (e.g., 95% non-churn, 5% churn), a model that always predicts 'non-churn' would have 95% accuracy but be useless. The exam expects you to consider precision, recall, or F1 in imbalanced cases.
Assuming more features always help. Too many features can cause overfitting.
Using the same data for training and evaluation. This is a guaranteed mistake—the exam will present this as 'the model was evaluated on the training data' and ask you to identify the error.
Focus your study on SageMaker's training, tuning, and evaluation capabilities, and on metric definitions. These are high-weight exam topics.
Training a model means adjusting its internal weights so that its predictions match the training data as closely as possible.
Hyperparameter tuning is the process of finding the best settings for the learning algorithm, not the model's weights.
Always split your data into three sets: training (to learn), validation (to tune), and test (to evaluate).
Overfitting is when the model performs well on training data but poorly on test data because it memorised noise.
Precision answers 'how many of my positive predictions were correct?' while recall answers 'how many of the actual positives did I catch?'
Amazon SageMaker Automatic Model Tuning is the AWS-native service for optimising hyperparameters at scale.
These come up on the exam all the time. Here's how to tell them apart.
Training Data
Used to teach the model the patterns and relationships
The model sees this data during training and adjusts its weights
Performance on training data can be artificially high due to memorisation
Test Data
Used only for final, unbiased evaluation of the trained model
The model has never seen this data before evaluation
Performance on test data reflects real-world generalisation ability
Overfitting
Model performs well on training data but poorly on test data
Caused by too complex a model or too much training
High variance, low bias
Underfitting
Model performs poorly on both training and test data
Caused by too simple a model or not enough training
High bias, low variance
Precision
Measures how many of the positive predictions were correct
Formula: TP / (TP + FP)
Important when false positives are costly (e.g., spam filter flagging important emails)
Recall
Measures how many of the actual positive cases were captured
Formula: TP / (TP + FN)
Important when false negatives are costly (e.g., cancer detection missing a tumour)
Hyperparameters
Set before training begins (e.g., learning rate, number of trees)
Not learned from the data; chosen by the data scientist
Tuned using the validation set
Model Parameters
Learned from the training data during training (e.g., weights in neural networks)
Adjust automatically during training to minimise error
Final values define the trained model
Automatic Model Tuning
SageMaker runs many training jobs in parallel with different hyperparameter combinations
Saves time and finds better hyperparameters systematically
Recommended for production workloads on the exam
Manual Tuning
Data scientist tries one combination at a time by hand
Slow and prone to missing good combinations
Acceptable only for very small experiments or when learning
Accuracy
Percentage of all predictions that were correct (both positive and negative)
Misleading when classes are imbalanced (e.g., 95% negatives, 5% positives)
Easy to calculate and interpret
F1-Score
Harmonic mean of precision and recall
Better for imbalanced datasets because it accounts for both false positives and false negatives
More robust single metric for classification performance
Mistake
Once a model is trained, you can evaluate it on the same training data to see how good it is.
Correct
You must evaluate the model on a separate test dataset that was never used during training or tuning. Evaluating on training data gives an unrealistically high score (overfitting).
It seems logical to test on what you already have, but the model has already 'seen' the answers, so it looks better than it truly is.
Mistake
Hyperparameter tuning is done by adjusting the model's internal weights during training.
Correct
Hyperparameters are settings that control the training process (like learning rate), not the weights themselves. They are set before training begins and are adjusted manually or via automatic tuning after training.
The word 'parameter' is confusing. People assume all parameters are the same, but hyperparameters are a special category that control how the model learns, not what it learns.
Mistake
If a model has low accuracy on the test set, the only fix is to train for longer.
Correct
Low test accuracy could be due to overfitting, underfitting, bad hyperparameters, not enough data, or wrong algorithm. Training longer might make overfitting worse.
Beginners think 'more training = better', but more training can make the model memorise noise instead of the real pattern.
Mistake
AWS SageMaker is only for training deep learning models on GPUs.
Correct
Amazon SageMaker supports many algorithms (XGBoost, linear learner, etc.) and works on CPUs too. It is a general platform for building, training, and deploying any ML model.
The name 'SageMaker' sounds advanced, so people assume it is only for complex deep learning. In reality, it handles everything from simple regressions to deep neural networks.
Mistake
Validation and test datasets are interchangeable terms.
Correct
They are distinct. The validation set is used during development to tune hyperparameters. The test set is used only once at the end for final evaluation. Using the test set for tuning makes the final score biased.
The words seem similar and both are 'held-out' data, so beginners treat them as synonyms. The exam tests this distinction heavily.
Reveal each answer, then mark whether you got it right. Score 60%+ to unlock the next chapter.
Training data is used to teach the model. Validation data is used during tuning to compare different hyperparameter settings. Test data is used only once, at the end, to get an honest estimate of how well the model will perform on new, unseen data.
If your model has very high accuracy on the training data (e.g., 99%) but much lower accuracy on the test data (e.g., 65%), it is overfitting. The model has memorised the training examples instead of learning general patterns.
It is a feature of SageMaker that automatically tries many hyperparameter combinations to find the set that gives the best performance on your validation data. You define ranges for each hyperparameter, and SageMaker runs multiple training jobs in parallel to find the optimal values.
No. You should only use the validation set for tuning. Using the test set for tuning gives you a biased result because the model has effectively 'seen' the test data during development. The test set must be kept untouched until the final evaluation.
Accuracy is the percentage of all predictions (both positive and negative) that were correct. Precision is the percentage of positive predictions that were actually correct. For example, in spam detection, accuracy tells you how many emails you classified correctly overall, while precision tells you how many of the emails you flagged as spam were actually spam.
Yes. Amazon SageMaker provides a visual interface called Amazon SageMaker Canvas that lets you import data, train models, and make predictions using a point-and-click interface. No coding is required, though the exam focuses on the code-based SageMaker experience.
F1-score is the harmonic mean of precision and recall. It gives a single score that balances both metrics. You should use it when you have an imbalanced dataset (e.g., many more negatives than positives, or vice versa) because accuracy alone can be misleading.
You've finished Training, Tuning, and Evaluating Models. Continue through the MLS-C01 study guide to build a complete picture of the exam.
Done with this chapter?