If you don't properly evaluate your machine learning model, you could deploy a model that fails spectacularly in the real world, costing your company millions in bad predictions and lost trust. That is why Model Evaluation, Validation, and Testing are not optional extras - they are the core safety net that ensures your model actually works when it meets real data. For the Google PMLE exam, you must understand how to use the right metrics and techniques to prove your model is genuinely good, not just good at memorising its training data.
Jump to a section
A simple way to picture Model Evaluation, Validation, and Testing
10 runners are training for a marathon. You, the coach, have designed a new training plan. Before race day, you need to check if the plan works. You give the plan to Runner A (your validation set). Runner A runs a 10-mile practice run. You watch Runner A the whole time, adjusting the plan based on how they perform. Runner A completes the run in 60 minutes. You then adjust the plan slightly, based on Runner A's performance, and give the adjusted plan to Runner B (your test set). Runner B also runs the same 10-mile course, but you do NOT watch them or adjust the plan. Runner B finishes in 58 minutes. You now have a good idea of how the plan will work for new runners. Runner A's performance helped you improve the plan. Runner B's performance gave you an honest, final estimate of the plan's ability. If you had only used Runner A and watched them the whole time, you would have over-optimised the plan for Runner A's specific running style, and you might get a nasty surprise on race day with Runner C (new, unseen data). This is precisely what model evaluation, validation, and testing does for machine learning: it prevents you from fooling yourself about how good your model really is.
Model evaluation, validation, and testing are the processes used to check how well a machine learning model performs and to make sure it will work on new, unseen data. Think of it like a final exam for your model. The model has been studying (training) on a set of past data. But you don't want to just ask it questions it has already seen the answers to - that would be cheating. You want to give it brand-new questions to see if it has truly learned the underlying patterns.
First, let's define the key terms. When you build a machine learning model, you start with a dataset. A dataset is just a collection of examples, like rows in a spreadsheet. You split this dataset into three separate parts: a training set, a validation set, and a test set.
The training set is the largest part, usually around 60-80% of your data. This is the data the model uses to learn patterns. The model sees the training set examples and their correct answers (labels) and adjusts its internal parameters to predict those answers correctly.
The validation set is a separate, smaller portion of the data (typically 10-20%). You use this set during the training process to make decisions about your model. For example, you might try different settings (called hyperparameters, like the learning rate or the number of layers in a neural network). You train the model on the training set, then check its performance on the validation set. Based on how it does on the validation set, you adjust your hyperparameters and try again. This is like a 'practice exam' - you use it to guide your decisions, but you don't take the score as the final grade.
The test set is the final, untouched portion of the data (usually 10-20%). You never look at the test set until your model is completely finished. Once you have tuned your model using the validation set and you are happy with its performance, you run the model on the test set one single time. The score on the test set is your final, honest estimate of how your model will perform on brand-new data in the real world.
Why is this separation so important? There is a concept called data leakage. Data leakage happens when information from outside the training set accidentally leaks into the model during training, causing the model to appear much better than it actually is. If you use your test set to make decisions (like tuning hyperparameters), you are causing data leakage. The model will seem great on that test set, but it will fail on new, real-world data because it was secretly 'cheating' by having seen the test data indirectly.
There are two main methods for creating these splits: holdout validation and cross-validation. Holdout validation is the simplest - you just split the data into training, validation, and test sets once. Cross-validation is a more robust method where you split the data into 'k' folds (like 5 or 10). You train the model on k-1 folds and validate on the remaining fold. You repeat this k times, each time using a different fold for validation. The final performance is the average across all k runs. Cross-validation gives a more reliable estimate of performance, especially when you have a small dataset.
Now, let's talk about metrics. A metric is a number that measures some aspect of your model's performance. The choice of metric depends on the type of problem you are solving.
For classification problems (where the model predicts a category, like 'spam' or 'not spam'):
Accuracy: The percentage of predictions that were correct. Simple, but can be misleading if the classes are imbalanced (e.g., 95% of emails are not spam - a model that always says 'not spam' would be 95% accurate but useless).
Precision: Out of all the examples the model predicted as positive (e.g., spam), how many were actually positive? High precision means fewer false positives.
Recall (also called sensitivity): Out of all the actual positive examples, how many did the model correctly identify? High recall means fewer false negatives.
F1-score: The harmonic mean of precision and recall. A good all-around metric when you care about both false positives and false negatives.
Confusion matrix: A table that shows the counts of true positives, true negatives, false positives, and false negatives. It gives a complete picture of your model's mistakes.
ROC curve and AUC: The ROC curve plots the true positive rate against the false positive rate at various thresholds. AUC (Area Under the Curve) is a single number that summarises the overall ability of the model to distinguish between classes.
For regression problems (where the model predicts a continuous number, like house price):
Mean Absolute Error (MAE): The average absolute difference between predicted and actual values. Easy to understand - it's the average error in the same units as the target.
Mean Squared Error (MSE): The average squared difference between predicted and actual values. Squares the errors, so large errors are penalised more heavily.
Root Mean Squared Error (RMSE): The square root of MSE. It is in the same units as the target, making it more interpretable than MSE.
R-squared (R²): The proportion of the variance in the target variable that is predictable from the features. Ranges from 0 to 1, with 1 being a perfect fit.
For ranking problems (like search results or recommendations):
Precision@k: How many of the top k results are relevant.
Recall@k: Out of all relevant items, how many appear in the top k.
Mean Average Precision (MAP): The average of the precision values across different recall levels.
Normalised Discounted Cumulative Gain (NDCG): Measures the quality of the ranking, giving higher weight to relevant items that appear earlier in the list.
The PMLE exam expects you to know which metric to use for which scenario. A common trap is using accuracy on an imbalanced dataset. Another trap is using validation performance as a final estimate of model quality - you must use the test set for that.
Finally, there is the concept of overfitting and underfitting. Overfitting means your model has memorised the training data too well, including noise and random fluctuations, so it performs poorly on new data. It is like a student who memorised the exact answers to practice questions but cannot solve a new, slightly different problem. Underfitting means your model is too simple and has not learned the underlying patterns even in the training data. Validation and testing help you detect overfitting: if your model does very well on the training set but poorly on the validation/test set, it is overfitting.
Split the Dataset
Divide your dataset into three non-overlapping parts: a training set (e.g., 70%), a validation set (e.g., 15%), and a test set (e.g., 15%). Ensure the split is stratified if you have imbalanced classes, meaning each set has roughly the same proportion of positive and negative examples. This step is critical because it establishes the foundation for honest evaluation.
Train the Model on the Training Set
Feed the training set into your chosen machine learning algorithm. The model learns patterns from the features and their corresponding labels. This is the only set used to adjust the model's internal parameters. The goal is to minimise the loss function on this set.
Validate the Model Using the Validation Set
After each training run (or after a set of hyperparameter experiments), evaluate the model's performance on the validation set. Use the validation score to decide which hyperparameters to try next. For example, if the model overfits (high training accuracy, low validation accuracy), you might reduce the model complexity. This step is iterative and exploratory.
Select the Best Model and Hyperparameters
Based on the validation set scores, choose the model and hyperparameter combination that gives the best validation performance. This is your 'finalist' model. You may also use cross-validation at this stage to get a more robust estimate before final selection.
Test the Final Model on the Test Set
Now that you have selected your final model, run it on the unseen test set exactly once. Record the performance metrics. This score is your final, honest estimate of how the model will perform on new, real-world data. Do not adjust the model based on test results.
Deploy and Monitor Continuously
Deploy the model to production and continuously monitor its performance. Track if the input data distribution changes over time (data drift) or if the relationship between features and target changes (concept drift). If performance drops below a threshold, trigger retraining using the same evaluation framework.
Imagine you are a machine learning engineer at a company that runs an e-commerce website. Your boss asks you to build a model that predicts whether a customer will buy a product within the next 30 days, based on their past browsing and purchase history. This is a binary classification problem (will buy vs. will not buy). You have a dataset of 1 million customer records from the last year.
Here is exactly what you do step by step:
First, you split the 1 million records into three sets: 700,000 for training, 150,000 for validation, and 150,000 for test. You ensure the split is stratified, meaning the proportion of buyers (positive examples) is roughly the same in all three sets. This prevents bias.
You train several different model types on the training set: a logistic regression, a random forest, and a gradient boosted tree. For each model, you have hyperparameters to tune. For the random forest, you try different numbers of trees and different maximum depths. For each combination of hyperparameters, you train the model on the training set and evaluate its performance on the validation set. You use a metric called F1-score, because the company cares about both catching buyers (recall) and not wasting resources on false alarms (precision). You track the F1-score on the validation set and pick the hyperparameters that give the highest F1-score.
After two weeks of experimentation, you have your best model: a gradient boosted tree with 500 trees and a learning rate of 0.1. Its F1-score on the validation set is 0.82. You are tempted to declare victory, but you must not use the validation set for your final number. Instead, you run the model one single time on the untouched test set. The F1-score on the test set is 0.79. This is your final, honest estimate. You report this number to your boss.
What if you had skipped the test set and just used the validation set? Your model was tuned to perform well on that specific validation set, so it would likely perform worse on new customers (the test set). Deploying the model without a proper test would lead to a nasty surprise: the model would underperform in production, causing the company to miss sales opportunities and upset customers with irrelevant offers.
In practice, your job doesn't stop at reporting the F1-score. You also:
Analyse the confusion matrix to see which specific mistakes the model makes. Are there more false positives (predicting a customer will buy when they don't) or false negatives (missing a customer who will buy)? Based on this, you might adjust the classification threshold.
Check for fairness. Does the model perform differently for different customer demographics? If so, you might need to retrain or adjust the model.
Set up monitoring in production. After deployment, you log the model's predictions and compare them to actual outcomes to detect data drift (when the real-world data starts to look different from your training data). If the model's performance drops, you trigger a retraining pipeline.
The PMLE exam will test your ability to reason through these real-world scenarios. They will describe a business problem and ask you to choose the correct evaluation strategy or metric.
The PMLE exam tests Model Evaluation, Validation, and Testing extensively. You need to be prepared for several specific question types and traps.
Here are the exact concepts they love to test:
The difference between training, validation, and test sets: This is probably the most tested concept in this area. Expect questions where they describe a scenario and ask which set to use for a specific purpose (e.g., hyperparameter tuning must use the validation set, final performance report must use the test set). The trap is always choosing the test set for tuning or using the validation set for the final score.
Data leakage: They will describe a situation where data from the future or from outside the training data leaks into training. For example, if you normalise your entire dataset before splitting into train/test, you have caused data leakage. Another common leak: using a feature like 'total purchases up to today' when the training data includes future dates. You must be able to spot these leaks.
Metric selection based on problem type: Questions will describe a business scenario (e.g., 'We need to minimise false positives in a spam filter because flagging legitimate emails as spam makes users angry'). You must pick the right metric (e.g., optimise for precision). Another example: 'We need to catch as many fraudulent transactions as possible, even if we annoy some legitimate customers' - optimise for recall.
Imbalanced datasets: They love to test this. If a dataset has 99% negative examples and 1% positive examples, accuracy is a terrible metric. You should use precision, recall, F1-score, or AUC-ROC. They will ask what happens if you use accuracy on an imbalanced dataset (the answer: a model that always predicts the majority class will have 99% accuracy but is useless).
Cross-validation: They will ask when to use cross-validation (when you have a small dataset and want a more reliable performance estimate). They may ask about the number of folds (5 or 10). They may ask about stratified cross-validation (ensuring each fold has the same proportion of classes as the whole dataset).
Confusion matrix interpretation: You will need to read a confusion matrix and calculate precision, recall, F1-score, and accuracy.
ROC and AUC: They expect you to know that AUC ranges from 0 to 1, with 1 being a perfect model and 0.5 being random. They may ask what an AUC of 0.5 means (the model is no better than random guessing).
Regression metrics: You need to know MAE, MSE, RMSE, and R-squared. They may ask which is more sensitive to outliers (MSE and RMSE). They may ask what an R-squared of 0.9 means (the model explains 90% of the variance in the target).
Overfitting and underfitting detection: They will show a graph of training and validation error over epochs. If training error keeps decreasing but validation error starts increasing, that is overfitting. If both errors are high and plateaued, that is underfitting.
Traps to watch out for:
The 'validation set for final evaluation' trap: They will phrase it as 'The model scored 0.85 on the training set and 0.84 on the validation set, so the model is ready to deploy.' This is wrong because the validation set was used for tuning, so it is not an unbiased estimate.
The 'accuracy is always good' trap: On imbalanced data, accuracy is misleading.
The 'more data always helps' trap: More data can help, but if the data is noisy or irrelevant, it can hurt performance.
The 'correlation implies causation' trap: Just because a feature is correlated with the target does not mean it causes the prediction. The exam tests your understanding of evaluation in the context of causal inference.
Key definitions to memorise:
Training set: Used to fit the model parameters.
Validation set: Used to tune hyperparameters and make model selection decisions.
Test set: Used for final, unbiased evaluation of the chosen model.
Overfitting: Model performs well on training data but poorly on new data.
Underfitting: Model performs poorly on both training and new data.
Bias: Error due to overly simplistic assumptions in the learning algorithm.
Variance: Error due to too much sensitivity to small fluctuations in the training set.
Bias-variance tradeoff: A fundamental tradeoff in machine learning. Simple models have high bias and low variance. Complex models have low bias and high variance. The goal is to find a balance.
Data leakage: When information from outside the training set is used to create the model, leading to overly optimistic performance estimates.
Stratification: Ensuring that each split of the data has the same proportion of classes as the original dataset.
Always split your data into three separate sets: training, validation, and test, and never let the test set influence any model decision until the final evaluation.
The validation set is used for hyperparameter tuning and model selection, while the test set is used for a single, final unbiased performance estimate.
Accuracy is a misleading metric for imbalanced datasets; use precision, recall, F1-score, or AUC-ROC instead.
Overfitting occurs when a model performs well on the training set but poorly on the validation or test set, indicating it has memorised noise instead of learning patterns.
Cross-validation provides a more reliable performance estimate than a single train/validation split, especially for small datasets.
Data leakage causes overly optimistic performance estimates and must be systematically prevented by careful data splitting and preprocessing.
The choice of evaluation metric must align with the business problem: optimise for precision when false positives are costly, and recall when false negatives are costly.
A confusion matrix gives a complete picture of classification model errors, including true positives, true negatives, false positives, and false negatives.
These come up on the exam all the time. Here's how to tell them apart.
Training Set
Largest portion of data (typically 60-80%)
Used to fit model parameters directly
Model sees this data many times during training
Validation Set
Smaller portion (typically 10-20%)
Used to tune hyperparameters and select model
Model does not learn from this data; it is only evaluated
Validation Set
Used iteratively during model development
Influences model decisions (hyperparameter tuning)
Performance estimate is biased (optimistic)
Test Set
Used only once at the very end
Does not influence any model decisions
Performance estimate is unbiased (honest)
Accuracy
Percentage of all predictions that are correct
Misleading on imbalanced datasets
Treats all classes equally
F1-Score
Harmonic mean of precision and recall
Better for imbalanced datasets
Penalises extreme imbalance between precision and recall
Mean Absolute Error (MAE)
Average of absolute errors
Less sensitive to outliers
Easier to interpret in same units as target
Root Mean Squared Error (RMSE)
Square root of average squared errors
More sensitive to outliers (penalises large errors more)
Also in same units, but gives higher weight to large errors
Overfitting
Model has low bias, high variance
Performs well on training set, poorly on validation/test
Caused by overly complex model or too many features
Underfitting
Model has high bias, low variance
Performs poorly on both training and validation/test
Caused by overly simple model or insufficient features
Holdout Validation
Single train-validation split
Faster to compute
More variance in performance estimate (depends on the one split)
Cross-Validation
Multiple splits (e.g., k-fold)
More computationally expensive
Lower variance and more reliable performance estimate
Mistake
A high accuracy score on the test set means the model is perfect and ready for deployment.
Correct
Accuracy can be misleading, especially on imbalanced datasets. You must also check precision, recall, and the confusion matrix to understand the types of errors the model makes. A model might have high accuracy but miss all the rare but critical positive cases.
People naturally associate high numbers with 'good'. Accuracy is also easy to understand, so beginners latch onto it without considering the class distribution.
Mistake
The validation set and the test set are the same thing, just different names.
Correct
They serve completely different purposes. The validation set is used iteratively during model development to tune hyperparameters. The test set is used only once at the very end to provide an unbiased estimate of final model performance. Using the test set for tuning violates this purpose and leads to data leakage.
The names are confusingly similar ('validation' vs 'test'). Many tutorials also use the terms interchangeably, so beginners don't realise the strict separation required.
Mistake
If I have a small dataset, I should skip the test set and just use all data for training to get the best model.
Correct
Skipping the test set means you have no unbiased estimate of how the model will perform on new data. You must always hold out a test set. For small datasets, use cross-validation to get a reliable performance estimate while still having a held-out test set for final evaluation.
Beginners think 'more data always improves the model' and they are afraid of 'wasting' data on a test set. They don't realise the test set is not for improving the model - it's for measuring it honestly.
Mistake
Cross-validation means I don't need a separate test set.
Correct
Cross-validation is a technique for estimating model performance on the validation set, but you still need a separate held-out test set for final evaluation. The test set is only used once. Cross-validation on the training data does not replace the final test.
The term 'validation' in cross-validation makes people think it is the final evaluation. They don't understand the layered structure: cross-validation is used to pick the best model, then the test set is used to evaluate that model.
Mistake
The AUC-ROC score is always a better metric than accuracy, so I should always use it.
Correct
AUC-ROC is a good metric for imbalanced datasets and gives a general sense of model discrimination, but it has limitations. For example, it can be overly optimistic if there are a lot of negative examples. For highly imbalanced problems, Precision-Recall AUC (AUPRC) is often more informative. The right metric depends on the specific business problem.
Beginners hear that 'accuracy is bad' and 'AUC is good' and take it as an absolute rule. They don't learn the nuances of when each metric is appropriate.
Mistake
Data leakage is only a problem if you accidentally include the target variable as a feature.
Correct
Data leakage can happen in many ways. Examples include: using future data to predict the past (e.g., using 'total purchases up to December' to predict a purchase in November), normalising the data before splitting, using features that are influenced by the target (e.g., using 'number of customer service calls' to predict 'customer churn' when the calls happened after the churn decision), or including information from the test set during feature engineering.
The classic example of 'target in features' is so vivid that beginners assume it is the only form of leakage. They don't think about leakage through time, through preprocessing, or through feature engineering.
Reveal each answer, then mark whether you got it right. Score 60%+ to unlock the next chapter.
The validation set is used during model development to tune hyperparameters and guide model selection. The test set is held back and used only once at the very end to provide an unbiased, final estimate of model performance.
Using the test set for tuning causes data leakage, because the model is 'seeing' the test data indirectly through the tuning decisions. This makes the test score overly optimistic and no longer a reliable estimate of real-world performance.
Use cross-validation when you have a small dataset and want a more reliable estimate of model performance. It reduces the variance of the performance estimate by averaging results across multiple splits.
Data leakage is when information from outside the training set accidentally influences the model, causing overly optimistic performance. Prevent it by splitting data before any preprocessing, not using future data to predict the past, and keeping the test set completely separate until final evaluation.
Do not use accuracy. Instead, use precision, recall, F1-score, or AUC-ROC. For highly imbalanced problems, Precision-Recall AUC (AUPRC) is often better than AUC-ROC.
This is a classic sign of overfitting. The model has memorised the training data's noise and patterns that do not generalise to new data. You need to simplify the model or use regularisation techniques.
You've finished Model Evaluation, Validation, and Testing. Continue through the PMLE study guide to build a complete picture of the exam.
Done with this chapter?