If you deploy a machine learning model and never check whether it is still making good predictions, you will eventually make decisions based on nonsense. That is how companies lose millions, misdiagnose patients, or approve fraudulent loans. This chapter explains how to watch your model for signs of age and decay so you catch problems before they cause real damage.
Jump to a section
A simple way to picture Monitoring Model Performance and Drift Detection
Every weekday morning at 7:15 AM, the number 42 bus picks you up from the stop on Elm Street and delivers you to work by 7:45 AM. You trust this bus. You've bet your career on it — you built your entire morning routine around its reliability.
But one Tuesday, you arrive at the stop at 7:14 AM and the bus is already gone. It came at 7:05 AM. The next day, it doesn't show up until 7:28 AM. Then on Thursday, the bus arrives right on time but takes a different route — you end up at a warehouse on the industrial estate instead of your office. Something has changed. The world around the bus has shifted. A new housing estate opened, altering the traffic patterns (data drift). The driver retired and the replacement takes a different line through the roundabout (concept drift). The bus itself is still a bus, but its behaviour no longer serves your goal of arriving at the office.
You cannot keep blindly boarding the bus. You need to watch the actual arrival time and journey outcome every single day. You need to set an alarm: 'If bus does not arrive within 2 minutes of 7:15 AM, alert me.' You need to check if the bus still drops you at the office. In machine learning, your model is that bus. It was perfect when you deployed it, but the world moves. Monitoring model performance means never assuming the 7:15 bus still runs on time.
Machine learning models are like baking soda volcanoes — they work brilliantly for the science fair, but if you leave them in the garage for six months, they do not erupt anymore. The model you train today is frozen in time. It is a snapshot of the patterns that existed in your training data. But the real world never freezes. Customer preferences change, economic conditions shift, new products launch, seasons change, and data collection systems get updated. Your frozen model does not know any of this. It keeps making predictions based on the old world. This gap between the world your model was trained on and the current world is the root of all monitoring problems.
To understand monitoring, you first need to understand the two main ways a model can go bad: data drift and concept drift.
Data drift means the input data your model receives has changed. Imagine your model was trained to predict house prices using three features (inputs): square footage, number of bedrooms, and age of the house. In the training data, square footage ranged from 800 to 3,000 square feet. After deployment, you start receiving houses with 4,500 square feet. That is data drift. The statistical properties of the input have shifted. The model has never seen this range of values before, so its predictions become unreliable. Data drift can happen gradually (seasonal trends) or suddenly (a new law changes how data is recorded).
Concept drift means the relationship between the input features and the target variable has changed. The input data looks the same, but what those inputs mean has shifted. For example, your house price model uses 'number of bedrooms' as a feature. In 2020, a three-bedroom house in your city cost $350,000. In 2023, a three-bedroom house costs $480,000 because of a housing shortage. The input (three bedrooms) still looks the same, but the target (price) has changed. The concept of 'what a three-bedroom house is worth' has drifted. The model keeps predicting $350,000, but the real market says $480,000. The model is now wrong. Concept drift is often harder to detect because the input statistics might look fine — it is the underlying relationship that broke.
So how do you actually monitor for these problems? You track metrics. Two categories of metrics exist: performance metrics and data metrics.
Performance metrics are about the model's accuracy on new, unseen data. If your model makes a prediction, and you later discover the ground truth (the actual correct answer), you can check if the model was right. For example, if your model predicts whether a customer will churn (cancel their subscription), and the customer either churns or does not, you can measure the model's accuracy, precision, recall, F1 score, and other classification metrics every week. If accuracy drops from 92% to 78% over a month, you have detected either data drift or concept drift. The problem is that ground truth often arrives with a delay. You might not know if a loan defaulted until six months after the loan was approved. Performance metrics are the gold standard but they are lagging indicators.
Data metrics give you an early warning. You can monitor the statistical distribution of each input feature. If the average customer age in your training data was 34, but this month the average age is 41, that is a red flag. You can monitor min, max, mean, standard deviation, and more exotic measures like the Kullback-Leibler divergence (KL divergence), which quantifies how different two probability distributions are. The key is to set thresholds: if the mean age shifts by more than 3 years in a week, raise an alert.
You also need to know where to monitor. The three typical checkpoints are: at inference (when the model makes a prediction, log the input features), at prediction storage (save every prediction the model makes), and at ground truth collection (when you eventually know the real answer).
Finally, what do you do when you detect drift? You do not just delete the model and panic. The standard response is to retrain the model on newer data. You might also perform feature engineering to build more robust features. If the drift is severe, you might need to go back to the drawing board and rebuild the model from scratch with new data. The critical takeaway is: monitoring is a continuous process, not a one-time task. It is the maintenance that keeps your model alive.
Establish a Baseline
Before deploying a model, you capture the statistical distribution of every feature in the training data. You record the mean, standard deviation, min, max, and histogram bins for numerical features, and frequency tables for categorical features. This baseline is the reference point against which all future production data will be compared. Without a baseline, you cannot tell if the new data has drifted.
Capture Inference Data
Every time the deployed model makes a prediction in production, you log the input features that were used. These logs become a dataset representing the current input distribution. You typically store these logs in Amazon S3, organised by date and time. This step is critical because you cannot compare distributions if you do not save the incoming data.
Compare Distributions Against Baseline
Using statistical tests like the KS test or JS divergence, you compare each feature's production distribution against its baseline distribution. If the p-value of the KS test drops below a threshold (e.g., 0.05) or the JS divergence exceeds a threshold, you flag that feature as drifted. Many practitioners use the population stability index (PSI), where values above 0.25 indicate significant shift.
Define and Configure Alerts
You set up automated alarms that fire when a drift metric exceeds a predefined threshold. For example, you might configure an Amazon CloudWatch alarm on a custom metric called 'credit_score_drift' that triggers when the KS statistic for the credit score feature exceeds 0.1. Alerts should include severity levels: informational (feature shift detected, low risk), warning (performance degradation possible), and critical (immediate model failure risk).
Diagnose Root Cause
When an alert fires, you do not immediately retrain. You investigate the context: Has the data source changed? Did a new version of the application go live? Is there a seasonal event (Black Friday, tax season)? You look at the specific features that drifted and talk to domain experts. The goal is to separate transient drift from permanent structural drift.
Decide and Execute Remediation
Based on the diagnosis, you choose a remediation. If the drift is permanent and significant, you retrain the model on recent data. If the drift is temporary, you may suppress alerts for the known event duration. If the drift is caused by a data pipeline bug, you fix the pipeline and replay clean data. Optionally, you may roll back to a previously validated version of the model while the new one is being built.
Imagine you work as a machine learning engineer for an e-commerce company called ShopSwift. You deployed a model six months ago that predicts which products a customer is likely to buy next. The model takes in features like: pages viewed in the last 7 days, time spent on each product page, past purchase history, and current season. It outputs a list of 5 recommended products for each customer. The business uses these recommendations on the homepage.
Three months ago, the click-through rate (the percentage of recommendations that customers click) was 14%. Last week, it dropped to 4%. The head of marketing is calling you. You need to diagnose why.
Step one: you check the data metrics first because they are available immediately. You pull up a dashboard that shows the distribution of 'pages viewed in the last 7 days'. In the training data, the average was 22 pages. Yesterday, the average dropped to 9 pages. You also notice that 'time spent on each product page' has dropped from 45 seconds to 12 seconds. Something has changed in user behaviour.
Step two: you investigate the cause. You talk to the product team and discover that the company redesigned the mobile app two weeks ago. The new app buries product pages behind an extra click. Customers are viewing fewer pages because it is harder to find them. The input data has drifted because of the app redesign — this is a classic case of data drift caused by a system change, not by the model itself.
Step three: you examine concept drift. You check whether the relationship between 'past purchase history' and 'items bought' has shifted. You look at customers who bought baby diapers six months ago. In the past, those customers often bought baby wipes next. Now they are buying baby toys. The model still recommends baby wipes, but customers ignore them. The concept of 'what follows a diaper purchase' has drifted because the baby cohort has aged.
Step four: you set up automated alerts using a tool like Amazon SageMaker Model Monitor (a service from AWS that monitors models in production). You configure two alerts: one that fires if the mean 'pages viewed' drops below 15, and another that fires if the click-through rate drops below 8%. The alerts will email you and the product team.
Step five: you decide to retrain the model using data from the last three months instead of the original training data that was six months old. You schedule this retraining monthly to stay ahead of drift.
In a real workplace, monitoring is often split between an ML engineer who sets up the measurement infrastructure and a monitoring team that watches dashboards. The tools include:
Amazon SageMaker Model Monitor for automated data drift detection
Amazon CloudWatch for custom metrics and alerts
Custom Python scripts using libraries like scikit-learn to calculate statistical distances
Notebooks for deep-dive analysis when drift is detected
The professional never assumes the model is fine. They check weekly, they set thresholds, and they have a playbook for exactly what to do when drift is found.
The MLA-C01 exam tests your understanding of monitoring model performance and drift detection extensively. Section 3.1 is worth a significant portion of the 'Deploy and Monitor' domain, which itself is one of four domains. You can expect roughly 5 to 8 questions on this topic across the exam.
The exam loves to test the distinction between data drift and concept drift. You will see scenario-based questions where they describe a change in the world and ask you to classify it. Example: 'A model predicts loan defaults. After a recession, the same input features (credit score, income) now correspond to higher default rates. What type of drift has occurred?' The answer is concept drift. The trap answer is data drift because students confuse 'input data changing' with 'the relationship changing'.
Another common question type asks about the correct order of operations for monitoring. They might list steps like: 'Collect inference data, compare distributions, set thresholds, alert, retrain.' They ask you to put them in sequence. Memorise this order: define metrics baseline, collect inference data, compare distributions, detect drift, alert, diagnose root cause, retrain or rollback.
They also test which AWS services you use for monitoring. The key services to remember:
Amazon SageMaker Model Monitor: purpose-built for monitoring models in production. It automatically detects data drift, model quality drift, bias drift, and feature attribution drift.
Amazon CloudWatch: for custom metrics and dashboards. If the question mentions 'setting an alarm on a custom metric created by a Lambda function', CloudWatch is the answer.
Amazon SageMaker Clarify: specifically for bias detection and explainability, but it also works with Model Monitor for drift.
Amazon S3: where logs of predictions and ground truth are stored for analysis.
The exam sets traps around 'ground truth latency'. They might say: 'A model's accuracy metric has remained stable for three months. Is the model healthy?' The correct answer is: 'You cannot determine health from accuracy alone if ground truth arrives with significant delay.' They want you to catch that performance metrics are lagging indicators.
Another trap: 'If data drift is detected, you should immediately retrain the model.' Not necessarily. First, diagnose the root cause. If the drift is due to a temporary event like a holiday sale, retraining on that data might make the model worse after the sale ends. The correct approach is to investigate and then decide on retraining or reverting to a previous version.
Memorise the specific metrics used for drift detection:
Kolmogorov-Smirnov test (KS test) for comparing two distributions of numerical features
Chi-squared test for categorical features
Jensen-Shannon divergence (JS divergence) as a symmetric alternative to KL divergence
Population stability index (PSI) widely used in finance
Finally, the exam will ask you to interpret a dashboard. They might show a graph where the baseline distribution of 'age' is a bell curve centered on 35, and the production distribution is shifted to 45. They ask: 'What does this indicate?' The answer is data drift. They love pairing graphs with multiple-choice.
Data drift means the input data distribution has changed; concept drift means the relationship between inputs and outputs has changed.
Performance metrics (like accuracy) are lagging indicators because ground truth often arrives with a delay.
Data metrics (like feature distribution statistics) provide early warning signs before accuracy drops.
The Kolmogorov-Smirnov test, Chi-squared test, and Jensen-Shannon divergence are standard statistical methods for quantifying drift.
Amazon SageMaker Model Monitor is the AWS service purpose-built for detecting data drift, model quality drift, bias drift, and feature attribution drift.
When drift is detected, diagnose the root cause before retraining — retraining on temporary anomalies can harm the model.
Monitoring must be continuous; you cannot validate a model once and assume it works forever.
Ground truth latency is a critical factor — you must account for the delay between prediction and actual outcome when evaluating model health.
These come up on the exam all the time. Here's how to tell them apart.
Data Drift
Change in the statistical distribution of input features (e.g., average age shifts from 35 to 45).
Detected by comparing feature distributions using KS test or JS divergence.
Fix often involves retraining on newer data that reflects the current feature distribution.
Concept Drift
Change in the relationship between input features and the target variable (e.g., the same age now predicts a different salary).
Detected by monitoring model performance metrics like accuracy or F1 score over time.
Fix may require feature engineering or changing the model architecture, not just retraining on new data.
Performance Metrics
Measure how accurate the model's predictions are against ground truth (e.g., accuracy, precision, recall).
Are lagging indicators because ground truth often arrives with a delay (days to months).
Can only be computed after the true label is known, so they cannot provide real-time alerts.
Data Metrics
Measure the statistical properties of the input features (e.g., mean, standard deviation, KS statistic).
Are leading indicators that can flag problems before predictions become inaccurate.
Can be computed immediately after each inference because no ground truth is needed.
Amazon SageMaker Model Monitor
Purpose-built for monitoring machine learning models in production.
Automatically computes data drift, model quality drift, bias drift, and feature attribution drift.
Generates drift reports and integrates directly with SageMaker endpoints.
Amazon CloudWatch
General-purpose monitoring service for all AWS resources, not ML-specific.
User must define custom metrics and alarms manually using logs or CloudWatch Agent.
Best for custom monitoring scenarios where you need to track metrics not covered by Model Monitor.
Kolmogorov-Smirnov (KS) Test
A statistical test that compares the maximum difference between two cumulative distribution functions.
Sensitive to shifts in location and shape of the distribution.
Works best on numerical features and assumes continuous distributions.
Jensen-Shannon (JS) Divergence
A symmetric measure of the difference between two probability distributions.
Always produces a finite value between 0 and 1 (or 0 and log(2)), making thresholds easier to set.
Works on discrete and continuous features and is more robust to small sample sizes.
Retraining on All New Data
Uses every piece of available data since the last retraining.
Can be computationally expensive and slow for large datasets.
May include outdated patterns that no longer reflect the current environment.
Retraining on Selected Recent Data
Uses only a window of the most recent data (e.g., last 3 months).
Is faster and more resource-efficient because it processes less data.
Better captures the current concept and ignores stale patterns.
Mistake
If the model's accuracy on new data is still high, no drift is happening.
Correct
Accuracy can stay high for a while even after drift begins, because early drift may affect only a small subset of predictions. Also, ground truth may arrive with a delay, so you are measuring accuracy on old data, not current data.
People assume accuracy is a real-time measure. They forget that ground truth takes time to collect, so the accuracy they see today is actually from predictions made weeks ago.
Mistake
Data drift and concept drift are the same thing and can be used interchangeably.
Correct
Data drift is a change in the input feature distribution. Concept drift is a change in the relationship between inputs and outputs. They are different problems requiring different detection methods and different fixes.
Both involve the word 'drift' and both degrade model performance. Beginners lump them together because the symptoms look similar — the model performs worse.
Mistake
If the model was trained on high-quality data and validated thoroughly, you do not need to monitor it.
Correct
The environment the model operates in changes constantly. No amount of initial validation can guarantee performance tomorrow. Monitoring is a continuous requirement, not a one-time due diligence task.
This comes from a software engineering mindset where a tested application generally works forever. Machine learning models are statistical, not deterministic — the world changes around them.
Mistake
Retraining the model on any new data is always the correct response to drift.
Correct
Retraining on corrupted or temporarily anomalous data can make the model worse. You must first diagnose whether the drift is permanent (structural shift) or temporary (seasonal spike). Retrain only when the drift reflects a lasting change in the underlying distribution.
People want a simple button to push when something goes wrong. Retraining feels like a reset button, but it is a surgical intervention, not a cure-all.
Mistake
If the model's predictions look 'reasonable' — not obviously wrong — then the model is fine.
Correct
A model can produce seemingly reasonable predictions that are nevertheless incorrect because the relationship between inputs and outputs has shifted. For example, a demand forecasting model might predict 1,200 units when the real demand is 1,800. The prediction of 1,200 is not absurdly low, but it is still 33% off.
Humans are bad at judging statistical correctness from individual examples. We look for extreme outliers, but drift often manifests as a subtle, gradual shift that is invisible without formal metrics.
Reveal each answer, then mark whether you got it right. Score 60%+ to unlock the next chapter.
You monitor data drift on the input features instead of waiting for ground truth. By tracking the statistical distribution of each feature, you can detect changes in the input data long before you know whether the predictions were correct.
Data drift is a change in the input features themselves (e.g., customer ages shift from 30-40 to 50-60). Concept drift is a change in the relationship between inputs and the target (e.g., a three-bedroom house now costs $200,000 more than it did last year, even though the inputs are identical).
Yes, in an ideal world, but in practice you prioritise features that the model is most sensitive to, based on feature importance scores. You also monitor the model's predicted probability distribution and the overall performance metrics when ground truth is available.
There is no universal schedule. Typical patterns include retraining monthly, quarterly, or whenever a drift alert triggers above a critical threshold. The correct frequency depends on how fast your data distribution changes. Some models in finance are retrained daily; others in retail are retrained monthly.
PSI is a metric that quantifies the shift between two distributions. It is commonly used in finance and credit scoring. A PSI value below 0.1 indicates no significant change, 0.1 to 0.25 indicates moderate change, and above 0.25 indicates a significant shift that requires investigation.
Yes, temporarily. If the drift only affects features that are not important for predictions, the model's performance may remain stable. However, ignoring drift is risky because it often worsens over time and eventually degrades performance. You should investigate any detected drift regardless of current accuracy.
You've finished Monitoring Model Performance and Drift Detection. Continue through the MLA-C01 study guide to build a complete picture of the exam.
Done with this chapter?