What happens when a machine learning model that was 95% accurate on launch day starts making bad predictions three months later? Models do not stay accurate on their own. Data changes, customer behaviour shifts, and even the way data is collected can slowly degrade performance. This chapter explains how to watch for that decay — a process called monitoring — and what to do when you catch it — a process called remediation. It is essential for the MLS-C01 exam because it tests whether you know how to keep a model useful in the real world, not just in a training lab.
Jump to a section
A simple way to picture Monitoring and Remediating Models in Production
Have you ever wondered how a restaurant that was amazing six months ago suddenly seems to be serving bland, disappointing food?
That is exactly the problem 'monitoring and remediation' solves for machine learning models. Imagine you are the head chef who created a perfect signature dish. Your customers love it, and the restaurant is thriving. But you do not just cook the dish once and then walk away. Ingredients change subtly over time. The supplier starts sending slightly different tomatoes. The cooking oil has a different smoke point. The kitchen temperature fluctuates. Your trusted sous-chef leaves, and the new one has a slightly heavier hand with salt.
A health inspector works for you, not the government. Every day, this inspector tastes a sample of the dish, checks the ingredient batches, and verifies the cooking process. They have a checklist. If the salt level creeps up by 5%, the inspector flags it. If the tomatoes are too watery, the inspector raises a warning. The inspector does not fix the dish — they just send an alert saying 'Problem detected in ingredient quality.'
Then you, the chef, decide on a fix. Maybe you find a new tomato supplier. Maybe you adjust the recipe to account for the new oil. This is 'remediation.' You are not starting from scratch. You are making a small, targeted correction based on the inspector's report.
In tech terms, the model is your signature dish. The data flowing in every day are your ingredients. Monitoring tools act as the health inspector, constantly checking for 'drift' — small changes that make the model's predictions less accurate. Remediation is your action plan to retrain the model or change the data source before your customers (the business) complain.
Let us start with the big picture. Once a machine learning model is 'deployed' — meaning it is live and being used by a business — it starts receiving new data and making predictions. Most beginners think the hard work is over. Actually, the hard work has just begun. The model is now in a dynamic, living environment. Everything around it changes. Customers change their habits, the economy shifts, and competitors change their prices. These changes mean the data the model sees today can look very different from the data it was trained on last year.
This difference is called 'data drift.' Data drift happens when the statistical properties of the input data change. For example, a model that predicts house prices was trained in a market where the average house cost £250,000. A year later, interest rates have dropped, and the average house now costs £350,000. The input data ('house price') has drifted upward. The model was never shown many £350,000 houses during training, so its predictions become less reliable.
There is also 'concept drift.' This is more subtle. Concept drift happens when the relationship between the input data and the target variable changes. Imagine a model that predicts whether an online shopper will buy a product based on how many emails they opened. In the past, opening 5 emails in a week meant a 70% chance of buying. But now, customers are overwhelmed by spam, so opening 5 emails only predicts a 10% chance. The 'concept' of what opening emails means has changed, even though the data (number of emails opened) looks the same.
How do you monitor for drift? You use a technique called 'statistical hypothesis testing' on the incoming data. You compare the distribution of new data to the distribution of the training data. If they are significantly different, you flag an alert. Common statistical tests include the Kolmogorov-Smirnov test (for numerical data) and the Chi-squared test (for categorical data). AWS SageMaker provides a built-in feature called 'Model Monitor' that does this automatically. It checks for drift in the input features and also in the model's predictions.
Once you detect drift, you need a 'remediation strategy.' This is your plan for fixing the model. The simplest remediation is 'retraining.' You take the new, drifted data, combine it with the old training data (or just use the new data), and train a new version of the model. This new model learns the current patterns. Another strategy is 'full redeployment.' You build a completely new model from scratch using only the most recent data. A third strategy is 'online learning', where the model updates itself incrementally with every new data point — but this is rare and complex.
AWS offers several tools for this whole process. 'Amazon SageMaker Model Monitor' is the monitoring tool. It runs on a schedule and publishes metrics to Amazon CloudWatch (a dashboard for logs and metrics). 'Amazon SageMaker Pipelines' helps automate the retraining and redeployment process. You can set up a 'trigger' so that when Model Monitor detects drift, it automatically starts a pipeline that trains a new model. This is called 'automated remediation.'
Why does this matter for the exam? The MLS-C01 exam tests your understanding of the difference between data drift and concept drift. It tests which AWS service monitors drift (SageMaker Model Monitor) and which service handles the retraining pipeline (SageMaker Pipelines). It also tests the concept of 'baseline' — the reference statistics from your training data that you compare against. If you do not create a baseline, you cannot detect drift because you have nothing to compare to.
Create a Baseline
Before you can monitor a live model, you must compute statistics from the training dataset. This baseline captures the distribution of each feature and the model's predictions. SageMaker Model Monitor uses this baseline to compare against live data. Without a baseline, drift detection is impossible because there is no reference point.
Configure Model Monitor
You enable SageMaker Model Monitor on your production endpoint or batch transform job. You specify a schedule (e.g., hourly or daily) and define which features to monitor. The service will run statistical tests on the incoming data at each interval.
Review Monitoring Output
Model Monitor publishes results to Amazon CloudWatch as metrics. You check the 'feature drift' and 'prediction drift' metrics. If the statistical test (e.g., Kolmogorov-Smirnov test) returns a low p-value (typically below 0.05), drift is detected.
Set Up Alerting
Create a CloudWatch Alarm that triggers when the drift metric exceeds a threshold (e.g., p-value < 0.05). Configure the alarm to send a notification via Amazon SNS (email, SMS, or Lambda invocation). This ensures the team is informed automatically.
Retrain the Model
The data science team collects recent data (e.g., last 3 months) and uses it to retrain the model. SageMaker Pipelines can automate this step: it loads the new data, preprocesses it, trains a new model, and evaluates it. If the new model outperforms the old one, it is deployed.
Redeploy and Re-monitor
The new model is deployed to the production endpoint, replacing the old one. You then re-enable Model Monitor to track the new model's baseline. The cycle begins again. This step is critical because the new model will eventually drift too.
Imagine you work for a bank called 'FinSave.' Two years ago, your team built a fraud detection model. It is a classification model that labels each transaction as 'Fraud' or 'Not Fraud.' It worked perfectly at launch, catching 98% of fraudulent transactions. The business was happy. But recently, the fraud rate has increased, and the model is only catching 70% of frauds. Customers are complaining. The bank is losing money.
Let us walk through what an IT professional would actually do, step by step.
First, they would check the SageMaker Model Monitor dashboard. They look at a graph called 'feature importance drift' — which shows if any input feature (like 'transaction amount' or 'location') has shifted. They see that 'transaction amount' has drifted. In the training data, the average fraudulent transaction was £500. In the last month, the average fraudulent transaction is £50. Criminals have changed their strategy to make small, frequent purchases instead of one large purchase. This is data drift.
Second, they check 'prediction drift.' They look at the model's output distribution. The training data had 2% of transactions flagged as fraud. Now the model is only flagging 0.5% as fraud. That is a strong signal. The model is becoming too confident that transactions are normal, because it was trained on larger amounts.
Third, they set up an alert. They use Amazon CloudWatch to create an alarm. The alarm triggers when the data drift metric exceeds a certain threshold (for example, a p-value of less than 0.05 on the Kolmogorov-Smirnov test). This alarm sends an email to the data science team.
Fourth, they initiate remediation. The team decides to retrain the model using the last three months of data, which includes the new £50 fraudulent transactions. They use SageMaker Pipelines to automate this retraining. The pipeline does the following:
Loads the new transaction data from Amazon S3 (a storage service).
Cleans and preprocesses the data.
Trains a new XGBoost model.
Evaluates the new model against a holdout test set.
If the new model performs better (higher recall), the pipeline automatically deploys it to a production endpoint, replacing the old model.
Fifth, they monitor the new model. They re-run the Model Monitor to confirm the drift is gone. The new model now shows a fraud detection rate of 95% on recent data. The business is happy again.
The key learning is that monitoring and remediation is not a one-time task. It is a continuous cycle. You monitor, detect drift, remediate, and then monitor again. In the exam, you will be asked to choose the right combination of AWS services to automate this cycle.
The MLS-C01 exam tests 'Monitoring and Remediating Models in Production' in several specific ways. You need to be ready for scenario-based questions where you must choose the correct monitoring service and the correct remediation action.
First, the exam loves to test the difference between 'data drift' and 'concept drift.' A typical question: 'A model predicts customer churn. The input features (age, income, usage) have not changed, but the model's predictions are suddenly less accurate. What is most likely happening?' The answer is concept drift — the relationship between the inputs and the output has changed. A trap answer is data drift, which would be incorrect because the inputs themselves did not change.
Second, the exam tests which AWS service is responsible for each step. They will ask: 'Which service should you use to automatically detect when the distribution of input data deviates from the baseline?' The answer is Amazon SageMaker Model Monitor. They might offer 'Amazon CloudWatch' as a trap — CloudWatch is for monitoring infrastructure (CPU usage, memory), not for statistical drift detection.
Third, they test the 'baseline' concept. A question might say: 'A data scientist wants to set up model monitoring. What must they create first?' The answer is a baseline from the training data. Without a baseline, the monitoring tool has no reference point. A trap is 'a new model' — but you already have a model; you just need the statistics of its training data.
Fourth, they test 'automated remediation pipelines.' They will ask: 'Which service can orchestrate the retraining and redeployment of a model when drift is detected?' The correct answer is Amazon SageMaker Pipelines. A trap is 'AWS Lambda' — Lambda can run code, but it is not designed for orchestrating multi-step ML workflows.
Fifth, the exam tests 'alerting mechanisms.' You might see: 'How should you be notified when drift is detected?' The exam expects you to use Amazon CloudWatch Alarms, which can send notifications via Amazon SNS (Simple Notification Service).
Here is a list of key concepts to memorise:
Data drift: change in the distribution of input features.
Concept drift: change in the relationship between inputs and the target.
Baseline: statistics computed from the training dataset to compare against live data.
SageMaker Model Monitor: the AWS service for detecting drift.
SageMaker Pipelines: the AWS service for automating retraining and deployment.
CloudWatch: the service for monitoring and sending alerts.
Remediation strategy: retraining, full redeployment, or online learning.
Finally, watch out for trick questions about 'real-time' vs 'batch' monitoring. SageMaker Model Monitor works for both real-time endpoints (where models answer requests instantly) and batch transform jobs (where models process large files). The exam may ask about configuring monitoring for one type versus the other.
A deployed model's accuracy degrades over time because real-world data changes, a phenomenon called drift.
Data drift is when the statistical properties of the input features change; concept drift is when the relationship between inputs and outputs changes.
Amazon SageMaker Model Monitor is the AWS service for automatically detecting data drift and concept drift in production models.
A baseline must be computed from the training data before monitoring can begin; the baseline is the reference distribution for comparison.
Amazon SageMaker Pipelines is the AWS service for orchestrating the automated retraining and redeployment of a model when drift is detected.
CloudWatch Alarms combined with Amazon SNS provides the alerting mechanism to notify teams when drift is detected.
Remediation strategies include retraining with recent data, full redeployment from scratch, and online learning, with retraining being the most common.
Monitoring and remediation form a continuous cycle: monitor, detect, alert, remediate, and then monitor again.
These come up on the exam all the time. Here's how to tell them apart.
Data Drift
Refers to change in the distribution of input features only.
Detected by comparing statistical properties like mean or distribution shape between training and live data.
Example: average transaction amount drops from £500 to £50.
Concept Drift
Refers to change in the relationship between inputs and the target variable.
Detected by monitoring the model's prediction distribution or performance metrics over time.
Example: same transaction amount now predicts a different fraud probability because fraud patterns changed.
SageMaker Model Monitor
Purpose-built for detecting model drift and data quality issues.
Compares live data to a baseline from the training set.
Outputs metrics like feature drift magnitude and prediction drift.
Amazon CloudWatch
General-purpose monitoring for infrastructure metrics (CPU, memory, latency).
Does not perform statistical comparison for ML model drift.
Can ingest metrics from Model Monitor and trigger alarms, but does not detect drift itself.
Retraining with Recent Data
Uses the most recent data (e.g., last 3 months) as the primary training set.
Faster and more resource-efficient because it uses a smaller dataset.
May forget older patterns; only suitable if recent data is representative of the current state.
Full Redeployment from Scratch
Uses all historical data combined with recent data.
More computationally expensive and slower.
Preserves long-term patterns; good if the drift is temporary or seasonal.
Automated Remediation (SageMaker Pipelines)
Drift detection automatically triggers a pipeline that retrains and deploys a new model.
Reduces human intervention and speeds up response time.
Requires initial setup and maintenance of the pipeline code.
Manual Remediation
A data scientist or engineer manually reviews drift reports and decides when to retrain.
More flexible but slower; risk of delays in critical situations.
Suitable for low-stakes models where immediate action is not required.
Mistake
Once a model is deployed and passes validation, it will stay accurate forever as long as the code does not change.
Correct
A model's accuracy degrades over time because the real-world data it receives changes, even if the code is identical. This is why monitoring for drift is necessary.
This mistake comes from thinking of a model like a fixed formula (like 2+2=4) rather than a statistical approximation of a changing environment. People assume that if the code is stable, the output logic is stable, but they forget the data is the variable.
Mistake
Data drift and concept drift are the same thing and can be used interchangeably.
Correct
Data drift refers to changes in the input data distribution (e.g., average age of customers increases). Concept drift refers to changes in the relationship between input and output (e.g., older customers used to buy more, now they buy less). They are distinct and require different detection methods.
The terms sound similar and both involve 'change,' so beginners often conflate them. The exam specifically tests the distinction, so this is a frequent source of error.
Mistake
If the model's performance metrics (accuracy, precision) are still high, it means no drift has occurred.
Correct
Drift can occur silently before it affects overall metrics, especially if the model is making confident wrong predictions on a small subset of data. Monitoring for drift in features is a proactive measure that catches problems before accuracy drops noticeably.
Beginners tend to only check final metrics, not the underlying data distributions. They assume high accuracy means everything is fine, ignoring that the model may be overconfident in wrong patterns.
Mistake
Remediation always means retraining the entire model from scratch with all historical data.
Correct
Remediation can be incremental. Depending on the situation, you might retrain only with recent data, use online learning to update the model continuously, or even adjust the decision threshold without retraining. Full retraining from scratch is just one option.
People assume 'fix' means 'start over' because that is the most intuitive approach. They do not realise that model retraining can be targeted and efficient.
Mistake
Amazon CloudWatch is the correct service for detecting model drift.
Correct
CloudWatch is used for monitoring infrastructure metrics like CPU usage and latency. Model drift is detected by Amazon SageMaker Model Monitor, which performs statistical tests. CloudWatch can host alerts triggered by Model Monitor's output, but it does not do the drift detection itself.
Beginners see 'monitoring' and think of CloudWatch because it is a general-purpose monitoring tool. They do not distinguish between infrastructure monitoring and model-specific statistical monitoring.
Reveal each answer, then mark whether you got it right. Score 60%+ to unlock the next chapter.
Data drift is when the input data distribution changes (e.g., average customer age increases). Concept drift is when the relationship between the input and the output changes (e.g., older customers used to buy more, now they buy less). The exam tests your ability to distinguish these scenarios.
You use Amazon SageMaker Model Monitor. It runs statistical tests on live data and compares them to a baseline from the training data. Amazon CloudWatch is used for infrastructure monitoring, not for drift detection.
Not necessarily. Drift requires investigation first. Sometimes the drift is temporary or due to a data quality issue that can be fixed without retraining. However, the most common remediation is retraining the model with recent data.
A baseline is a set of statistical summaries (mean, variance, distributions) computed from the training dataset. SageMaker Model Monitor uses the baseline as a reference to detect when new data deviates significantly from the data the model was trained on.
It works for both. You can configure Model Monitor for a real-time endpoint (where the model answers individual requests) and for batch transform jobs (where the model processes a large file). The exam may ask about either scenario.
You use Amazon SageMaker Pipelines to create a workflow that is triggered by a CloudWatch Alarm. The pipeline loads new data, trains a model, evaluates it, and deploys it if it performs better. This automates the remediation step.
You've finished Monitoring and Remediating Models in Production. Continue through the MLS-C01 study guide to build a complete picture of the exam.
Done with this chapter?