Machine learning model monitoring and alerting is a system that constantly checks whether a deployed model is still performing well and notifies you if something goes wrong. This matters for the PMLE exam because Google expects you to know how to detect when a model's predictions become unreliable due to changes in the real world, and how to set up the automated warnings that let a team respond before the business is impacted.
Jump to a section
A simple way to picture Monitoring ML Models and Setting Up Alerts
Your kitchen, late at night. You have just put a frozen pizza in the oven and walked away, trusting the timer you set. While the pizza bakes, you want to be confident that nothing is burning, that the temperature is right, and that you will be alerted the moment something goes wrong.
In this kitchen, the smoke alarm is your monitoring system. It does not just sit silently and hope for the best; it continuously checks the air for smoke particles, which is its version of 'model performance'. If the air is clear but the oven is actually on fire because the thermostat failed, the smoke alarm will miss the problem — that is 'drift', where the real-world conditions (a fire without visible smoke) have changed from what the alarm expects (smoke means fire). To catch this, you might install a heat detector next to the stove. That heat detector is an alert set up to watch a specific metric (temperature) that is more sensitive to the underlying issue.
Now, imagine your smoke alarm goes off because someone burnt toast. That is a 'false positive' alert — a warning that something is wrong when it is not. If you ignore it, you risk missing a real fire. If you disable the alarm entirely because of false alarms, you risk a real disaster. So, you adjust the sensitivity and set different rules: a small amount of smoke for ten seconds triggers a 'warning' (like a low-priority alert), but heavy smoke for thirty seconds triggers the full alarm (a critical alert). This is exactly how you monitor a machine learning model: you define the metrics you care about (like prediction accuracy or data quality), set thresholds for when those metrics indicate a problem (drift or skew), and configure different alerting rules so you get notified at the right level of urgency.
The final piece is the backup battery. If the power goes out, your smoke alarm cannot work. In monitoring, this is like having fallback alert channels — email, SMS, and a messaging app like Slack — so even if one system fails, you still get the warning. You also periodically test the alarm (testing your monitoring setup) to verify it still works as expected. Without all these pieces, you are just hoping nothing burns.
When you deploy a machine learning model — meaning you put it into a live production environment where it starts making real predictions for users — the model does not stay static. The world changes. People change their behaviour. New data arrives. Over time, the model that once worked perfectly can start to make worse and worse predictions. This is the fundamental problem that model monitoring and alerting solves: it gives you eyes on your model after it has left the training room.
First, you must understand what you are actually tracking. The most important concept is 'model performance'. This is a set of numbers that tell you how accurate, precise, or reliable the model's predictions are. For example, if your model predicts whether an email is spam, performance might be measured by what percentage of real spam it actually catches. You cannot just check this once; you need continuous measurements because performance degrades gradually.
The two biggest threats to a model after deployment are 'data drift' and 'concept drift'. Data drift is when the statistical properties of the input data change. Imagine you trained a model on housing prices using data from 2020, but now it is 2024 and average house prices have risen. The model was built on old numbers — that is data drift. Concept drift is when the relationship between the input data and the output prediction changes. For instance, after a pandemic, people's shopping habits changed; buying patterns that used to predict 'will buy winter coat' now no longer work because many people work from home. The 'concept' of what drives a purchase shifted.
Another critical type is 'skew'. Skew happens when the data your model sees in production is different from the data it was trained on. This can be a subtle difference, like a new category appearing in a feature (e.g., a new product type in an e-commerce catalogue that the model never saw before). Skew often happens because of a bug in the data pipeline. For example, if a field that used to contain 'male/female' now contains 'M/F' due to a formatting change, the model gets confused.
Now, how do you actually monitor these things? You set up 'metrics' and 'thresholds'. A metric is a specific measurement you collect, such as 'average prediction confidence', 'number of predictions per hour', 'mean absolute error', or 'distribution of feature A'. A threshold is a limit you set. For example, 'if the mean absolute error exceeds 0.05, generate an alert'. Choosing the right thresholds is hard — set them too tight and you get endless false alarms; set them too loose and you miss real problems.
Once you have metrics and thresholds, you need 'alerting channels'. These are the ways you get notified: email, SMS, a platform like PagerDuty, or a messaging app like Slack. You also need 'alert severity levels'. A 'warning' might be for a slight performance dip that just needs investigation. A 'critical' alert might be for when the model has completely stopped working or is producing dangerous predictions.
A key part of the monitoring system is a 'baseline'. Before you can detect drift, you need to know what 'normal' looks like. You establish a baseline by analysing the data used to train the model. Then, for every batch of new data that comes in, you compare its statistical distribution to that baseline. Common statistical tests used for this include the Kolmogorov-Smirnov test (for continuous features) and the Chi-squared test (for categorical features). These tests output a p-value or a score that tells you how likely it is that the new data is different from the old data. If the p-value drops below a threshold (e.g., 0.05), you have detected statistically significant drift.
Let us look at a real example. Suppose you have a model that predicts loan default risk. The training data had a feature called 'credit score' ranging from 300 to 850, with an average of 700. After deployment, you start receiving production data where the average credit score is 750. This is a shift in the input distribution — data drift. The model, which learned patterns based on a 700 average, might now overestimate risk because it is seeing healthier scores than it was trained on. Your monitoring system can automatically detect this shift by comparing the monthly average credit score against the baseline. If the difference exceeds your threshold, an alert fires.
Automated alerting rules are typically configured in a platform like Google Cloud's 'Model Monitoring' (part of Vertex AI) or using general cloud monitoring tools like Cloud Monitoring and Cloud Logging. You can set up scheduled jobs that run every hour, day, or week to compute the metrics and compare them. When an alert fires, you want it to contain enough information for someone to act: what metric changed, by how much, which model version is affected, and a link to the data.
Finally, monitoring is not just about detecting problems — it is about triggering a response. Common responses include retraining the model on newer data, rolling back to a previous model version, or investigating and fixing a data pipeline bug. Without monitoring, you only find out about failures when a user complains, which means you already lost money or reputation. With it, you catch issues proactively.
Establish a Baseline
Before you can detect drift, you need to know what 'normal' looks like. This involves analysing the distribution of every feature in the training dataset and recording key statistics (mean, standard deviation, histograms). This baseline is the reference against which all future production data is compared.
Select Metrics and Thresholds
Identify which metrics you will track, such as feature distribution distances, prediction accuracy, or data completeness. For each metric, set an initial threshold (e.g., a maximum allowed distance or a minimum allowed accuracy). These thresholds determine when an alert triggers.
Configure Monitoring Jobs
Set up a scheduled job in a cloud platform (like Vertex AI Model Monitoring or Cloud Monitoring) that runs at regular intervals (e.g., every hour, day, or week). This job pulls the latest batch of production data and compares it to the baseline using the selected statistical tests.
Define Alerting Channels and Severity Levels
Decide how you want to be notified when an alert fires. Common channels include email, SMS, Slack, and PagerDuty. Assign severity levels: a 'Warning' for borderline drift that needs investigation, a 'Critical' for severe performance degradation that requires immediate action.
Set Up a Response Workflow
Create a documented plan for what to do when an alert fires. This might include steps like: acknowledge the alert, inspect the relevant dashboards, decide whether to rollback or retrain the model, and communicate the incident to the team. Automate simple responses if possible, such as automatically rolling back to a healthy model version when a critical alert fires.
Review and Tune Alerting Rules Periodically
After deployment, monitor the alerting system itself. Are there too many false positives? Are real problems being missed? Adjust thresholds, add new metrics, or suppress noise based on observed patterns. This step happens weekly or monthly as part of normal operations.
An IT professional responsible for a machine learning model does not simply deploy it and walk away. They operate something like a watchtower. Let us walk through a concrete scenario involving a large e-commerce company, 'ShopStream', which uses a model to predict which products a customer is most likely to buy next (a recommendation model).
The model was deployed three months ago and has been doing well. Then, one Tuesday, a data engineer notices a spike in 'add to cart' but no corresponding spike in purchases. This might not be a model issue, but the engineer thinks, 'what if the model is starting to recommend items people do not actually want?' This is where monitoring kicks in.
The engineer opens a dashboard built in Vertex AI Model Monitoring. They see a panel labelled 'Feature Distribution Drift'. It shows that for the feature 'product category', the distribution has shifted. Specifically, a new product category called 'home office fitness equipment' has appeared in the data, which never existed in the training set. The model has literally never seen this category code before. - First, the engineer checks the 'Skew Detection' tab. It shows a red alert for the 'product_category' feature, indicating a high skew score. The system uses a technique called 'L-infinity distance' to compare the training and serving distributions, and the distance is above the alert threshold of 0.1. - Second, the engineer checks 'Performance Metrics'. The model's click-through rate (how often a recommended product is clicked) has dropped from 8% to 4%. This is a massive drop, triggering a 'Critical' alert. - Third, the engineer looks at the alert history. They see the first alert for 'feature skew' fired 12 hours ago, followed by the performance drop alert 2 hours ago. This tells a story: the skew caused the performance drop. - Fourth, the engineer needs to decide what to do. They open the model's log (stored in Cloud Logging) and look at the actual predictions made for customers. They notice that for users who browsed the 'home office' category, the model was recommending generic 'office chairs' instead of the new 'home office fitness equipment'. The model simply did not know that category existed, so it fell back to the most similar old category.
Now, the engineer takes action. They cannot retrain the model immediately because new training data with the correct labels is not ready yet. So, they use a technique called 'rollback' — they switch the production traffic back to the previous model version (the one from two months ago that still works, because the 'home office fitness equipment' category did not exist then so the model was not confused). This is a temporary fix.
Simultaneously, they create a new alerting rule specifically for 'novel categorical values', so that the next time a new category appears, they get an immediate 'Warning' alert (not Critical) so they can investigate before performance drops.
Finally, they document the incident. They note that the monitoring system's 'skew detection' caught the issue 10 hours before the performance drop became critical. If they had acted on the skew alert immediately, they could have avoided the performance drop entirely. This is the core value of proactive monitoring.
In a larger enterprise, this process would be automated further. For example, they might have a 'remediation pipeline' that automatically triggers retraining when drift is detected, but only if the drift exceeds a second, higher threshold. The monitoring system itself would be configured to send alerts to different teams: a 'Warning' goes to a Slack channel for the data science team, a 'Critical' alert pages the on-call engineer via PagerDuty. All these configurations are part of a 'monitoring plan' that the IT professional maintains.
The Google Professional Machine Learning Engineer exam is not a theoretical test. It is a practical, scenario-based exam that expects you to understand exactly when and why to monitor models. For objective 4.2, the exam is laser-focused on three things: recognising the difference between drift, skew, and performance degradation, selecting the right monitoring tools, and designing alerting strategies that minimise noise.
First, the exam loves to test definitions. You will get a scenario description and then a question like 'What type of issue is this?' where the options are 'data drift', 'concept drift', 'skew', or 'model staleness'. Common trap: they describe a situation where only the input data distribution changes (data drift) but the relationship between input and output stays the same. Many candidates confuse data drift with concept drift. The key is: if the input data's statistics change but the model's logic still works, it is data drift. If the underlying relationship changes (e.g., what makes a customer churn changes), it is concept drift.
Second, they test your ability to choose the right monitoring approach. They might ask: 'You want to detect if the distribution of a numerical feature has changed. Which statistical test should you use?' Options: Kolmogorov-Smirnov test (correct for continuous/numerical), Chi-squared test (correct for categorical), or t-test (often used for mean comparison, but not specifically for distribution shape). Trap: they might list 't-test' as a valid option, but it is not the standard test for distribution drift — KS is the go-to for numerical features.
Third, they test alerting strategy. Expect questions like: 'Your model is performing well but you are getting a high volume of false positive alerts. What should you do?' The correct answer is always to adjust the alerting threshold or use a more sophisticated drift detection method with a higher confidence interval. Wrong answers include: disabling monitoring entirely, or ignoring the alerts.
Key concepts to memorise for the exam:
The difference between 'online' and 'batch' monitoring. Online monitoring checks every prediction in real time (low latency, high cost). Batch monitoring checks data in chunks (e.g., once a day, cost effective, but delayed detection). The exam will test which is appropriate for a given scenario. For example, a fraud detection model needs online monitoring; a recommendation model can use batch.
The concept of a 'baseline window'. To detect drift, you need a reference period. The exam expects you to know that you should use a sufficiently large window (e.g., 7 days of data) to establish a stable baseline, not just the training data.
'Output monitoring' vs 'input monitoring'. Input monitoring checks the features coming into the model. Output monitoring checks the predictions leaving the model. The exam might describe a scenario where you only monitor input and miss a problem in the output (or vice versa). You need to monitor both.
'Model retraining triggers'. The exam might ask what to do after drift is detected. The answer is either 'retrain the model' or 'rollback to a previous version', but they will test if you know which is better in which context. Retraining is better if you have new labelled data; rollback is better if the current model is clearly broken and you have a known good earlier version.
'Data quality monitoring' is a subtopic. They will ask about checking for missing values, nulls, or out-of-range values in incoming data. This is separate from drift but often tested together.
Common exam traps:
The exam says 'The model's accuracy is dropping, but the input data distribution has not changed.' The trap answer is 'skew' — but that is wrong. Accuraccy dropping with no input change means concept drift, because the relationship changed.
They say 'You detect that the average value of a feature has shifted by 10%. Is this drift?' The answer is: not necessarily. Drift is about a shift in the entire distribution, not just the mean. You need a statistical test.
They ask about 'alert fatigue' and suggest disabling alerts. The PMLE expects you to know that the correct solution is to adjust thresholds or implement a tiered alerting system (warnings vs critical), not to turn off alerts.
They present a scenario where you have a model that monitors itself (auto-ML) and ask if you still need monitoring. The answer is yes — all models need external monitoring because self-monitoring only covers a subset of possible failures.
Prepare by practising with scenario-based questions. The official Google practice exams are helpful. Focus on the 'why' behind each tool, not just memorising product names.
Model monitoring is not optional; it is required because the real-world data that models see after deployment inevitably changes over time.
Data drift refers to changes in the statistical distribution of input features, while concept drift refers to changes in the relationship between inputs and outputs.
Skew is a specific type of drift that happens when the distribution of production data differs from the training data, often due to a data pipeline error.
Alerting thresholds must be carefully tuned to balance catching real problems against generating too many false positive alerts that cause alert fatigue.
You must monitor both the input features and the model's output predictions to get a complete picture of model health.
When drift is detected, the appropriate response is either to retrain the model on newer data or to rollback to a previously validated model version.
Statistical tests like the Kolmogorov-Smirnov test for numerical features and the Chi-squared test for categorical features are the standard tools for detecting distribution drift.
A robust monitoring strategy includes a baseline, periodic evaluation schedules, defined alert severity levels, and fallback communication channels.
These come up on the exam all the time. Here's how to tell them apart.
Data Drift
Change in the statistical distribution of input features (e.g., average age shifts).
Detected by comparing feature distributions using KS or Chi-squared tests.
Response often involves retraining the model on newer data that reflects the new distribution.
Concept Drift
Change in the relationship between inputs and the target variable (e.g., what makes a customer churn changes).
Detected by monitoring the model's performance metrics (accuracy, precision) over time.
Response may require feature engineering or model architecture changes, not just retraining.
Skew
Specifically compares production data distribution to the training data distribution.
Often caused by a data pipeline bug or a encoding mismatch.
Resolved by fixing the data pipeline or re-encoding features.
Data Drift (general)
Broadly refers to any change in the data distribution over time, not necessarily relative to training.
Can be gradual or abrupt, caused by real-world changes (e.g., seasonal trends).
Resolved by retraining the model on the most recent data.
Online Monitoring
Checks every prediction as it happens, in real-time.
Low latency but high computational cost.
Suitable for low-latency applications like fraud detection.
Batch Monitoring
Checks data in defined intervals (e.g., hourly or daily).
Lower cost but introduces a delay in detecting issues.
Suitable for applications where performance degradation is not immediately critical, like recommendation systems.
Input Monitoring
Monitors the features (attributes) entering the model, such as age, price, or category.
Detects data drift and skew by comparing feature distributions.
Validates data quality (missing values, out-of-range values).
Output Monitoring
Monitors the predictions (outputs) the model produces, such as predicted class or score.
Detects concept drift and overall performance degradation by tracking model accuracy, precision, or confidence.
Validates that predictions make business sense (e.g., no negative prices).
Retraining the Model
Involves training a new version of the model on the most recent labelled data.
Preferred when there is enough new, high-quality labelled data that reflects the current environment.
Addresses both data drift and concept drift by incorporating new patterns.
Rolling Back the Model
Involves reverting to a previously deployed, known-good version of the model.
Preferred as a temporary fix when the current model is clearly broken but new labelled data is not available.
Only a short-term solution; the root cause must still be addressed.
Mistake
Once a model is deployed and performing well, you do not need to monitor it unless there is a visible problem.
Correct
Models degrade over time due to changes in the real world, often before any visible performance drop. Continuous monitoring catches drift early, before it impacts users or revenue.
This mistake is common because beginners assume a machine learning model works like a traditional software program: if it works once, it works forever. But models are data-dependent and the data changes.
Mistake
Data drift and concept drift are the same thing, just worded differently.
Correct
Data drift is a change in the input data's statistical properties. Concept drift is a change in the relationship between the input and the output. They require different detection methods and solutions.
The terms sound similar, and many introductory resources do not clearly separate them. The exam deliberately tests the distinction.
Mistake
If you monitor the model's accuracy, that is enough to detect all problems.
Correct
Accuracy alone tells you that something is wrong, but not what is wrong. You need to monitor input feature distributions, output distributions, and data quality separately to diagnose the root cause.
Accuracy is an easy metric to understand, so beginners focus on it. But it is a lagging indicator — by the time accuracy drops, the problem has already caused harm.
Mistake
Setting up alerts is just about choosing a threshold; once set, it never needs changing.
Correct
Alerting thresholds should be reviewed and adjusted regularly based on the model's performance and business context. False positive rates change, and the cost of missing a real problem may change over time.
This mistake comes from thinking of alerts as a one-time configuration, like setting a smoke alarm. But a model's environment (e.g., holiday season shopping data) can temporarily shift, requiring temporary threshold adjustments.
Mistake
Skew is the same as data drift.
Correct
Skew specifically refers to a difference between the training data distribution and the serving data distribution. Data drift can occur after the model is deployed without any reference to the training set (e.g., a gradual shift over months). Skew is a type of drift, but drift is broader.
The terms overlap in common use, but the exam treats them as distinct concepts. Beginners often conflate them because both involve changes in data.
Reveal each answer, then mark whether you got it right. Score 60%+ to unlock the next chapter.
Data drift is when the statistical properties of the input data change (e.g., average user age shifts from 30 to 40). Concept drift is when the relationship between the input and the target variable changes (e.g., older users used to buy more, but now younger users buy more). Both require different monitoring and response strategies.
You can use Vertex AI Model Monitoring, which offers pre-built drift and skew detection. You configure a monitoring job by specifying your model's endpoint, the training data as a baseline, and which features to monitor. You can also use Cloud Monitoring to set custom metrics and alerts.
Skew is a specific type of data drift where the distribution of features in the production (serving) data differs from the distribution the model was trained on. It often results from a data pipeline bug, such as a feature being encoded differently or missing values being handled inconsistently.
It depends on how quickly the data changes and the cost of a bad prediction. For fast-moving data like fraud detection, run checks every few minutes or in real-time. For more stable data like a recommendation model, running checks once a day or once a week is usually sufficient.
The Kolmogorov-Smirnov (KS) test is commonly used for numerical (continuous) features. The Chi-squared test is used for categorical features. These tests compare the cumulative distribution of the production data against the baseline distribution and output a score indicating how likely the difference is due to chance.
First, acknowledge the alert to prevent it from escalating. Then, inspect the monitoring dashboard to understand which metric triggered the alert and how much it changed. Look at both input features and output predictions to differentiate between data drift, concept drift, or a data quality issue. Based on the diagnosis, decide whether to retrain, rollback, or investigate a pipeline problem.
No. A stable period does not guarantee future stability. The real world can change abruptly (e.g., a new competitor, a policy change, a seasonal shift). Monitoring should be continuous; you can adjust thresholds to reduce noise, but never disable it entirely.
You've finished Monitoring ML Models and Setting Up Alerts. Continue through the PMLE study guide to build a complete picture of the exam.
Done with this chapter?