PMLE Monitoring ML Solutions Practice Question

A company uses Vertex AI Model Monitoring on an Endpoint that serves a regression model. They configure monitoring for both feature skew and prediction drift with a 10% threshold. After a week, they receive an alert that prediction drift exceeds the threshold, but feature skew remains below threshold. They want to understand what this indicates about the model's performance. What should they conclude?

⚠ Common exam trap

The trap here is assuming that any drift alert means input data has changed, ignoring that prediction drift can occur independently of feature skew.

Answer choices

Why each option matters

Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.

Correct answer & explanation

✓

The model's predictions have changed distribution over time, which may indicate degradation in model performance.

Prediction drift monitors changes in the distribution of model outputs over time. When prediction drift exceeds a threshold while feature skew remains low, it suggests that the model's predictions are shifting even though input features are stable. This can indicate concept drift, where the relationship between features and target changes, potentially degrading model performance. The appropriate response is to investigate model performance and consider retraining.

Answer analysis

Option-by-option breakdown

For each option: why learners choose it and why it is or isn't the right answer here.

✗
The model is performing well because prediction drift is expected as new data arrives.
Why it's wrong here
Prediction drift exceeding a configured threshold is not a sign of good performance; it indicates that the output distribution has changed beyond acceptable limits. While some drift is normal, an alert means the change is significant enough to warrant investigation. Assuming it indicates good performance ignores the purpose of monitoring and could lead to unnoticed degradation.
✗
The model's input feature distributions have shifted significantly compared to training.
Why it's wrong here
Feature skew measures input distribution changes relative to training data. Since the alert indicates feature skew is below threshold, input features have not shifted significantly. Therefore, this conclusion contradicts the monitoring results. The drift is in the predictions, not the inputs, so this option misinterprets the alert.
✗
The training data used for the skew baseline is no longer valid, causing the prediction drift alert.
Why it's wrong here
Prediction drift is computed independently of the training data baseline; it compares recent predictions to a reference window of past predictions. Therefore, an invalid training baseline would affect feature skew, not prediction drift. Since feature skew is below threshold, the training baseline is likely still valid, and this option incorrectly links the two metrics.
✓
The model's predictions have changed distribution over time, which may indicate degradation in model performance.
Why this is correct
Prediction drift measures changes in the distribution of the model's outputs over time. An alert on prediction drift alone, with no feature skew, suggests that the model's predictions are shifting even though inputs appear stable. This can happen due to concept drift or model degradation, where the relationship between inputs and outputs changes. It signals a potential need to investigate model performance or retrain.

Go deeper

PMLE Monitoring ML Solutions Practice Question

Why each option matters

Option-by-option breakdown

Related to this question