PMLE Monitoring ML Solutions Practice Question

A media company uses a Vertex AI Endpoint to serve a video recommendation model. They have enabled Vertex AI Model Monitoring for prediction drift. After a major news event, they observe a significant increase in prediction drift alerts, but the model's recommendations remain relevant and user engagement is stable. They want to reduce unnecessary alerts without losing the ability to detect true model degradation. What should they do?

⚠ Common exam trap

The trap here is treating all prediction drift alerts as indicators of model failure, when legitimate external events can cause distribution shifts that require threshold recalibration rather than disabling monitoring.

Answer choices

Why each option matters

Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.

Correct answer & explanation

✓

Increase the prediction drift threshold to a higher value based on historical variance.

Prediction drift alerts are based on changes in the model's output distribution. A major news event can legitimately shift user behavior and thus prediction distributions without degrading model quality. Raising the drift threshold based on historical variance reduces false alerts while preserving the ability to detect significant deviations. Disabling monitoring, changing frequency, or switching to skew detection do not appropriately address the root cause.

Answer analysis

Option-by-option breakdown

For each option: why learners choose it and why it is or isn't the right answer here.

✗
Switch from prediction drift to training-serving skew monitoring.
Why it's wrong here
Training-serving skew measures input feature differences, not output prediction changes. The issue here is that prediction distributions shifted due to a news event, which is better captured by prediction drift. Switching to skew would not help reduce false alerts and might miss the output changes entirely. The team should keep prediction drift but calibrate its threshold.
✗
Reduce the monitoring frequency from daily to weekly to decrease the number of alerts.
Why it's wrong here
Changing frequency affects how often drift is evaluated, not the threshold for alerting. If drift is high, weekly checks would still trigger alerts, just less frequently. This does not address the root cause of false positives due to legitimate distribution shifts. It also delays detection of real drift. The correct fix is to adjust the threshold to account for expected variance.
✗
Disable prediction drift monitoring and rely on business metrics instead.
Why it's wrong here
Disabling prediction drift monitoring removes an automated safeguard that can detect model degradation early. Business metrics like user engagement may lag and not provide timely alerts. While the current alerts are false positives, turning off monitoring entirely is overkill and could leave the team blind to future genuine drift. A better approach is to tune the threshold rather than disable the feature.
✓
Increase the prediction drift threshold to a higher value based on historical variance.
Why this is correct
Prediction drift alerts fire when the distribution of model outputs changes beyond a threshold. A major news event can cause a legitimate shift in user behavior and thus in prediction distributions, without indicating model degradation. Raising the threshold based on observed historical variance reduces false alerts while still catching significant deviations that could signal real problems. This balances sensitivity and specificity for the current environment.

Go deeper

PMLE Monitoring ML Solutions Practice Question

Why each option matters

Option-by-option breakdown

Related to this question