PMLE Monitoring ML Solutions Practice Question

A media streaming company uses a recommendation model deployed on Vertex AI Endpoints. The model predicts whether a user will click on a recommended item. They have set up Vertex AI Model Monitoring with a training dataset and configured drift detection for both features and predictions. Recently, they observed that the prediction drift metric (Jensen-Shannon divergence) for the 'click' prediction has exceeded the threshold, but feature drifts are within normal ranges. What is the most likely cause of this prediction drift?

⚠ Common exam trap

The trap here is to assume that prediction drift must be accompanied by feature drift, overlooking concept drift as a distinct phenomenon.

Answer choices

Why each option matters

Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.

Correct answer & explanation

✓

A change in the relationship between features and the target variable (concept drift) that is not captured by feature drift alone.

Prediction drift with stable feature distributions indicates concept drift, where the relationship between features and the target has changed. This is common in dynamic environments like media streaming, where user preferences evolve. Detecting this early allows for timely model updates.

Answer analysis

Option-by-option breakdown

For each option: why learners choose it and why it is or isn't the right answer here.

✓
A change in the relationship between features and the target variable (concept drift) that is not captured by feature drift alone.
Why this is correct
Prediction drift can occur even when feature distributions remain stable if the underlying relationship between features and the target changes. This is known as concept drift. For example, user preferences may shift such that the same feature values now lead to different click behavior. Monitoring prediction distribution helps detect such changes, which may require model retraining or adaptation.
✗
A bug in the Vertex AI Model Monitoring configuration that incorrectly computes the prediction drift metric.
Why it's wrong here
While bugs are possible, they are less likely than genuine concept drift. Vertex AI Model Monitoring is a managed service with well-tested drift metrics. If feature drifts are normal, the prediction drift is likely a true signal of change in the target relationship. Assuming a bug without evidence could lead to ignoring a real issue.
✗
An increase in the overall volume of prediction requests, leading to a higher variance in the prediction distribution.
Why it's wrong here
An increase in request volume does not inherently change the prediction distribution; it would only affect the sample size and potentially the stability of the metric. The distribution itself (the proportion of clicks) would remain the same if the underlying model and data relationships are unchanged. Thus, volume increase is not a likely cause of prediction drift.
✗
A data pipeline error that has corrupted the feature values, causing the model to produce different predictions.
Why it's wrong here
A data pipeline error would typically cause feature drift as well, because the corrupted feature values would deviate from the training distribution. Since feature drifts are within normal ranges, a pipeline error is less likely. Prediction drift without feature drift points more to concept drift rather than data corruption.

Go deeper

PMLE Monitoring ML Solutions Practice Question

Why each option matters

Option-by-option breakdown

Related to this question