PMLE Practice Question: A company deploys a custom ML model on Vertex AI…

A company deploys a custom ML model on Vertex AI to predict customer churn. The model retrains weekly, and predictions are served via a Vertex AI endpoint. After a recent retraining, the monitoring dashboard shows a sudden increase in prediction requests but a decrease in predicted churn probabilities. The model's accuracy on the validation set remains stable. What is the most likely cause of the observed behavior?

⚠ Common exam trap

Google Cloud often tests the distinction between covariate shift (data distribution change) and concept drift (relationship change), trapping candidates who assume any change in predictions must be due to model degradation or data leakage.

Answer choices

Why each option matters

Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.

Correct answer & explanation

✓

The incoming data distribution has changed, e.g., due to a new marketing campaign attracting different customers.

A sudden increase in prediction requests alongside a decrease in predicted churn probabilities, while validation accuracy remains stable, indicates a shift in the incoming data distribution (covariate shift). This is typical when a new marketing campaign attracts a different customer segment that inherently has lower churn risk. The model itself hasn't degraded; it's simply seeing a different population than it was trained on, which changes the base rate of churn in the live traffic.

Answer analysis

Option-by-option breakdown

For each option: why learners choose it and why it is or isn't the right answer here.

✗
A training-serving skew exists between the training pipeline and the serving endpoint.
Why it's wrong here
Training-serving skew would depress validation accuracy too, since the same transformed features feed both paths; here validation accuracy holds steady while live prediction volume rises. It is tempting because skew genuinely causes silent degradation, but only when training and serving pipelines apply divergent transformations — not when both share one pipeline.
✗
Concept drift has occurred, changing the relationship between features and churn.
Why it's wrong here
Concept drift alters the feature-to-label relationship, which would degrade validation accuracy once labels arrive; the stem reports stable validation accuracy, so the input-output mapping is unchanged. Concept drift is the right diagnosis when accuracy decays over time without any request-volume change.
✓
The incoming data distribution has changed, e.g., due to a new marketing campaign attracting different customers.
Why this is correct
A covariate shift in the live feature distribution explains both symptoms: new marketing-driven customers produce different input patterns, inflating request volume while shifting predicted probabilities downward. Validation accuracy stays stable because the validation set still reflects the old distribution, so the drift is invisible there.
✗
Data leakage during training caused the model to overfit to historical patterns.
Why it's wrong here
Leakage inflates validation accuracy rather than leaving it stable, and it would not raise live request volume. Leakage is the correct finding when offline metrics look implausibly strong yet production performance collapses — a signature absent from this scenario.

Go deeper

PMLE Practice Question: A company deploys a custom ML model on Vertex AI…

Why each option matters

Option-by-option breakdown

Related to this question