PMLE Practice Question: Monitoring a production model that is…

You are monitoring a production model that is experiencing gradual decay in AUC. Which THREE metrics should you set up alerts for to diagnose the root cause? (Choose three.)

⚠ Common exam trap

Google Cloud often tests the distinction between metrics that indicate a symptom (e.g., latency, staleness) versus metrics that directly measure the cause of performance decay (drift scores), leading candidates to select operational metrics instead of diagnostic ones.

Answer choices

Why each option matters

Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.

Correct answer & explanation

✓

Concept drift score measured by comparing predicted vs actual outcomes.

Option A is correct because concept drift — measured by comparing predicted values against actual outcomes — directly captures the degradation in the input-to-target relationship that would cause AUC decay. Option B is correct because training-serving skew in high-importance categorical features means the model is receiving feature values at inference that differ from those seen during training, which degrades ranking quality and AUC. Option D is correct because feature drift in key numerical features signals that the input distribution has shifted, a common root cause of gradual AUC decay. Option C is not correct because average prediction latency is an operational performance metric, not a model-quality diagnostic for AUC decay. Option E is not correct because model staleness is a coarse proxy for retraining cadence and does not by itself diagnose the root cause of AUC degradation.

Answer analysis

Option-by-option breakdown

For each option: why learners choose it and why it is or isn't the right answer here.

✓
Concept drift score measured by comparing predicted vs actual outcomes.
Why this is correct
Concept drift compares predicted outcomes against actual labels, revealing whether the relationship the model learned has changed. Since AUC decay stems from that relationship shifting, alerting on concept drift pinpoints the root cause rather than merely the symptom.
✓
Training-serving skew for categorical features with high importance.
Why this is correct
Training-serving skew for high-importance categorical features directly exposes distribution drift between the training data and live inference requests, which is a leading cause of gradual AUC decay. Alerting on this metric isolates whether feature preprocessing or category vocabulary mismatches are degrading predictions, satisfying the need to diagnose the root cause.
✗
Average prediction latency over the past hour.
Why it's wrong here
Latency measures serving speed, not predictive quality, so it cannot explain a falling AUC. It is tempting because latency alerts are standard production monitoring, and they would be correct for diagnosing throughput or infrastructure regressions, but not gradual accuracy decay.
✓
Feature drift score for key numerical features.
Why this is correct
Feature drift scores for key numerical features quantify how far live input distributions have moved from the training baseline. Alerting on these scores flags the input shift responsible for gradual AUC decay before it compounds further.
✗
Model staleness (days since last retraining).
Why it's wrong here
Staleness is a plausible cause of decay, but it is a training-pipeline signal rather than a diagnostic metric of the model's input or output distribution. It is tempting because retraining cadence correlates with drift, yet feature drift, label drift and prediction distribution are the metrics to alert on.

Go deeper

PMLE Practice Question: Monitoring a production model that is…

Why each option matters

Option-by-option breakdown

Related to this question