Monitoring ML solutions practice questions

Monitor prediction drift (prediction distribution)

Why wrong: Prediction drift indicates output change but not which feature drifted.

Monitor feature attribution drift using SHAP values

Why wrong: Feature attribution drift measures impact on predictions, not input distribution.

Monitor residual distribution drift

Why wrong: Residuals measure error, not input drift.

A sudden increase in the number of prediction requests

Why wrong: Request rate is stable, so not the cause.

The model was replaced with a larger version without updating the endpoint

Why wrong: Model replacement requires redeployment; latency change would be immediate, not gradual.

A misconfiguration in the autoscaling policy

Why wrong: Autoscaling issues typically cause errors under load, not gradual latency increase with stable traffic.

Monitor the accuracy of the model on the latest batch of labeled data

Why wrong: Labels are often delayed; early detection is not possible.

Monitor feature distribution drift using KS test

Why wrong: Feature drift may not capture changes in the relationship between features and target.

Monitor the freshness of the training data

Why wrong: Freshness alone does not indicate concept drift.

Increase in model size due to retraining

Why wrong: Model is served elsewhere; batch pipeline typically runs predictions using the same model.

Decrease in the number of worker machines

Why wrong: Fewer workers would cause slower processing, not OOM.

Increase in input data size

Why wrong: Data volume has not changed.

p50 latency of prediction requests

Why wrong: Median latency does not capture worst-case.

Prediction drift score

Why wrong: Unrelated to latency SLA.

Number of prediction requests per second

Why wrong: Request count does not measure latency.

The model file is corrupted

Why wrong: Would likely cause loading errors.

The model file was accidentally replaced with a different model

Why wrong: Would likely cause different but not necessarily negative probabilities.

The container is using an incompatible version of the serving framework

Why wrong: Would likely cause errors or crashes, not silent wrong predictions.

Switch from BigQuery to Cloud Storage for data source

Why wrong: Changing source may not address the underlying query issue.

Increase the memory allocated to the pipeline step

Why wrong: Memory increase does not affect BigQuery resource limits.

Reduce the batch size of the data being read

Why wrong: Batch size may not be the issue if the query itself is large.

Check the monitoring dashboard configuration

Why wrong: Dashboard does not affect latency.

Check if the feature engineering logic was changed

Why wrong: Feature engineering change would be part of model update.

Check the geographic location of the endpoint

Why wrong: Location affects network latency, not compute latency.

Model accuracy on recent labeled data

Why wrong: Labels are delayed; not suitable for early drift detection.

Model prediction latency

Why wrong: Latency is a performance metric, not drift.

Training data size

Why wrong: Unrelated to drift.

Team member roles and responsibilities

Why wrong: Not relevant to monitoring.

Training pipeline code version

Why wrong: Not a runtime monitoring metric.

Immediately roll back to the previous model version

Why wrong: Rolling back may not help if drift is ongoing.

Increase logging for future predictions

Why wrong: Only helps with diagnosis, not correction.

Reduce the traffic to the model to minimize impact

Why wrong: Does not address root cause.

The autoscaling policy is not scaling up fast enough, causing increased latency and prediction errors.

Why wrong: Latency increase is a symptom, but precision drop is not caused by scaling issues.

The model is overfitting to recent transaction patterns due to weekly retraining.

Why wrong: Overfitting would likely affect both precision and recall.

The model was replaced with a different version without updating the endpoint.

Why wrong: Model version is unchanged, so this is not the case.

Retrain the model daily with the latest data to adapt to changing patterns.

Why wrong: Retraining is a fix, not a diagnostic; the team should first understand why MAE increased.

Compare the distribution of training labels with serving labels using a two-sample t-test.

Why wrong: Label distribution shift can cause MAE increase, but monitoring serving MAE is more direct and actionable.

Monitor input feature distributions for drift using the Kolmogorov-Smirnov test.

Why wrong: Input drift detection is useful but does not directly measure prediction error; the complaint is about MAE increase.

The model is receiving more traffic; scale the number of replicas.

Why wrong: Traffic increase would show a sudden change, not gradual latency increase over two weeks.

The Vertex AI endpoint has changed its URL; update the client application.

Why wrong: The endpoint URL is static and does not change unless explicitly updated.

The model file size has grown due to feature engineering; reduce feature count.

Why wrong: Model file size does not change unless a new model version is deployed.

The distribution of input features has shifted significantly, causing the model to produce incorrect probabilities.

Why wrong: Feature drift can cause poor performance, but the problem statement does not mention feature drift; calibration degradation is specifically addressed.

The model overfits to noise in the training data, leading to poor generalization.

Why wrong: Overfitting would show poor test set performance, but the test set had good Brier score.

The production data has a different class imbalance than the training data, causing the model to be biased toward the majority class.

Why wrong: Class imbalance alone does not explain miscalibration; the model's probability estimates can still be calibrated if the imbalance is accounted for.

The new model artifact is temporarily unavailable, so the endpoint falls back to the previous version.

Why wrong: Vertex AI endpoints do not fall back to older versions; they only serve the version they are configured to serve.

The prediction cache is returning cached results from the old model.

Why wrong: Caching affects predictions, not the model version served; the endpoint still selects the model.

The pipeline failed to update the endpoint with the new model ID.

Why wrong: If the update failed, the old model would be served permanently, not just for a few minutes.

Pearson correlation coefficient

Why wrong: Pearson correlation measures linear relationship between two variables, not distribution drift.

Chi-squared statistic

Why wrong: Chi-squared test is for categorical features.

Population Stability Index (PSI)

Why wrong: PSI is primarily used for categorical features or binned numerical data, but not as a direct metric for continuous distributions.

Remove the drifted categories from the feature set to eliminate the alert.

Why wrong: Removing categories reduces model information and could degrade performance.

Ignore the alert because the model is performing well; monitoring alerts are often false positives.

Why wrong: Ignoring alerts is not recommended; the team should investigate to confirm it's a false positive.

Increase the minReplicaCount to 3 to handle requests in parallel.

Why wrong: Adding more replicas does not address the low GPU utilization per replica; it may spread the load but each replica still underutilizes GPU.

Reduce the CPU utilization target to 40% to trigger more aggressive autoscaling.

Why wrong: CPU utilization is not the bottleneck; lowering the target would add more replicas, not improve GPU efficiency.

Quantize the model to FP16 to reduce compute time per inference.

Why wrong: Quantization can help, but the primary issue is low GPU utilization due to small batches; optimizing batching should be the first step.

A training-serving skew exists between the training pipeline and the serving endpoint.

Why wrong: Training-serving skew would cause feature mismatches, but the volume increase and prediction shift point to covariate shift.

Concept drift has occurred, changing the relationship between features and churn.

Why wrong: Concept drift would typically degrade accuracy, which is stable here.

Data leakage during training caused the model to overfit to historical patterns.

Why wrong: Data leakage would likely cause validation accuracy to drop when exposed to new data, not remain stable.