PMLE Monitoring ML Solutions — 10 Questions

Question 1

A data scientist has deployed a model on Vertex AI Endpoints and wants to monitor the model's predictions for any drift over time. Which Vertex AI service should they use?

Accepted Answer

Vertex AI Model Monitoring. Vertex AI Model Monitoring is specifically designed to monitor deployed models for feature drift, feature skew, and prediction drift. It uses statistical methods to compare serving distributions over time or against training data.

Answer

Vertex AI Feature Store

Answer

Vertex AI Predictions

Answer

Vertex AI Explainable AI

Question 2

An MLOps engineer needs to collect ground truth labels for a deployed classification model to compare predictions against actuals. Where should the engineer store the ground truth data to enable Vertex AI model quality monitoring?

Accepted Answer

BigQuery. Vertex AI Model Monitoring expects ground truth data to be uploaded to BigQuery tables, which can then be used to compute confusion matrices and other quality metrics over time.

Answer

Firestore

Answer

Cloud Spanner

Answer

Cloud Storage

Question 3

A team is monitoring a deployed model and notices that the prediction distribution has changed significantly over the last week. They want to detect which features are contributing most to the drift. Which tool should they use?

Accepted Answer

Vertex AI Explainable AI. Vertex AI Explainable AI provides feature attributions (e.g., SHAP values) that can be used to identify which features are most important for predictions. By comparing feature importance over time, they can pinpoint which features are drifting.

Answer

Vertex AI Feature Store

Answer

Vertex AI Model Monitoring

Answer

Vertex AI Pipelines

Question 4

An engineer wants to configure alerting when the data distribution of a serving feature deviates from the training data distribution. The model is deployed on Vertex AI Endpoints. Which divergence metric should they use to compare the training and serving distributions?

Accepted Answer

Jensen-Shannon divergence. Vertex AI Model Monitoring supports Jensen-Shannon divergence for comparing distributions. It is a symmetric and bounded metric suitable for detecting feature skew between training and serving data.

Answer

Kullback-Leibler divergence

Answer

Population Stability Index (PSI)

Answer

Chi-squared test

Question 5

A team is monitoring a model on Vertex AI Endpoints and wants to track the p99 latency of online predictions. Which approach should they use to set up latency monitoring and alerting?

Accepted Answer

Use Cloud Logging to create log-based metrics from prediction logs and set up alerts in Cloud Monitoring. Vertex AI Endpoints automatically export request/response logs to Cloud Logging, which can be used to create log-based metrics for latency percentiles. These metrics can then be visualized in Cloud Monitoring dashboards and used for alerting.

Answer

Enable Vertex AI Model Monitoring and select 'latency' as a metric

Answer

Enable Vertex AI Explainable AI to output latency statistics

Answer

Configure Cloud Monitoring to scrape Prometheus metrics from the endpoint

Question 6

An ML team wants to automatically retrain a model when data drift is detected. They have set up a Cloud Monitoring alert on drift. What service should they use to trigger a retraining pipeline in response to the alert?

Accepted Answer

Cloud Functions. Cloud Monitoring alerts can send notifications to Pub/Sub topics. A Pub/Sub message can then trigger a Cloud Function or Cloud Run service that starts a Vertex AI Pipeline for retraining.

Answer

Cloud Scheduler

Answer

Vertex AI Feature Store

Answer

Vertex AI Model Monitoring

Question 7

A company has a model serving predictions on Vertex AI Endpoints and wants to monitor for prediction drift. They enable Vertex AI Model Monitoring but also need to see a confusion matrix over time. How should they set up the confusion matrix monitoring?

Accepted Answer

Upload ground truth data to BigQuery and use Vertex AI Model Monitoring's model quality monitoring. Vertex AI Model Monitoring can compute confusion matrices when ground truth labels are provided in BigQuery. The service compares predictions against actuals and can output confusion matrices over time windows.

Answer

Use Cloud Monitoring to create a custom dashboard with a confusion matrix chart

Answer

Export predictions to Cloud Storage and run a Dataflow job to compute confusion matrices

Answer

Enable Vertex AI Explainable AI and configure it to output confusion matrices

Question 8

An MLOps engineer is configuring Vertex AI Model Monitoring for a deployed model. They want to monitor for feature skew between training and serving data, but only for a subset of features. The training data has 100 features, and they want to monitor only the top 10 most important features to reduce cost and noise. How can they achieve this?

Accepted Answer

Use the 'feature_names' parameter in the ModelMonitoringObjectConfig to specify which features to monitor. Vertex AI Model Monitoring allows you to specify a list of feature names to monitor via the 'feature_names' attribute in the monitoring configuration. This can be set when creating the monitoring job, targeting only the features of interest.

Answer

Set the 'monitoring_interval' to a low value so that only frequent features are monitored

Answer

Train a new model with only the top 10 features and redeploy it

Answer

Set the 'sampling_rate' to 100% and ignore the rest

Question 9

A team is monitoring a model and observes that the error rate (prediction failures) has increased. They have enabled request/response logging on the Vertex AI Endpoint. How can they set up a metric and alert for prediction error rate?

Accepted Answer

Create a log-based metric in Cloud Logging for error logs and set up an alert in Cloud Monitoring. Vertex AI Endpoint logs contain information about failed predictions. You can create a log-based metric in Cloud Logging that counts error logs, and then create an alert in Cloud Monitoring based on that metric.

Answer

Configure Cloud Monitoring to pull error rate from Cloud Endpoints

Answer

Use Vertex AI Model Monitoring to monitor error rate directly

Answer

Enable Vertex AI Pipelines to track errors

Question 10

A company has deployed a model for image classification and wants to monitor for feature drift using XRAI attributions. However, they notice that the XRAI attribution maps are too large and are causing high latency in the monitoring pipeline. What is the most effective way to reduce the overhead of explainability monitoring for image models?

Accepted Answer

Reduce the sampling rate for the explainability feature. Vertex AI Explainable AI supports XRAI for image models, but generating XRAI attributions can be computationally expensive. Sampling a subset of predictions reduces the number of explanations generated, lowering latency and cost.

Answer

Disable XRAI and use integrated gradients instead

Answer

Use a smaller image size for the model

Answer

Increase the number of replicas on the endpoint