PMLE Monitoring ML Solutions — 15 Questions

Question 1

A data scientist has deployed a model on Vertex AI Endpoints and wants to monitor the model's predictions for any drift over time. Which Vertex AI service should they use?

Accepted Answer

Vertex AI Model Monitoring. Vertex AI Model Monitoring is specifically designed to monitor deployed models for feature drift, feature skew, and prediction drift. It uses statistical methods to compare serving distributions over time or against training data.

Answer

Vertex AI Feature Store

Answer

Vertex AI Predictions

Answer

Vertex AI Explainable AI

Question 2

An MLOps engineer needs to collect ground truth labels for a deployed classification model to compare predictions against actuals. Where should the engineer store the ground truth data to enable Vertex AI model quality monitoring?

Accepted Answer

BigQuery. Vertex AI Model Monitoring expects ground truth data to be uploaded to BigQuery tables, which can then be used to compute confusion matrices and other quality metrics over time.

Answer

Firestore

Answer

Cloud Spanner

Answer

Cloud Storage

Question 3

A team is monitoring a deployed model and notices that the prediction distribution has changed significantly over the last week. They want to detect which features are contributing most to the drift. Which tool should they use?

Accepted Answer

Vertex AI Explainable AI. Vertex AI Explainable AI provides feature attributions (e.g., SHAP values) that can be used to identify which features are most important for predictions. By comparing feature importance over time, they can pinpoint which features are drifting.

Answer

Vertex AI Feature Store

Answer

Vertex AI Model Monitoring

Answer

Vertex AI Pipelines

Question 4

An engineer wants to configure alerting when the data distribution of a serving feature deviates from the training data distribution. The model is deployed on Vertex AI Endpoints. Which divergence metric should they use to compare the training and serving distributions?

Accepted Answer

Jensen-Shannon divergence. Vertex AI Model Monitoring supports Jensen-Shannon divergence for comparing distributions. It is a symmetric and bounded metric suitable for detecting feature skew between training and serving data.

Answer

Kullback-Leibler divergence

Answer

Population Stability Index (PSI)

Answer

Chi-squared test

Question 5

A team is monitoring a model on Vertex AI Endpoints and wants to track the p99 latency of online predictions. Which approach should they use to set up latency monitoring and alerting?

Accepted Answer

Use Cloud Logging to create log-based metrics from prediction logs and set up alerts in Cloud Monitoring. Vertex AI Endpoints automatically export request/response logs to Cloud Logging, which can be used to create log-based metrics for latency percentiles. These metrics can then be visualized in Cloud Monitoring dashboards and used for alerting.

Answer

Enable Vertex AI Model Monitoring and select 'latency' as a metric

Answer

Enable Vertex AI Explainable AI to output latency statistics

Answer

Configure Cloud Monitoring to scrape Prometheus metrics from the endpoint

Question 6

An ML team wants to automatically retrain a model when data drift is detected. They have set up a Cloud Monitoring alert on drift. What service should they use to trigger a retraining pipeline in response to the alert?

Accepted Answer

Cloud Functions. Cloud Monitoring alerts can send notifications to Pub/Sub topics. A Pub/Sub message can then trigger a Cloud Function or Cloud Run service that starts a Vertex AI Pipeline for retraining.

Answer

Cloud Scheduler

Answer

Vertex AI Feature Store

Answer

Vertex AI Model Monitoring

Question 7

A company has a model serving predictions on Vertex AI Endpoints and wants to monitor for prediction drift. They enable Vertex AI Model Monitoring but also need to see a confusion matrix over time. How should they set up the confusion matrix monitoring?

Accepted Answer

Upload ground truth data to BigQuery and use Vertex AI Model Monitoring's model quality monitoring. Vertex AI Model Monitoring can compute confusion matrices when ground truth labels are provided in BigQuery. The service compares predictions against actuals and can output confusion matrices over time windows.

Answer

Use Cloud Monitoring to create a custom dashboard with a confusion matrix chart

Answer

Export predictions to Cloud Storage and run a Dataflow job to compute confusion matrices

Answer

Enable Vertex AI Explainable AI and configure it to output confusion matrices

Question 8

An MLOps engineer is configuring Vertex AI Model Monitoring for a deployed model. They want to monitor for feature skew between training and serving data, but only for a subset of features. The training data has 100 features, and they want to monitor only the top 10 most important features to reduce cost and noise. How can they achieve this?

Accepted Answer

Use the 'feature_names' parameter in the ModelMonitoringObjectConfig to specify which features to monitor. Vertex AI Model Monitoring allows you to specify a list of feature names to monitor via the 'feature_names' attribute in the monitoring configuration. This can be set when creating the monitoring job, targeting only the features of interest.

Answer

Set the 'monitoring_interval' to a low value so that only frequent features are monitored

Answer

Train a new model with only the top 10 features and redeploy it

Answer

Set the 'sampling_rate' to 100% and ignore the rest

Question 9

A team is monitoring a model and observes that the error rate (prediction failures) has increased. They have enabled request/response logging on the Vertex AI Endpoint. How can they set up a metric and alert for prediction error rate?

Accepted Answer

Create a log-based metric in Cloud Logging for error logs and set up an alert in Cloud Monitoring. Vertex AI Endpoint logs contain information about failed predictions. You can create a log-based metric in Cloud Logging that counts error logs, and then create an alert in Cloud Monitoring based on that metric.

Answer

Configure Cloud Monitoring to pull error rate from Cloud Endpoints

Answer

Use Vertex AI Model Monitoring to monitor error rate directly

Answer

Enable Vertex AI Pipelines to track errors

Question 10

A company has deployed a model for image classification and wants to monitor for feature drift using XRAI attributions. However, they notice that the XRAI attribution maps are too large and are causing high latency in the monitoring pipeline. What is the most effective way to reduce the overhead of explainability monitoring for image models?

Accepted Answer

Reduce the sampling rate for the explainability feature. Vertex AI Explainable AI supports XRAI for image models, but generating XRAI attributions can be computationally expensive. Sampling a subset of predictions reduces the number of explanations generated, lowering latency and cost.

Answer

Disable XRAI and use integrated gradients instead

Answer

Use a smaller image size for the model

Answer

Increase the number of replicas on the endpoint

Question 11

An organization is deploying a loan approval model and wants to monitor for fairness across demographic subgroups. They have ground truth labels stored in BigQuery. How can they use Vertex AI to evaluate performance disparities between groups?

Accepted Answer

Upload predictions and labels to Vertex AI Model Evaluation and specify slicing columns. Vertex AI Model Evaluation provides sliced evaluation, which computes metrics (e.g., accuracy, precision) per subgroup when slicing columns are specified. This enables detection of performance disparities.

Answer

Query BigQuery and manually compute metrics per group, then visualize in Looker

Answer

Enable Vertex AI Model Monitoring with fairness detection

Answer

Use Vertex AI Explainable AI to get feature attributions per subgroup

Question 12

A company wants to monitor the cost of their Vertex AI prediction endpoint. They are charged per hour per replica and per request for GPU instances. Which approach should they use to track these costs?

Accepted Answer

Set up Cloud Billing budget alerts and export billing data to BigQuery for analysis. Vertex AI prediction costs are tracked via Cloud Billing. Budget alerts can be set up to notify when spending exceeds a threshold. Cost breakdown can be viewed in the Billing reports.

Answer

Use Vertex AI Model Monitoring to track cost metrics

Answer

Enable Cloud Monitoring dashboards for cost metrics

Answer

Use Vertex AI Pipelines to track cost per job

Question 13

An MLOps engineer is setting up monitoring for a deployed model on Vertex AI Endpoints. Which TWO actions are required to enable Vertex AI Model Monitoring for feature skew and drift? (Choose two.)

Accepted Answer

Enable request/response logging on the Vertex AI Endpoint. To enable model monitoring, you must enable request/response logging on the endpoint (to capture serving data) and create a monitoring job with the desired configuration.

Answer

Export ground truth labels to Cloud Storage

Answer

Enable Vertex AI Pipelines to run scheduled monitoring

Answer

Deploy the model with an explanation spec

Question 14

A team is using Vertex AI Model Monitoring and wants to set up automated retraining when drift is detected. Which THREE services are needed to implement this pipeline? (Choose three.)

Accepted Answer

Cloud Functions. The typical pipeline: Cloud Monitoring alert on drift → Pub/Sub message → Cloud Function or Cloud Run → triggers Vertex AI Pipeline for retraining. Optionally, Cloud Scheduler is not needed as it's event-driven.

Answer

Cloud Scheduler

Answer

Vertex AI Model Monitoring

Question 15

An ML engineer wants to monitor a deployed model for fairness across different age groups and genders. Which TWO Vertex AI services should they use together to achieve this? (Choose two.)

Accepted Answer

BigQuery. Vertex AI Model Evaluation provides sliced evaluation when ground truth is available in BigQuery. Vertex AI Explainable AI can help understand feature importance but is not required for fairness monitoring.

Answer

Vertex AI Feature Store

Answer

Vertex AI Explainable AI

Answer

Cloud Monitoring