PMLE Monitoring ML Solutions — Timed 10 Questions

Question 1

A team is monitoring a deployed model and notices that the prediction distribution has changed significantly over the last week. They want to detect which features are contributing most to the drift. Which tool should they use?

10 minute time limit — choose an answer to begin.

Accepted Answer

Vertex AI Explainable AI. Vertex AI Explainable AI provides feature attributions (e.g., Sampled Shapley or integrated gradients) that show how much each input feature contributed to a given prediction. When prediction distributions shift, applying Explainable AI to the drifted predictions reveals which features are driving the change, enabling targeted root-cause analysis. This is exactly the capability needed to identify the features contributing most to drift.

Answer

Vertex AI Feature Store

Answer

Vertex AI Model Monitoring

Answer

Vertex AI Pipelines

Question 2

An engineer wants to configure alerting when the data distribution of a serving feature deviates from the training data distribution. The model is deployed on Vertex AI Endpoints. Which divergence metric should they use to compare the training and serving distributions?

Accepted Answer

Jensen-Shannon divergence. Jensen-Shannon divergence is a symmetric and bounded measure of similarity between two probability distributions, making it suitable for comparing training and serving feature distributions. It is based on Kullback-Leibler divergence but is symmetric and always finite, which is ideal for monitoring drift in Vertex AI Endpoints.

Answer

Kullback-Leibler divergence

Answer

Population Stability Index (PSI)

Answer

Chi-squared test

Question 3

A team is monitoring a model on Vertex AI Endpoints and wants to track the p99 latency of online predictions. Which approach should they use to set up latency monitoring and alerting?

Accepted Answer

Use Cloud Logging to create log-based metrics from prediction logs and set up alerts in Cloud Monitoring. Vertex AI online prediction latency metrics are not exposed through Model Monitoring (which focuses on skew/drift) or Explainable AI. Instead, prediction request/response logs contain latency information that can be surfaced via Cloud Logging log-based metrics, which can then be charted and alerted on in Cloud Monitoring. This is the documented approach for p99 latency alerting on Vertex AI Endpoints.

Answer

Enable Vertex AI Model Monitoring and select 'latency' as a metric

Answer

Enable Vertex AI Explainable AI to output latency statistics

Answer

Configure Cloud Monitoring to scrape Prometheus metrics from the endpoint

Question 4

An MLOps engineer is configuring Vertex AI Model Monitoring for a deployed model. They want to monitor for feature skew between training and serving data, but only for a subset of features. The training data has 100 features, and they want to monitor only the top 10 most important features to reduce cost and noise. How can they achieve this?

Accepted Answer

Use the 'feature_names' parameter in the ModelMonitoringObjectConfig to specify which features to monitor. Vertex AI Model Monitoring allows you to scope monitoring to specific features by passing a list of feature names in the ModelMonitoringObjectConfig (specifically via the feature_names field in the monitoring objective configuration). This lets the engineer monitor only the top 10 most important features for skew and drift, reducing cost and alert noise without retraining or redeploying the model. It is the designed mechanism for selective feature monitoring.

Answer

Set the 'monitoring_interval' to a low value so that only frequent features are monitored

Answer

Train a new model with only the top 10 features and redeploy it

Answer

Set the 'sampling_rate' to 100% and ignore the rest

Question 5

An organization is deploying a loan approval model and wants to monitor for fairness across demographic subgroups. They have ground truth labels stored in BigQuery. How can they use Vertex AI to evaluate performance disparities between groups?

Accepted Answer

Upload predictions and labels to Vertex AI Model Evaluation and specify slicing columns. Vertex AI Model Evaluation accepts a predictions file and a ground truth (labels) file, and supports slicing by specified columns so metrics are computed per subgroup. By uploading predictions and BigQuery-stored labels and designating the demographic column as a slice, the engineer can directly compare fairness metrics (e.g., precision, recall, F1) across groups without manual computation. This is the native, supported path for subgroup fairness analysis in Vertex AI.

Answer

Query BigQuery and manually compute metrics per group, then visualize in Looker

Answer

Enable Vertex AI Model Monitoring with fairness detection

Answer

Use Vertex AI Explainable AI to get feature attributions per subgroup

Question 6

A team is using Vertex AI Model Monitoring and wants to set up automated retraining when drift is detected. Which THREE services are needed to implement this pipeline? (Choose three.)

Accepted Answer

Cloud Functions. Cloud Functions (C) is correct because it provides the serverless compute that receives the drift alert and executes the retraining logic, such as calling the Vertex AI training pipeline. Cloud Monitoring (D) is correct because Vertex AI Model Monitoring publishes drift metrics to Cloud Monitoring, which evaluates alerting policies and fires an alert when drift thresholds are exceeded. Cloud Pub/Sub (E) is correct because it acts as the event/messaging backbone that transports the drift notification from the monitoring alert to the Cloud Function trigger, decoupling the alert from the retraining action. Cloud Scheduler (A) is not required because it only schedules time-based jobs and does not react to drift events, and Vertex AI Model Monitoring (B) is the source of the drift signal rather than an additional service needed to build the automated retraining pipeline.

Answer

Cloud Scheduler

Answer

Vertex AI Model Monitoring

Question 7

An ML engineer wants to monitor a deployed model for fairness across different age groups and genders. Which TWO Vertex AI services should they use together to achieve this? (Choose two.)

Accepted Answer

BigQuery. Vertex AI Model Evaluation (E) is correct because it computes fairness and bias metrics for a deployed model, including breakdowns across sensitive slices such as age groups and genders, using metrics like disparate impact and equal opportunity difference. BigQuery (C) is correct because it stores the model's prediction and training data, allowing the engineer to join predictions with sensitive attributes and run slice-based fairness analysis at scale with SQL. Together, Model Evaluation surfaces the fairness metrics while BigQuery provides the data foundation for slicing and deeper analysis. Vertex AI Feature Store (A) is for organizing and serving ML features, not for fairness measurement. Vertex AI Explainable AI (B) provides feature attributions for individual predictions, not group fairness metrics. Cloud Monitoring (D) tracks operational metrics like latency and error rates, not fairness across demographic groups.

Answer

Vertex AI Feature Store

Answer

Vertex AI Explainable AI

Answer

Cloud Monitoring

Question 8

An ML engineer has deployed a model on Vertex AI Endpoints and wants to detect when the serving data distribution differs from the training data distribution. Which monitoring feature should they enable?

Accepted Answer

Feature skew monitoring. Feature skew monitoring in Vertex AI Model Monitoring detects differences between the feature distributions at training time and at serving time — i.e., training-serving skew. Since the engineer wants to compare serving data against the training distribution, this is the correct feature.

Answer

Prediction drift monitoring

Answer

Feature drift monitoring

Answer

Model quality monitoring

Question 9

A data scientist notices that the model's prediction latency has increased over the last week. They need to investigate the root cause by examining request and response logs for the Vertex AI Endpoint. What is the recommended way to capture these logs?

Accepted Answer

Configure request/response logging by specifying a BigQuery destination in the endpoint deployment. Vertex AI Endpoints support request-response logging, which captures the full request and response payloads for online predictions. Configuring this logging with a BigQuery destination during endpoint deployment stores the logs in a queryable table, enabling the data scientist to investigate latency by examining request and response details. This is the recommended way to capture per-request logs for debugging.

Answer

Enable Vertex AI Model Monitoring with sampling rate 100%

Answer

Export logs to Cloud Storage using a sink and use gcloud logging read

Answer

Use Cloud Monitoring custom metrics to capture latency per request

Question 10

A company has deployed a model to Vertex AI Endpoints and wants to monitor for feature drift using Jensen-Shannon divergence. They have set a threshold of 0.1. After one week, the monitoring job reports a divergence of 0.15 for a feature. What should the engineer do next to diagnose which features are contributing to the drift?

Accepted Answer

Use Vertex AI Explainability to compute feature attributions and identify drifted features. When feature drift is detected, the next step is to diagnose which features are contributing to the drift. Vertex AI Explainability provides feature attributions that show how much each feature influenced the model's predictions. By comparing attributions over time or examining the drifted feature's impact, engineers can identify the root cause. Deploying a new model or increasing sampling does not directly diagnose the drifted features.

Answer

Deploy a new model version immediately

Answer

Check the model's confusion matrix in BigQuery

Answer

Increase the sampling rate to capture more data