PMLE Monitoring ML Solutions — Timed 20 Questions

Question 1

A team is monitoring a deployed model and notices that the prediction distribution has changed significantly over the last week. They want to detect which features are contributing most to the drift. Which tool should they use?

20 minute time limit — choose an answer to begin.

Accepted Answer

Vertex AI Explainable AI. Vertex AI Explainable AI provides feature attributions (e.g., Sampled Shapley or integrated gradients) that show how much each input feature contributed to a given prediction. When prediction distributions shift, applying Explainable AI to the drifted predictions reveals which features are driving the change, enabling targeted root-cause analysis. This is exactly the capability needed to identify the features contributing most to drift.

Answer

Vertex AI Feature Store

Answer

Vertex AI Model Monitoring

Answer

Vertex AI Pipelines

Question 2

An engineer wants to configure alerting when the data distribution of a serving feature deviates from the training data distribution. The model is deployed on Vertex AI Endpoints. Which divergence metric should they use to compare the training and serving distributions?

Accepted Answer

Jensen-Shannon divergence. Jensen-Shannon divergence is a symmetric and bounded measure of similarity between two probability distributions, making it suitable for comparing training and serving feature distributions. It is based on Kullback-Leibler divergence but is symmetric and always finite, which is ideal for monitoring drift in Vertex AI Endpoints.

Answer

Kullback-Leibler divergence

Answer

Population Stability Index (PSI)

Answer

Chi-squared test

Question 3

A team is monitoring a model on Vertex AI Endpoints and wants to track the p99 latency of online predictions. Which approach should they use to set up latency monitoring and alerting?

Accepted Answer

Use Cloud Logging to create log-based metrics from prediction logs and set up alerts in Cloud Monitoring. Vertex AI online prediction latency metrics are not exposed through Model Monitoring (which focuses on skew/drift) or Explainable AI. Instead, prediction request/response logs contain latency information that can be surfaced via Cloud Logging log-based metrics, which can then be charted and alerted on in Cloud Monitoring. This is the documented approach for p99 latency alerting on Vertex AI Endpoints.

Answer

Enable Vertex AI Model Monitoring and select 'latency' as a metric

Answer

Enable Vertex AI Explainable AI to output latency statistics

Answer

Configure Cloud Monitoring to scrape Prometheus metrics from the endpoint

Question 4

An MLOps engineer is configuring Vertex AI Model Monitoring for a deployed model. They want to monitor for feature skew between training and serving data, but only for a subset of features. The training data has 100 features, and they want to monitor only the top 10 most important features to reduce cost and noise. How can they achieve this?

Accepted Answer

Use the 'feature_names' parameter in the ModelMonitoringObjectConfig to specify which features to monitor. Vertex AI Model Monitoring allows you to scope monitoring to specific features by passing a list of feature names in the ModelMonitoringObjectConfig (specifically via the feature_names field in the monitoring objective configuration). This lets the engineer monitor only the top 10 most important features for skew and drift, reducing cost and alert noise without retraining or redeploying the model. It is the designed mechanism for selective feature monitoring.

Answer

Set the 'monitoring_interval' to a low value so that only frequent features are monitored

Answer

Train a new model with only the top 10 features and redeploy it

Answer

Set the 'sampling_rate' to 100% and ignore the rest

Question 5

An organization is deploying a loan approval model and wants to monitor for fairness across demographic subgroups. They have ground truth labels stored in BigQuery. How can they use Vertex AI to evaluate performance disparities between groups?

Accepted Answer

Upload predictions and labels to Vertex AI Model Evaluation and specify slicing columns. Vertex AI Model Evaluation accepts a predictions file and a ground truth (labels) file, and supports slicing by specified columns so metrics are computed per subgroup. By uploading predictions and BigQuery-stored labels and designating the demographic column as a slice, the engineer can directly compare fairness metrics (e.g., precision, recall, F1) across groups without manual computation. This is the native, supported path for subgroup fairness analysis in Vertex AI.

Answer

Query BigQuery and manually compute metrics per group, then visualize in Looker

Answer

Enable Vertex AI Model Monitoring with fairness detection

Answer

Use Vertex AI Explainable AI to get feature attributions per subgroup

Question 6

A team is using Vertex AI Model Monitoring and wants to set up automated retraining when drift is detected. Which THREE services are needed to implement this pipeline? (Choose three.)

Accepted Answer

Cloud Functions. Cloud Functions (C) is correct because it provides the serverless compute that receives the drift alert and executes the retraining logic, such as calling the Vertex AI training pipeline. Cloud Monitoring (D) is correct because Vertex AI Model Monitoring publishes drift metrics to Cloud Monitoring, which evaluates alerting policies and fires an alert when drift thresholds are exceeded. Cloud Pub/Sub (E) is correct because it acts as the event/messaging backbone that transports the drift notification from the monitoring alert to the Cloud Function trigger, decoupling the alert from the retraining action. Cloud Scheduler (A) is not required because it only schedules time-based jobs and does not react to drift events, and Vertex AI Model Monitoring (B) is the source of the drift signal rather than an additional service needed to build the automated retraining pipeline.

Answer

Cloud Scheduler

Answer

Vertex AI Model Monitoring

Question 7

An ML engineer wants to monitor a deployed model for fairness across different age groups and genders. Which TWO Vertex AI services should they use together to achieve this? (Choose two.)

Accepted Answer

BigQuery. Vertex AI Model Evaluation (E) is correct because it computes fairness and bias metrics for a deployed model, including breakdowns across sensitive slices such as age groups and genders, using metrics like disparate impact and equal opportunity difference. BigQuery (C) is correct because it stores the model's prediction and training data, allowing the engineer to join predictions with sensitive attributes and run slice-based fairness analysis at scale with SQL. Together, Model Evaluation surfaces the fairness metrics while BigQuery provides the data foundation for slicing and deeper analysis. Vertex AI Feature Store (A) is for organizing and serving ML features, not for fairness measurement. Vertex AI Explainable AI (B) provides feature attributions for individual predictions, not group fairness metrics. Cloud Monitoring (D) tracks operational metrics like latency and error rates, not fairness across demographic groups.

Answer

Vertex AI Feature Store

Answer

Vertex AI Explainable AI

Answer

Cloud Monitoring

Question 8

An ML engineer has deployed a model on Vertex AI Endpoints and wants to detect when the serving data distribution differs from the training data distribution. Which monitoring feature should they enable?

Accepted Answer

Feature skew monitoring. Feature skew monitoring in Vertex AI Model Monitoring detects differences between the feature distributions at training time and at serving time — i.e., training-serving skew. Since the engineer wants to compare serving data against the training distribution, this is the correct feature.

Answer

Prediction drift monitoring

Answer

Feature drift monitoring

Answer

Model quality monitoring

Question 9

A data scientist notices that the model's prediction latency has increased over the last week. They need to investigate the root cause by examining request and response logs for the Vertex AI Endpoint. What is the recommended way to capture these logs?

Accepted Answer

Configure request/response logging by specifying a BigQuery destination in the endpoint deployment. Vertex AI Endpoints support request-response logging, which captures the full request and response payloads for online predictions. Configuring this logging with a BigQuery destination during endpoint deployment stores the logs in a queryable table, enabling the data scientist to investigate latency by examining request and response details. This is the recommended way to capture per-request logs for debugging.

Answer

Enable Vertex AI Model Monitoring with sampling rate 100%

Answer

Export logs to Cloud Storage using a sink and use gcloud logging read

Answer

Use Cloud Monitoring custom metrics to capture latency per request

Question 10

A company has deployed a model to Vertex AI Endpoints and wants to monitor for feature drift using Jensen-Shannon divergence. They have set a threshold of 0.1. After one week, the monitoring job reports a divergence of 0.15 for a feature. What should the engineer do next to diagnose which features are contributing to the drift?

Accepted Answer

Use Vertex AI Explainability to compute feature attributions and identify drifted features. When feature drift is detected, the next step is to diagnose which features are contributing to the drift. Vertex AI Explainability provides feature attributions that show how much each feature influenced the model's predictions. By comparing attributions over time or examining the drifted feature's impact, engineers can identify the root cause. Deploying a new model or increasing sampling does not directly diagnose the drifted features.

Answer

Deploy a new model version immediately

Answer

Check the model's confusion matrix in BigQuery

Answer

Increase the sampling rate to capture more data

Question 11

A machine learning engineer wants to monitor the fairness of a credit approval model across demographic subgroups. They have ground truth labels in BigQuery. Which approach should they use to evaluate performance disparities?

Accepted Answer

Use Vertex AI Model Evaluation with sliced evaluation in BigQuery. Vertex AI Model Evaluation supports sliced evaluation, allowing comparison of metrics (like accuracy, precision) across subgroups defined by features like age, gender, etc.

Answer

Use Vertex AI Explainability to compute feature attributions per subgroup

Answer

Use Cloud Monitoring custom metrics to track predictions per subgroup

Answer

Use Vertex AI Model Monitoring to detect skew in demographic features

Question 12

A company wants to be alerted when the prediction error rate on their Vertex AI Endpoint exceeds 5% in any 5-minute window. What is the best way to set up this alert?

Accepted Answer

Create a Cloud Monitoring alert on the metric 'prediction/error_count' with a threshold of 5% of total predictions. Cloud Monitoring can alert directly on the Vertex AI Endpoint metric 'prediction/error_count' with a threshold condition comparing it to total predictions over a 5-minute window. This is the native, lowest-latency approach because the metric is already emitted by the endpoint and Cloud Monitoring supports ratio-based alerting policies. No custom code or log pipeline is needed.

Answer

Set up a scheduled Cloud Function to query logs and check error rate

Answer

Enable BigQuery log sink and create a scheduled query alert

Answer

Use Vertex AI Model Monitoring to detect prediction drift

Question 13

An engineer is configuring Vertex AI Model Monitoring for a model deployed on an endpoint. They want to monitor feature skew using the training dataset as a baseline. The training dataset is large (10 TB). What is the most efficient way to provide the baseline distribution?

Accepted Answer

Use the BigQuery table containing the training data as the baseline source in the monitoring configuration. Using the BigQuery table containing the training data as the baseline source in the monitoring configuration is the most efficient way because Vertex AI Model Monitoring can directly query BigQuery to compute the baseline distribution without exporting or duplicating the 10 TB dataset. This avoids data movement and leverages BigQuery's scalability. It is the native, recommended approach for large datasets.

Answer

Compute the baseline distribution offline and upload a JSON file with feature statistics

Answer

Upload the entire training dataset as a CSV file to Cloud Storage and reference it in the monitoring config

Answer

Sample 1% of the training data and use that as baseline

Question 14

An ML engineer needs to set up automated retraining triggered by data drift. They have decided to use Cloud Monitoring alerts to detect drift. Which TWO additional services are required to complete the retraining pipeline? (Choose 2)

Accepted Answer

Vertex AI Pipeline. Option D (Vertex AI Pipeline) is correct because it is the managed service that orchestrates and executes the actual retraining workflow — defining the training steps, model evaluation, and deployment as a reproducible pipeline that can be triggered programmatically. Option E (Cloud Functions) is correct because a Cloud Monitoring alert must publish to a notification channel, and a Cloud Function (typically invoked via Pub/Sub) acts as the glue that receives the alert and calls the Vertex AI Pipeline API to start the retraining run. Together they complete the loop: Monitoring detects drift, the alert fires, Cloud Functions receives it, and Vertex AI Pipeline performs the retraining. Option A (Cloud Dataflow) is not required since batch data processing is not the missing piece here, and drift detection is already handled by Cloud Monitoring. Option B (Cloud Build) is for CI/CD build and container image creation, not for orchestrating ML retraining. Option C (Cloud Scheduler) is unnecessary because the trigger is event-driven from a Monitoring alert, not a time-based cron schedule.

Answer

Cloud Dataflow

Answer

Cloud Build

Answer

Cloud Scheduler

Question 15

A data science team wants to monitor model quality by comparing predictions against ground truth labels. They have deployed a model on Vertex AI Endpoints and enable request/response logging to BigQuery. Which THREE actions should they take to set up model quality monitoring? (Choose 3)

Accepted Answer

Create a scheduled query to compute metrics like accuracy and confusion matrix over time. Option A is correct because model quality monitoring requires periodically computing quality metrics such as accuracy, precision, recall, and confusion matrices from the joined prediction and label data, and a scheduled query in BigQuery is the standard way to compute these metrics over time. Option B is correct because predictions logged to BigQuery must be joined with ground truth labels using a common key (such as the request ID) so that each prediction can be compared against its actual label to derive quality metrics. Option E is correct because ground truth labels must be available in BigQuery (typically in a separate table) before they can be joined with prediction logs and used to compute model quality metrics. Option C is not correct because Vertex AI Model Monitoring's prediction drift detection compares incoming prediction feature distributions against a training baseline, not predictions against ground truth labels, so it does not measure model quality. Option D is not correct because Vertex AI Explainability produces feature attributions for individual predictions and does not compare predictions with ground truth labels to assess model quality.

Answer

Configure Vertex AI Model Monitoring to detect prediction drift

Answer

Use Vertex AI Explainability to compute feature attributions

Question 16

A financial services company deploys a model on Vertex AI Endpoints with GPU acceleration. They notice that the p99 latency for predictions has increased from 200ms to 1.2s over the past week. CPU utilisation is low, but GPU utilisation is high. Which action should they take to reduce latency?

Accepted Answer

Increase the number of replicas in the endpoint deployment.. High GPU utilisation with low CPU utilisation indicates the GPU is the bottleneck, and adding more replicas distributes the inference load across additional GPU-backed instances, directly reducing per-request latency. Increasing replicas scales horizontally, which is the correct action when a single instance's GPU is saturated. This addresses the p99 latency spike without changing the model or hardware type.

Answer

Increase the CPU machine type for the endpoint.

Answer

Switch to a more powerful GPU type (e.g., from T4 to A100).

Answer

Reduce the sampling rate for monitoring to free up resources.

Question 17

A company wants to implement a retraining trigger for their ML model. They have set up Cloud Monitoring alerts that fire when drift exceeds a threshold. What should be the target of the alert to automatically start a Vertex AI Pipeline for retraining?

Accepted Answer

Pub/Sub topic. Cloud Monitoring alerting policies can route notifications to a Pub/Sub topic, which then acts as the event source that triggers a Vertex AI Pipeline run. Pub/Sub provides the decoupled, asynchronous messaging backbone that lets the alert fire a pipeline execution without manual intervention, typically via an Eventarc or Cloud Function subscriber that calls the Vertex AI Pipelines API.

Answer

Vertex AI Model Registry

Answer

Cloud Storage bucket

Answer

Cloud Functions HTTP trigger

Question 18

A team uses Vertex AI Explainable AI with integrated gradients for a deep learning model. They want to reduce the computational cost of explanations without significantly reducing explanation quality. Which configuration change should they make?

Accepted Answer

Reduce the number of integral approximation steps.. Integrated gradients approximates Shapley values by integrating gradients along a path. Reducing the number of steps (integral approximation steps) reduces computation, but may reduce quality. A moderate reduction balances cost and quality.

Answer

Switch from integrated gradients to XRAI.

Answer

Apply feature attribution to only a random subset of predictions.

Answer

Use sampled Shapley instead, as it is always cheaper.

Question 19

An ML engineer is monitoring a model on Vertex AI Endpoint and sees that feature 'age' has a training distribution of (mean=45, std=10) but the serving distribution over the last hour shows (mean=30, std=15). JS divergence is 0.12, but the alert threshold is 0.1. The engineer suspects this is due to a temporary campaign targeting younger users. What should they do first?

Accepted Answer

Use Vertex AI Explainable AI to understand the importance of the 'age' feature for the model.. Before taking action like retraining, the engineer should investigate the root cause. Using Vertex AI Explainable AI to check feature importance can confirm whether 'age' is a critical feature. If it is not important, the drift may be harmless.

Answer

Immediately trigger retraining to avoid model degradation.

Answer

Increase the sampling rate to get more data before making a decision.

Answer

Adjust the alert threshold to 0.2 to avoid false positives.

Question 20

A team wants to monitor prediction drift on a Vertex AI Endpoint for a classification model. They have configured Vertex AI Model Monitoring with default settings. Which metric will be used to detect prediction drift?

Accepted Answer

Jensen-Shannon divergence. Vertex AI Model Monitoring uses Jensen-Shannon divergence as the default metric for detecting prediction drift on classification models. It measures the similarity between the distribution of live prediction outputs and the training baseline, with a default threshold. KL divergence and PSI are not the default for prediction drift in Vertex AI.

Answer

Kullback-Leibler divergence

Answer

L-infinity distance

Answer

Population Stability Index (PSI)