PMLE Monitoring ML Solutions Practice Test 2 — 15 Questions

Question 1

An ML engineer has deployed a model on Vertex AI Endpoints and wants to detect when the serving data distribution differs from the training data distribution. Which monitoring feature should they enable?

Accepted Answer

Feature skew monitoring. Feature skew monitoring compares the training data distribution (stored in a baseline) with the serving data distribution to detect skew. Feature drift tracks changes over time in serving data only.

Answer

Prediction drift monitoring

Answer

Feature drift monitoring

Answer

Model quality monitoring

Question 2

A data scientist notices that the model's prediction latency has increased over the last week. They need to investigate the root cause by examining request and response logs for the Vertex AI Endpoint. What is the recommended way to capture these logs?

Accepted Answer

Configure request/response logging by specifying a BigQuery destination in the endpoint deployment. Vertex AI Endpoint can be configured to log request/response data to BigQuery via a log sink. This data can then be analyzed to understand latency issues.

Answer

Enable Vertex AI Model Monitoring with sampling rate 100%

Answer

Export logs to Cloud Storage using a sink and use gcloud logging read

Answer

Use Cloud Monitoring custom metrics to capture latency per request

Question 3

A company has deployed a model to Vertex AI Endpoints and wants to monitor for feature drift using Jensen-Shannon divergence. They have set a threshold of 0.1. After one week, the monitoring job reports a divergence of 0.15 for a feature. What should the engineer do next to diagnose which features are contributing to the drift?

Accepted Answer

Use Vertex AI Explainability to compute feature attributions and identify drifted features. To identify which features are drifting, the engineer can use Vertex AI Explainability to compute feature attributions (e.g., SHAP values) and correlate them with drift metrics.

Answer

Deploy a new model version immediately

Answer

Check the model's confusion matrix in BigQuery

Answer

Increase the sampling rate to capture more data

Question 4

An ML team has set up automated retraining triggered by Cloud Monitoring alerts. When a feature drift alert fires, a Cloud Function publishes to Pub/Sub, which triggers a Vertex AI Pipeline. However, the retraining pipeline is failing because the training data is not updated. What is the most likely cause?

Accepted Answer

The training data in the pipeline input is stale or not refreshed. Cloud Monitoring alerts can trigger retraining, but the pipeline typically expects the most recent data. If the training data is not refreshed, the pipeline may fail or produce a stale model.

Answer

The Cloud Function does not have permission to start the pipeline

Answer

The Pub/Sub topic is incorrectly configured

Answer

The model endpoint is overloaded

Question 5

A machine learning engineer wants to monitor the fairness of a credit approval model across demographic subgroups. They have ground truth labels in BigQuery. Which approach should they use to evaluate performance disparities?

Accepted Answer

Use Vertex AI Model Evaluation with sliced evaluation in BigQuery. Vertex AI Model Evaluation supports sliced evaluation, allowing comparison of metrics (like accuracy, precision) across subgroups defined by features like age, gender, etc.

Answer

Use Vertex AI Explainability to compute feature attributions per subgroup

Answer

Use Cloud Monitoring custom metrics to track predictions per subgroup

Answer

Use Vertex AI Model Monitoring to detect skew in demographic features

Question 6

An ML engineer needs to monitor the online prediction latency of a Vertex AI Endpoint. Which metrics should they look at in Cloud Monitoring?

Accepted Answer

p50, p95, p99 latency. Cloud Monitoring provides latency metrics for Vertex AI Endpoints, including p50, p95, and p99 latency, which are key for understanding performance.

Answer

Request count and error rate

Answer

Skew and drift scores

Answer

CPU/GPU utilization

Question 7

A company wants to be alerted when the prediction error rate on their Vertex AI Endpoint exceeds 5% in any 5-minute window. What is the best way to set up this alert?

Accepted Answer

Create a Cloud Monitoring alert on the metric 'prediction/error_count' with a threshold of 5% of total predictions. Cloud Monitoring can create an alerting policy based on the 'Prediction error count' metric with a rolling 5-minute window and a threshold of 5% (or absolute count).

Answer

Set up a scheduled Cloud Function to query logs and check error rate

Answer

Enable BigQuery log sink and create a scheduled query alert

Answer

Use Vertex AI Model Monitoring to detect prediction drift

Question 8

A model deployed on a Vertex AI Endpoint uses an image model with XRAI explainability. The team notices that the prediction distributions are shifting over time. They want to monitor prediction drift. However, the explainability feature is not enabled. What must the engineer do to enable monitoring prediction drift?

Accepted Answer

Configure Vertex AI Model Monitoring to monitor prediction drift. Prediction drift monitoring is part of Vertex AI Model Monitoring and does not require explainability to be enabled. It can be configured independently.

Answer

Re-deploy the model with a sampling rate of 100%

Answer

Enable Vertex AI Explainability with XRAI on the endpoint deployment

Answer

Enable request/response logging to BigQuery and build custom drift detection

Question 9

An ML engineer needs to track the costs incurred by Vertex AI prediction endpoints. Which tool should they use to set budget alerts and monitor spending?

Accepted Answer

Google Cloud Budgets & Alerts. Google Cloud Budgets & Alerts allow setting budget thresholds and sending notifications. Billing reports provide cost breakdowns by service.

Answer

Vertex AI Model Monitoring

Answer

Cloud Logging

Answer

Cloud Monitoring with custom metrics

Question 10

A data scientist has deployed a model with Vertex AI Endpoints and enabled request/response logging to BigQuery. They want to compute a confusion matrix over time to monitor model quality. What should they do?

Accepted Answer

Upload ground truth labels to BigQuery and join with prediction logs, then compute confusion matrix in a scheduled query. To compute a confusion matrix, ground truth labels are needed. The team can upload ground truth labels to BigQuery and join with prediction logs to compare predictions vs actuals.

Answer

Use Vertex AI Model Monitoring to automatically generate confusion matrices

Answer

Use Cloud Monitoring to create a confusion matrix dashboard

Answer

Enable Vertex AI Explainability to get confusion matrix

Question 11

An ML team is using Population Stability Index (PSI) to monitor feature drift on a Vertex AI Endpoint. The PSI value for a feature is 0.25, which exceeds the alert threshold of 0.2. The feature has high SHAP importance. The team wants to automatically retrain the model. What is the correct end-to-end setup?

Accepted Answer

Cloud Monitoring alert → Pub/Sub → Cloud Functions → Vertex AI Pipeline (with training and deployment steps). The correct setup involves Cloud Monitoring alert (based on PSI metric) → Pub/Sub → Cloud Function → Vertex AI Pipeline to trigger retraining with updated data.

Answer

Vertex AI Model Monitoring alert → Cloud Scheduler → Vertex AI Training

Answer

Cloud Functions → Pub/Sub → Cloud Monitoring → Vertex AI Pipeline

Answer

Cloud Monitoring alert → Cloud Logging → Cloud Functions → Vertex AI Training

Question 12

An engineer is configuring Vertex AI Model Monitoring for a model deployed on an endpoint. They want to monitor feature skew using the training dataset as a baseline. The training dataset is large (10 TB). What is the most efficient way to provide the baseline distribution?

Accepted Answer

Use the BigQuery table containing the training data as the baseline source in the monitoring configuration. Vertex AI Model Monitoring can automatically compute the baseline distribution from the training dataset stored in BigQuery. This avoids manual computation or exporting large datasets.

Answer

Compute the baseline distribution offline and upload a JSON file with feature statistics

Answer

Upload the entire training dataset as a CSV file to Cloud Storage and reference it in the monitoring config

Answer

Sample 1% of the training data and use that as baseline

Question 13

An ML engineer needs to set up automated retraining triggered by data drift. They have decided to use Cloud Monitoring alerts to detect drift. Which TWO additional services are required to complete the retraining pipeline? (Choose 2)

Accepted Answer

Vertex AI Pipeline. The typical architecture uses Cloud Monitoring alert -> Pub/Sub -> Cloud Functions -> Vertex AI Pipeline. Cloud Functions processes the alert and triggers the pipeline. Vertex AI Pipeline orchestrates training and deployment.

Answer

Cloud Dataflow

Answer

Cloud Build

Answer

Cloud Scheduler

Question 14

A data science team wants to monitor model quality by comparing predictions against ground truth labels. They have deployed a model on Vertex AI Endpoints and enable request/response logging to BigQuery. Which THREE actions should they take to set up model quality monitoring? (Choose 3)

Accepted Answer

Create a scheduled query to compute metrics like accuracy and confusion matrix over time. To monitor model quality, the team needs to upload ground truth labels to BigQuery, join with prediction logs, compute metrics (e.g., accuracy, confusion matrix) over time, and optionally create dashboards.

Answer

Configure Vertex AI Model Monitoring to detect prediction drift

Answer

Use Vertex AI Explainability to compute feature attributions

Question 15

An ML engineer wants to monitor the performance of a Vertex AI Endpoint. Which TWO metrics are available in Cloud Monitoring for Vertex AI Endpoints? (Choose 2)

Accepted Answer

Error count. Cloud Monitoring for Vertex AI Endpoints includes metrics like prediction latency (p50, p95, p99) and error count/rate. CPU/GPU utilization is also available for endpoint machines.

Answer

Model accuracy

Answer

Feature skew score

Answer

SHAP values