PMLE Monitoring ML Solutions Practice Test 3 — 15 Questions

Question 1

A data scientist has deployed a classification model on a Vertex AI Endpoint and wants to monitor for feature drift in the serving data compared to the training data. Which Vertex AI service should be used?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Vertex AI Model Monitoring. Vertex AI Model Monitoring is specifically designed to monitor deployed models for feature skew, feature drift, and prediction drift. It supports algorithms like Jensen-Shannon divergence and Population Stability Index.

Answer

Vertex AI Explainable AI

Answer

Vertex AI Model Evaluation

Answer

Vertex AI Continuous Training

Question 2

An ML engineer has set up Vertex AI Model Monitoring on an endpoint with a sampling rate of 0.1 (10%). They notice that the monitoring job runs hourly but the reported drift metrics seem inconsistent. What is the most likely cause?

Accepted Answer

The sampling rate is too low, leading to insufficient data for reliable drift statistics.. A low sampling rate means only a small fraction of requests are logged for monitoring, which can lead to statistical noise and inconsistent drift metrics. Increasing the sampling rate would improve accuracy.

Answer

Prediction drift monitoring is not enabled; only feature drift is configured.

Answer

The drift detection algorithm is not suited for this model; try changing from JS divergence to L-infinity distance.

Answer

The monitoring frequency is too low; it should be set to every 5 minutes.

Question 3

A team wants to collect ground truth labels for their model deployed on Vertex AI Endpoint to perform model quality monitoring. They have a process that generates actual outcomes within 24 hours of prediction. What is the recommended approach for storing these labels?

Accepted Answer

Upload the ground truth labels to a BigQuery table with a schema that includes prediction timestamp and model version.. Vertex AI Model Monitoring for model quality requires ground truth data to be uploaded to BigQuery. Labels are stored in a BigQuery table with the prediction timestamp and model ID for comparison.

Answer

Use Vertex AI Experiments to log ground truth alongside training runs.

Answer

Store the ground truth labels in Cloud Storage as CSV files and reference them in the monitoring config.

Answer

Insert ground truth labels directly into the Vertex AI Endpoint's log sink.

Question 4

A financial services company deploys a model on Vertex AI Endpoints with GPU acceleration. They notice that the p99 latency for predictions has increased from 200ms to 1.2s over the past week. CPU utilisation is low, but GPU utilisation is high. Which action should they take to reduce latency?

Accepted Answer

Increase the number of replicas in the endpoint deployment.. High GPU utilisation with low CPU utilisation suggests the model is compute-bound on GPU. Scaling out by adding more replicas distributes the load, reducing queuing and latency.

Answer

Increase the CPU machine type for the endpoint.

Answer

Switch to a more powerful GPU type (e.g., from T4 to A100).

Answer

Reduce the sampling rate for monitoring to free up resources.

Question 5

An ML engineer needs to monitor the error rate of prediction jobs on a Vertex AI Endpoint. Where can they view the number of failed prediction requests over time?

Accepted Answer

Cloud Monitoring. Vertex AI Endpoint metrics are integrated with Cloud Monitoring. Specific metrics like 'predictions/failed_count' can be viewed in Cloud Monitoring dashboards.

Answer

Cloud Console endpoint details page

Answer

Vertex AI Experiments

Answer

Cloud Logging

Question 6

A company wants to implement a retraining trigger for their ML model. They have set up Cloud Monitoring alerts that fire when drift exceeds a threshold. What should be the target of the alert to automatically start a Vertex AI Pipeline for retraining?

Accepted Answer

Pub/Sub topic. Cloud Monitoring alerts can send notifications to Pub/Sub topics. A Pub/Sub message can then trigger a Cloud Function that starts a Vertex AI Pipeline run.

Answer

Vertex AI Model Registry

Answer

Cloud Storage bucket

Answer

Cloud Functions HTTP trigger

Question 7

A team uses Vertex AI Explainable AI with integrated gradients for a deep learning model. They want to reduce the computational cost of explanations without significantly reducing explanation quality. Which configuration change should they make?

Accepted Answer

Reduce the number of integral approximation steps.. Integrated gradients approximates Shapley values by integrating gradients along a path. Reducing the number of steps (integral approximation steps) reduces computation, but may reduce quality. A moderate reduction balances cost and quality.

Answer

Switch from integrated gradients to XRAI.

Answer

Apply feature attribution to only a random subset of predictions.

Answer

Use sampled Shapley instead, as it is always cheaper.

Question 8

An organisation wants to monitor fairness of their loan approval model across demographic subgroups. They have predictions stored in BigQuery along with ground truth. Which GCP service can evaluate model performance for each subgroup and identify disparities?

Accepted Answer

Vertex AI Model Evaluation. Vertex AI Model Evaluation supports sliced evaluation, where metrics are computed for each subgroup defined by feature values (e.g., gender, race) in BigQuery. This helps identify performance disparities.

Answer

Cloud Data Loss Prevention (DLP)

Answer

Vertex AI Explainable AI

Answer

Vertex AI Model Monitoring

Question 9

A company wants to log all prediction requests and responses from a Vertex AI Endpoint to BigQuery for auditing and debugging. How can they achieve this?

Accepted Answer

Enable request/response logging on the endpoint and create a BigQuery sink for the log.. Vertex AI endpoints can be configured to enable request/response logging. The logs can be sent to a BigQuery table via a log sink.

Answer

Export endpoint logs from Cloud Logging to Cloud Storage and then load into BigQuery manually.

Answer

Use a Cloud Function to intercept predictions and write to BigQuery.

Answer

Vertex AI endpoints do not support request/response logging.

Question 10

An ML engineer is monitoring a model on Vertex AI Endpoint and sees that feature 'age' has a training distribution of (mean=45, std=10) but the serving distribution over the last hour shows (mean=30, std=15). JS divergence is 0.12, but the alert threshold is 0.1. The engineer suspects this is due to a temporary campaign targeting younger users. What should they do first?

Accepted Answer

Use Vertex AI Explainable AI to understand the importance of the 'age' feature for the model.. Before taking action like retraining, the engineer should investigate the root cause. Using Vertex AI Explainable AI to check feature importance can confirm whether 'age' is a critical feature. If it is not important, the drift may be harmless.

Answer

Immediately trigger retraining to avoid model degradation.

Answer

Increase the sampling rate to get more data before making a decision.

Answer

Adjust the alert threshold to 0.2 to avoid false positives.

Question 11

A team wants to monitor prediction drift on a Vertex AI Endpoint for a classification model. They have configured Vertex AI Model Monitoring with default settings. Which metric will be used to detect prediction drift?

Accepted Answer

Jensen-Shannon divergence. Vertex AI Model Monitoring uses Jensen-Shannon divergence as the default algorithm for prediction drift detection. It compares the distribution of predicted classes over time against a reference distribution (e.g., training predictions).

Answer

Kullback-Leibler divergence

Answer

L-infinity distance

Answer

Population Stability Index (PSI)

Question 12

A company wants to track the cost of their Vertex AI prediction endpoint. They use a custom machine type with 1 n1-standard-4 (4 vCPU, 15 GB memory) and 1 NVIDIA T4 GPU. The endpoint is configured for automatic scaling with min=1, max=5 replicas. Which cost monitoring approach should they use?

Accepted Answer

Use Cloud Billing budget alerts and export cost data to BigQuery for analysis.. Vertex AI prediction costs are composed of per-hour per-replica compute and GPU charges plus per-request usage. The best approach is to use Cloud Billing export to BigQuery and query cost data by service and SKU, or use the Vertex AI cost tables in the console.

Answer

Calculate cost manually based on replica count and GPU hours from endpoint logs.

Answer

Use Vertex AI Experiments to track cost.

Answer

Monitor only the CPU utilisation metrics to infer cost.

Question 13

A data science team is configuring Vertex AI Model Monitoring for a deployed model. They want to detect both feature skew and feature drift. Which TWO configurations must they set?

Accepted Answer

Configure the monitoring frequency (e.g., hourly).. To detect feature skew, they need to specify a training dataset or statistics as a reference. To detect drift over time, they need to set a monitoring frequency. Sampling rate is optional but recommended.

Answer

Enable request/response logging.

Answer

Select the Jensen-Shannon divergence algorithm.

Answer

Set the sampling rate to 1.0 (100%).

Question 14

An ML engineer is troubleshooting why a Vertex AI Endpoint is returning high prediction latency. They have enabled request/response logging and see that some requests take >1 second while most are fast. Which THREE actions should they take to diagnose the issue?

Accepted Answer

Check CPU and GPU utilisation metrics on the endpoint.. Check Cloud Monitoring metrics for p99 latency trends, CPU/GPU utilisation to identify bottlenecks, and review request/response logs for patterns (e.g., large payloads causing slow inference).

Answer

Disable Vertex AI Model Monitoring to reduce overhead.

Answer

Increase the number of replicas to resolve the issue immediately.

Question 15

A company wants to automatically retrain their model when data drift is detected. Which THREE components are needed to implement this pipeline?

Accepted Answer

Cloud Function to invoke Vertex AI Pipeline. The typical flow: Cloud Monitoring alert on drift → Pub/Sub topic → Cloud Function → Vertex AI Pipeline for training. Model Registry stores the new model after training.

Answer

Vertex AI Feature Store

Answer

Cloud Storage bucket for storing training data