PMLE Monitoring ML Solutions Practice Test 2 — 15 Questions

Question 1

A financial services company deploys a model on Vertex AI Endpoints with GPU acceleration. They notice that the p99 latency for predictions has increased from 200ms to 1.2s over the past week. CPU utilisation is low, but GPU utilisation is high. Which action should they take to reduce latency?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Increase the number of replicas in the endpoint deployment.. High GPU utilisation with low CPU utilisation indicates the GPU is the bottleneck, and adding more replicas distributes the inference load across additional GPU-backed instances, directly reducing per-request latency. Increasing replicas scales horizontally, which is the correct action when a single instance's GPU is saturated. This addresses the p99 latency spike without changing the model or hardware type.

Answer

Increase the CPU machine type for the endpoint.

Answer

Switch to a more powerful GPU type (e.g., from T4 to A100).

Answer

Reduce the sampling rate for monitoring to free up resources.

Question 2

A company wants to implement a retraining trigger for their ML model. They have set up Cloud Monitoring alerts that fire when drift exceeds a threshold. What should be the target of the alert to automatically start a Vertex AI Pipeline for retraining?

Accepted Answer

Pub/Sub topic. Cloud Monitoring alerting policies can route notifications to a Pub/Sub topic, which then acts as the event source that triggers a Vertex AI Pipeline run. Pub/Sub provides the decoupled, asynchronous messaging backbone that lets the alert fire a pipeline execution without manual intervention, typically via an Eventarc or Cloud Function subscriber that calls the Vertex AI Pipelines API.

Answer

Vertex AI Model Registry

Answer

Cloud Storage bucket

Answer

Cloud Functions HTTP trigger

Question 3

A team uses Vertex AI Explainable AI with integrated gradients for a deep learning model. They want to reduce the computational cost of explanations without significantly reducing explanation quality. Which configuration change should they make?

Accepted Answer

Reduce the number of integral approximation steps.. Integrated gradients approximates Shapley values by integrating gradients along a path. Reducing the number of steps (integral approximation steps) reduces computation, but may reduce quality. A moderate reduction balances cost and quality.

Answer

Switch from integrated gradients to XRAI.

Answer

Apply feature attribution to only a random subset of predictions.

Answer

Use sampled Shapley instead, as it is always cheaper.

Question 4

An ML engineer is monitoring a model on Vertex AI Endpoint and sees that feature 'age' has a training distribution of (mean=45, std=10) but the serving distribution over the last hour shows (mean=30, std=15). JS divergence is 0.12, but the alert threshold is 0.1. The engineer suspects this is due to a temporary campaign targeting younger users. What should they do first?

Accepted Answer

Use Vertex AI Explainable AI to understand the importance of the 'age' feature for the model.. Before taking action like retraining, the engineer should investigate the root cause. Using Vertex AI Explainable AI to check feature importance can confirm whether 'age' is a critical feature. If it is not important, the drift may be harmless.

Answer

Immediately trigger retraining to avoid model degradation.

Answer

Increase the sampling rate to get more data before making a decision.

Answer

Adjust the alert threshold to 0.2 to avoid false positives.

Question 5

A team wants to monitor prediction drift on a Vertex AI Endpoint for a classification model. They have configured Vertex AI Model Monitoring with default settings. Which metric will be used to detect prediction drift?

Accepted Answer

Jensen-Shannon divergence. Vertex AI Model Monitoring uses Jensen-Shannon divergence as the default metric for detecting prediction drift on classification models. It measures the similarity between the distribution of live prediction outputs and the training baseline, with a default threshold. KL divergence and PSI are not the default for prediction drift in Vertex AI.

Answer

Kullback-Leibler divergence

Answer

L-infinity distance

Answer

Population Stability Index (PSI)

Question 6

A data scientist notices that the prediction distribution of a deployed model has changed significantly over the past week. They want to identify which features are contributing most to the drift. Which approach should they use?

Accepted Answer

Use Vertex AI Explainability to get feature importance and identify which features have high importance and significant drift. Vertex AI Explainability provides feature attributions (e.g., via Sampled Shapley or Integrated Gradients) that quantify each feature's contribution to individual predictions. By combining these importance scores with drift metrics (e.g., from Model Monitoring), the data scientist can pinpoint which features are both highly influential and drifting, directly addressing the goal of identifying features contributing most to prediction drift. This integrated approach is more actionable than raw distribution comparisons or correlation analysis.

Answer

Compute Pearson correlation between each feature's drift score and the model's prediction drift

Answer

Use Vertex AI Model Monitoring to compare training and serving distributions for each feature

Answer

Enable request/response logging to BigQuery and manually analyze feature distributions

Question 7

A company deploys a model on Vertex AI Endpoints and configures Vertex AI Model Monitoring with a sampling rate of 0.1 and monitoring frequency of every hour. They notice that the monitoring alert fires only after several hours of drift. What is the most likely cause?

Accepted Answer

The sampling rate is too low, causing the drift detection algorithm to require more time to accumulate a representative distribution. The most likely cause is that the sampling rate of 0.1 is too low, meaning only 10% of requests are analyzed. With low traffic, it takes longer to accumulate a representative sample of the data distribution, delaying drift detection. The monitoring frequency of every hour is not the issue; even with hourly checks, if the sample size is insufficient, drift alerts will be delayed until enough data is collected.

Answer

The alerting threshold is set too high; it should be lowered

Answer

The monitoring frequency is too low; it should be every minute

Answer

The model endpoint is not receiving enough traffic to generate a statistically significant sample

Question 8

An ML team wants to automatically retrain a model when prediction drift is detected on the deployed endpoint. They have Vertex AI Model Monitoring configured to send alerts to Cloud Monitoring. Which minimal set of additional services should they use to trigger a retraining pipeline?

Accepted Answer

Cloud Monitoring + Cloud Functions + Vertex AI Pipeline. Cloud Monitoring receives the drift alert from Vertex AI Model Monitoring, Cloud Functions acts as the event-driven trigger that reacts to the alert, and Vertex AI Pipeline orchestrates the retraining workflow. This trio forms the minimal serverless chain from alert to retraining without introducing unnecessary components.

Answer

Cloud Monitoring + Cloud Functions + Vertex AI Endpoint

Answer

Cloud Logging + Cloud Scheduler + Vertex AI Training

Answer

Cloud Monitoring + Cloud Pub/Sub + Cloud Functions + Vertex AI Pipeline

Question 9

Which algorithm does Vertex AI Model Monitoring use by default to detect feature drift in a categorical feature?

Accepted Answer

Jensen-Shannon divergence. Vertex AI Model Monitoring uses Jensen-Shannon divergence by default to detect feature drift in categorical features. This metric measures the similarity between two probability distributions and is symmetric, making it suitable for comparing training and serving distributions.

Answer

Population Stability Index (PSI)

Answer

Wasserstein distance

Answer

Kullback-Leibler divergence

Question 10

An ML engineer wants to monitor the latency of online predictions from a Vertex AI Endpoint. They need to track p50, p95, and p99 latency over time and set up alerts if p99 exceeds 1 second. Which approach should they take?

Accepted Answer

Configure a Cloud Monitoring dashboard using the built-in Vertex AI Endpoint metrics for latency. Configuring a Cloud Monitoring dashboard using the built-in Vertex AI Endpoint metrics for latency is correct because Vertex AI automatically publishes metrics such as prediction latency, request count, and error rate to Cloud Monitoring under the 'aiplatform.googleapis.com/Endpoint' resource. These metrics include percentiles (p50, p95, p99) and can be charted directly in a dashboard. Alerts can be created on these metrics using Cloud Monitoring alerting policies, with a threshold condition on the p99 latency metric exceeding 1 second.

Answer

Use Cloud Monitoring custom metrics and publish latency percentiles from the application code

Answer

Enable request/response logging to BigQuery and calculate latency percentiles using SQL queries

Answer

Use Vertex AI Model Monitoring to track prediction latency

Question 11

An organization wants to collect ground truth labels for model quality monitoring and store them in BigQuery. They also want to compute and visualize a confusion matrix over time. Which TWO actions should they take? (Choose 2)

Accepted Answer

Use Vertex AI Model Evaluation to compare predictions with ground truth and generate confusion matrix. Option E is correct because storing the actual observed labels (ground truth) in a BigQuery table is exactly how Vertex AI Model Monitoring ingests ground truth for continuous model quality monitoring; the table must contain the prediction timestamps and true labels so the service can join them with logged predictions. Option D is correct because Vertex AI Model Evaluation can compare logged predictions against the ground truth labels and produce evaluation metrics including a confusion matrix, which can then be visualized over time in the model monitoring/Model Evaluation UI. Option A is incorrect because prediction drift monitoring detects distribution shifts in incoming feature data, not ground truth label quality or confusion matrices. Option B is incorrect because sliced evaluation evaluates a model on a static dataset across data slices; it does not collect ground truth or track a confusion matrix over time. Option C is incorrect because Cloud Logging request/response logging captures prediction payloads but does not store ground truth labels in BigQuery or compute a confusion matrix.

Answer

Configure Vertex AI Model Monitoring for prediction drift

Answer

Use Vertex AI Model Evaluation to run sliced evaluation

Answer

Enable request/response logging to Cloud Logging

Question 12

An engineer wants to set up request/response logging for a Vertex AI Endpoint to analyze prediction behavior. Which TWO resources must be configured? (Choose 2)

Accepted Answer

A Cloud Logging log sink that routes endpoint logs to BigQuery. Request/response logging on a Vertex AI Endpoint captures prediction payloads and delivers them through Cloud Logging, so you must configure a Cloud Logging log sink that routes the endpoint's logs to BigQuery (option C) — the sink is the mechanism that actually exports the request/response log entries out of Logging. You also need a BigQuery dataset (option D) as the sink's destination, since the sink must write to an existing dataset where the logged prediction requests and responses are stored for analysis. Model Monitoring (option A) is a separate feature for detecting skew/drift and does not produce request/response logs, a Pub/Sub topic (option B) is not the required destination for this logging feature, and Cloud Functions (option E) are not needed because the sink writes directly to BigQuery without transformation.

Answer

Vertex AI Model Monitoring enabled

Answer

A Cloud Pub/Sub topic for streaming logs

Answer

Cloud Functions to transform logs

Question 13

A retail company has a model deployed to a Vertex AI Endpoint that predicts customer churn. The ML team has enabled Vertex AI Model Monitoring with training-serving skew detection on the top 10 features, using a monitoring frequency of 24 hours. After two weeks, they notice that no skew alerts have fired even though they suspect the input feature distributions have shifted. They confirm that monitoring is enabled and the endpoint is receiving traffic. What is the most likely reason for the lack of alerts?

Accepted Answer

The monitoring job compares incoming data to the training dataset, but the training dataset used for skew detection was not provided or is outdated.. Training-serving skew detection in Vertex AI Model Monitoring relies on a training dataset baseline to compare against live traffic. If that baseline is missing, inaccessible, or outdated, skew metrics may not be computed correctly, and alerts will not fire even if distributions shift. The team should verify that the monitoring job has a valid training schema and dataset, and that alerting policies are configured on the skew metrics.

Answer

Cloud Monitoring alerts for skew must be manually created; Vertex AI Model Monitoring does not send alerts by default.

Answer

The monitoring frequency is set to 24 hours, which is too infrequent to catch the shift within two weeks.

Answer

The model monitoring job only analyzes a random sample of requests, and the default sampling rate is too low to detect the shift.

Question 14

A financial institution has deployed a fraud detection model on a Vertex AI Endpoint. The model receives a mix of numerical and categorical features. The MLOps team wants to monitor prediction drift for the categorical features using Vertex AI Model Monitoring. They need to ensure that drift detection works correctly and that alerts are triggered when drift exceeds a threshold. Which TWO configurations are required to achieve this? (Choose two.)

Accepted Answer

Configure the drift threshold for each categorical feature using a distance metric such as Jensen-Shannon divergence.. To monitor prediction drift for categorical features, you must enable prediction drift detection and explicitly include the categorical features in the monitoring configuration. Additionally, you must set drift thresholds for each feature; otherwise, no alerts will be generated. Providing training data is for skew detection, not drift. Monitoring frequency and traffic splits are not required for enabling categorical drift detection.

Answer

Provide a training dataset to generate baseline statistics for the categorical features.

Answer

Set the monitoring frequency to at least every 24 hours to collect enough data.

Answer

Deploy the model with a traffic split to ensure sufficient prediction volume for drift analysis.

Question 15

You are configuring Vertex AI Model Monitoring for a newly deployed model on a Vertex AI Endpoint. You want to detect when the distribution of a key input feature deviates from the training data distribution. Which type of monitoring should you enable?

Accepted Answer

Feature skew detection. Feature skew detection in Vertex AI Model Monitoring compares the distribution of live input features against the training data distribution. It alerts when the live distribution deviates beyond a configured threshold. This is exactly what is needed to detect when a key input feature deviates from the training distribution. Prediction drift, training-serving skew, and model performance monitoring serve different purposes and do not directly address this requirement.

Answer

Prediction drift detection

Answer

Training-serving skew detection

Answer

Model performance monitoring