PMLE Monitoring ML Solutions Practice Test 4 — 15 Questions

Question 1

An ML engineer needs to monitor the online prediction latency of a Vertex AI Endpoint. Which metrics should they look at in Cloud Monitoring?

Accepted Answer

p50, p95, p99 latency. To monitor online prediction latency on a Vertex AI Endpoint, the engineer should look at percentile latency metrics such as p50, p95, and p99. These metrics provide a distribution of latency, showing typical and tail latencies, which are crucial for understanding user experience and identifying outliers.

Answer

Request count and error rate

Answer

Skew and drift scores

Answer

CPU/GPU utilization

Question 2

A model deployed on a Vertex AI Endpoint uses an image model with XRAI explainability. The team notices that the prediction distributions are shifting over time. They want to monitor prediction drift. However, the explainability feature is not enabled. What must the engineer do to enable monitoring prediction drift?

Accepted Answer

Configure Vertex AI Model Monitoring to monitor prediction drift. Vertex AI Model Monitoring is the managed service that detects prediction drift (and training-serving skew) by comparing live prediction distributions against a baseline. It operates independently of explainability features — XRAI is only for feature attribution, not drift detection. Enabling Model Monitoring on the endpoint is the correct, purpose-built action to monitor prediction drift.

Answer

Re-deploy the model with a sampling rate of 100%

Answer

Enable Vertex AI Explainability with XRAI on the endpoint deployment

Answer

Enable request/response logging to BigQuery and build custom drift detection

Question 3

An ML engineer needs to track the costs incurred by Vertex AI prediction endpoints. Which tool should they use to set budget alerts and monitor spending?

Accepted Answer

Google Cloud Budgets & Alerts. Google Cloud Budgets & Alerts is the dedicated billing-scoped tool that lets you define a budget amount (or filter by project/service/label) and attach alert thresholds at percentages of actual or forecasted spend. It is the only option that operates on billing data and can trigger Pub/Sub notifications or email alerts when Vertex AI endpoint costs cross a threshold. Model Monitoring, Cloud Logging, and Cloud Monitoring track operational metrics, not monetary spend.

Answer

Vertex AI Model Monitoring

Answer

Cloud Logging

Answer

Cloud Monitoring with custom metrics

Question 4

A data scientist has deployed a model with Vertex AI Endpoints and enabled request/response logging to BigQuery. They want to compute a confusion matrix over time to monitor model quality. What should they do?

Accepted Answer

Upload ground truth labels to BigQuery and join with prediction logs, then compute confusion matrix in a scheduled query. To compute a confusion matrix over time for a deployed model on Vertex AI Endpoints with request/response logging to BigQuery, you need ground truth labels. Vertex AI does not automatically generate confusion matrices from prediction logs alone. You must upload the actual labels (ground truth) to BigQuery and join them with the prediction logs, then compute the confusion matrix using a scheduled query. This allows you to monitor model quality over time. Vertex AI Model Monitoring (A) can detect skew and drift but does not automatically generate confusion matrices. Cloud Monitoring (B) is for infrastructure and application metrics, not model quality. Explainability (D) provides feature attributions, not confusion matrices.

Answer

Use Vertex AI Model Monitoring to automatically generate confusion matrices

Answer

Use Cloud Monitoring to create a confusion matrix dashboard

Answer

Enable Vertex AI Explainability to get confusion matrix

Question 5

An ML team is using Population Stability Index (PSI) to monitor feature drift on a Vertex AI Endpoint. The PSI value for a feature is 0.25, which exceeds the alert threshold of 0.2. The feature has high SHAP importance. The team wants to automatically retrain the model. What is the correct end-to-end setup?

Accepted Answer

Cloud Monitoring alert → Pub/Sub → Cloud Functions → Vertex AI Pipeline (with training and deployment steps). The correct end-to-end flow is: Vertex AI Model Monitoring detects the PSI threshold breach and emits a metric to Cloud Monitoring; a Cloud Monitoring alerting policy fires and publishes to a Pub/Sub topic; a Cloud Function subscribes and triggers a Vertex AI Pipeline that retrains and redeploys the model. This chain uses the native monitoring-to-automation path on GCP.

Answer

Vertex AI Model Monitoring alert → Cloud Scheduler → Vertex AI Training

Answer

Cloud Functions → Pub/Sub → Cloud Monitoring → Vertex AI Pipeline

Answer

Cloud Monitoring alert → Cloud Logging → Cloud Functions → Vertex AI Training

Question 6

An ML engineer wants to monitor the performance of a Vertex AI Endpoint. Which TWO metrics are available in Cloud Monitoring for Vertex AI Endpoints? (Choose 2)

Accepted Answer

Error count. Vertex AI Endpoints automatically publish request-level metrics to Cloud Monitoring, including aiplatform.googleapis.com/endpoint/error_count, which tracks failed prediction requests and is essential for detecting serving problems. Option E (Prediction latency (p50, p95, p99)) is correct because Vertex AI Endpoints expose latency metrics such as aiplatform.googleapis.com/endpoint/prediction_latencies, which Cloud Monitoring reports as percentiles (p50, p95, p99) to characterize response-time distribution. Option A (Model accuracy) is not a built-in Cloud Monitoring metric for Endpoints; accuracy must be computed separately, for example via Vertex AI Model Monitoring or custom evaluation jobs. Option C (Feature skew score) belongs to Vertex AI Model Monitoring's skew/drift detection outputs, not to the standard Endpoint metrics in Cloud Monitoring. Option D (SHAP values) are explainability artifacts produced by Vertex Explainable AI, not time-series metrics available for an Endpoint in Cloud Monitoring.

Answer

Model accuracy

Answer

Feature skew score

Answer

SHAP values

Question 7

A data scientist has deployed a classification model on a Vertex AI Endpoint and wants to monitor for feature drift in the serving data compared to the training data. Which Vertex AI service should be used?

Accepted Answer

Vertex AI Model Monitoring. Vertex AI Model Monitoring is the purpose-built service that detects training-serving skew and prediction drift by comparing live serving data distributions against a baseline (typically the training dataset). It computes statistics like Jensen-Shannon divergence and can alert when drift exceeds configured thresholds. Explainable AI, Model Evaluation, and Continuous Training serve different purposes and do not provide ongoing drift detection on deployed endpoints.

Answer

Vertex AI Explainable AI

Answer

Vertex AI Model Evaluation

Answer

Vertex AI Continuous Training

Question 8

An ML engineer has set up Vertex AI Model Monitoring on an endpoint with a sampling rate of 0.1 (10%). They notice that the monitoring job runs hourly but the reported drift metrics seem inconsistent. What is the most likely cause?

Accepted Answer

The sampling rate is too low, leading to insufficient data for reliable drift statistics.. A sampling rate of 0.1 means only 10% of prediction requests are logged and analyzed, which can produce statistically noisy or inconsistent drift metrics, especially for low-traffic endpoints. Drift detection relies on sufficient sample size to compute reliable distribution comparisons (e.g., JS divergence), so a low sampling rate is the most likely cause of the inconsistency.

Answer

Prediction drift monitoring is not enabled; only feature drift is configured.

Answer

The drift detection algorithm is not suited for this model; try changing from JS divergence to L-infinity distance.

Answer

The monitoring frequency is too low; it should be set to every 5 minutes.

Question 9

A team wants to collect ground truth labels for their model deployed on Vertex AI Endpoint to perform model quality monitoring. They have a process that generates actual outcomes within 24 hours of prediction. What is the recommended approach for storing these labels?

Accepted Answer

Upload the ground truth labels to a BigQuery table with a schema that includes prediction timestamp and model version.. Vertex AI Model Monitoring expects ground truth labels to be stored in a BigQuery table whose schema includes the prediction timestamp and model version (plus the prediction output and the actual label). This lets the service join ground truth back to logged predictions within the monitoring window and compute quality metrics like accuracy, precision, and recall. BigQuery is the only supported sink for ground truth in Vertex AI model quality monitoring.

Answer

Use Vertex AI Experiments to log ground truth alongside training runs.

Answer

Store the ground truth labels in Cloud Storage as CSV files and reference them in the monitoring config.

Answer

Insert ground truth labels directly into the Vertex AI Endpoint's log sink.

Question 10

An ML engineer needs to monitor the error rate of prediction jobs on a Vertex AI Endpoint. Where can they view the number of failed prediction requests over time?

Accepted Answer

Cloud Monitoring. Cloud Monitoring (formerly Stackdriver) is the Google Cloud service that collects, stores, and visualizes metrics such as prediction request counts and error rates for Vertex AI Endpoints. It provides dashboards, alerting, and time-series charts for failed prediction requests over time. This is the correct place to monitor error rate metrics.

Answer

Cloud Console endpoint details page

Answer

Vertex AI Experiments

Answer

Cloud Logging

Question 11

An organisation wants to monitor fairness of their loan approval model across demographic subgroups. They have predictions stored in BigQuery along with ground truth. Which GCP service can evaluate model performance for each subgroup and identify disparities?

Accepted Answer

Vertex AI Model Evaluation. Vertex AI Model Evaluation is the correct answer because it provides built-in functionality to compute performance metrics (e.g., accuracy, precision, recall, AUC) for specific slices of data, such as demographic subgroups. It allows you to evaluate fairness by comparing metrics across these subgroups to identify disparities. This service directly supports the requirement to monitor fairness by analyzing model performance on different segments.

Answer

Cloud Data Loss Prevention (DLP)

Answer

Vertex AI Explainable AI

Answer

Vertex AI Model Monitoring

Question 12

A company wants to log all prediction requests and responses from a Vertex AI Endpoint to BigQuery for auditing and debugging. How can they achieve this?

Accepted Answer

Enable request/response logging on the endpoint and create a BigQuery sink for the log.. Vertex AI endpoints can be configured to enable request/response logging. The logs can be sent to a BigQuery table via a log sink.

Answer

Export endpoint logs from Cloud Logging to Cloud Storage and then load into BigQuery manually.

Answer

Use a Cloud Function to intercept predictions and write to BigQuery.

Answer

Vertex AI endpoints do not support request/response logging.

Question 13

A company wants to track the cost of their Vertex AI prediction endpoint. They use a custom machine type with 1 n1-standard-4 (4 vCPU, 15 GB memory) and 1 NVIDIA T4 GPU. The endpoint is configured for automatic scaling with min=1, max=5 replicas. Which cost monitoring approach should they use?

Accepted Answer

Use Cloud Billing budget alerts and export cost data to BigQuery for analysis.. For tracking Vertex AI prediction endpoint costs, the most comprehensive approach is to use Cloud Billing budget alerts and export cost data to BigQuery for detailed analysis. This allows the company to monitor actual spend, set alerts, and analyze costs by labels, projects, or services, including Vertex AI endpoints with custom machine types and GPUs.

Answer

Calculate cost manually based on replica count and GPU hours from endpoint logs.

Answer

Use Vertex AI Experiments to track cost.

Answer

Monitor only the CPU utilisation metrics to infer cost.

Question 14

A data science team is configuring Vertex AI Model Monitoring for a deployed model. They want to detect both feature skew and feature drift. Which TWO configurations must they set?

Accepted Answer

Configure the monitoring frequency (e.g., hourly).. Option C is correct because Vertex AI Model Monitoring requires a monitoring schedule/frequency (for example hourly or daily) so the job knows how often to evaluate incoming prediction data against the baseline; without a configured frequency, no monitoring analysis runs. Option E is correct because detecting feature skew and drift requires a baseline: for skew you supply the training dataset (or its generated statistics), and for drift you compare recent production data against the baseline statistics, so specifying a training dataset or statistics is mandatory. Option A is not required because request/response logging is a separate feature for capturing payloads and is not the configuration that enables skew/drift detection. Option B is not required because Jensen-Shannon divergence is only one selectable distance metric for drift/skew detection, not a mandatory setting. Option D is not required because a 1.0 sampling rate (100%) is optional; monitoring can run on a lower sample of requests.

Answer

Enable request/response logging.

Answer

Select the Jensen-Shannon divergence algorithm.

Answer

Set the sampling rate to 1.0 (100%).

Question 15

An ML engineer is troubleshooting why a Vertex AI Endpoint is returning high prediction latency. They have enabled request/response logging and see that some requests take >1 second while most are fast. Which THREE actions should they take to diagnose the issue?

Accepted Answer

Check CPU and GPU utilisation metrics on the endpoint.. Option A is correct because high prediction latency on a Vertex AI Endpoint is frequently caused by resource saturation, so inspecting CPU and GPU utilisation metrics in Cloud Monitoring reveals whether the underlying nodes are maxed out and throttling inference. Option B is correct because request/response logging for Vertex AI Endpoints can be routed to BigQuery, where the engineer can correlate payload size with latency and confirm whether large inputs (e.g., big images or long text) are the outliers exceeding 1 second. Option D is correct because p99 latency in Cloud Monitoring exposes the tail behaviour of the distribution, showing whether the slow requests are a persistent tail problem or isolated spikes, which is exactly the symptom described. Option C is not appropriate because Model Monitoring runs asynchronously on sampled data and does not add per-request prediction latency, so disabling it would not diagnose the issue. Option E is not appropriate because blindly scaling replicas is a remediation step, not a diagnostic one, and it would not identify the root cause of the >1 second requests.

Answer

Disable Vertex AI Model Monitoring to reduce overhead.

Answer

Increase the number of replicas to resolve the issue immediately.