PMLE Monitoring ML solutions Practice Test 2 — 15 Questions

Question 1

A company implements an ML pipeline using Vertex AI Pipelines. The pipeline trains a model using custom training jobs and then deploys it to an endpoint. The team notices that the endpoint occasionally serves an older model version for a few minutes after a new pipeline run completes. What is the most likely cause?

Accepted Answer

The endpoint is configured with a canary traffic split, and the old model is still receiving a fraction of traffic during the rollout.. D is correct because Vertex AI endpoints can be configured with a canary (gradual) traffic rollout strategy. When a new model is deployed, traffic is shifted incrementally from the old model to the new one over a specified duration. During this rollout window, the old model continues to serve a fraction of requests, which explains why users occasionally see the older model version for a few minutes after the pipeline completes.

Answer

The new model artifact is temporarily unavailable, so the endpoint falls back to the previous version.

Answer

The prediction cache is returning cached results from the old model.

Answer

The pipeline failed to update the endpoint with the new model ID.

Question 2

A team has deployed a model on Vertex AI Prediction and wants to monitor for data drift. Which TWO metrics should they use to detect drift in numerical features?

Accepted Answer

Jensen-Shannon divergence (JSD). Jensen-Shannon divergence (JSD) is a symmetric, bounded (0 to 1) measure of the difference between two probability distributions, making it ideal for detecting drift in numerical features by comparing the training distribution to the serving distribution. It is a smoothed and normalized version of Kullback-Leibler divergence, and Vertex AI Prediction's Model Monitoring natively supports JSD for numerical feature drift detection.

Answer

Pearson correlation coefficient

Answer

Chi-squared statistic

Answer

Population Stability Index (PSI)

Question 3

A company uses Vertex AI Model Monitoring to detect training-serving skew. They have a categorical feature 'product_category' with high cardinality. The monitoring job alerts for skew, but the data scientists believe the model performance is still acceptable. Which THREE actions should the team take to investigate and resolve the alert?

Accepted Answer

Examine which categories have the largest distribution changes to understand the nature of the shift.. Option A is correct because examining which categories have the largest distribution changes allows the team to pinpoint the root cause of the training-serving skew. In Vertex AI Model Monitoring, the skew alert is based on statistical distance metrics (e.g., Jensen-Shannon divergence) between training and serving distributions. By drilling down into the specific categories driving the divergence, the team can assess whether the shift is benign (e.g., seasonal) or problematic, rather than relying on aggregate model performance alone.

Answer

Remove the drifted categories from the feature set to eliminate the alert.

Answer

Ignore the alert because the model is performing well; monitoring alerts are often false positives.

Question 4

You are an ML engineer at a logistics company. You have deployed a deep learning model on Vertex AI Endpoints using a custom container with GPU acceleration. The model predicts delivery times based on route features. After one week, you notice that the endpoint's GPU utilization is consistently at 10%, but the prediction latency has increased by 50%. The number of prediction requests per second has remained stable. You check the container logs and see no errors. The model is served using TensorFlow Serving with batching enabled (batch size: 32, batch timeout: 100ms). The custom container uses a single NVIDIA T4 GPU. You have also set the Vertex AI endpoint to use autoscaling with minReplicaCount: 1 and maxReplicaCount: 5, and the CPU utilization target is 60%. Which action should you take to reduce latency?

Accepted Answer

Increase the batch size to 64 and batch timeout to 200ms to improve GPU utilization.. The core issue is low GPU utilization (10%) despite increased latency, indicating that the GPU is underutilized and the bottleneck is likely in batching or data pipeline overhead. Increasing the batch size to 64 and batch timeout to 200ms allows TensorFlow Serving to accumulate more requests per batch, improving GPU throughput and reducing per-request latency by better leveraging GPU parallelism. This directly addresses the mismatch between low GPU utilization and high latency.

Answer

Increase the minReplicaCount to 3 to handle requests in parallel.

Answer

Reduce the CPU utilization target to 40% to trigger more aggressive autoscaling.

Answer

Quantize the model to FP16 to reduce compute time per inference.

Question 5

A company deploys a custom ML model on Vertex AI to predict customer churn. The model retrains weekly, and predictions are served via a Vertex AI endpoint. After a recent retraining, the monitoring dashboard shows a sudden increase in prediction requests but a decrease in predicted churn probabilities. The model's accuracy on the validation set remains stable. What is the most likely cause of the observed behavior?

Accepted Answer

The incoming data distribution has changed, e.g., due to a new marketing campaign attracting different customers.. Option C is correct because a sudden increase in prediction requests alongside a decrease in predicted churn probabilities, while validation accuracy remains stable, indicates a shift in the incoming data distribution (covariate shift). This is typical when a new marketing campaign attracts a different customer segment that inherently has lower churn risk. The model itself hasn't degraded; it's simply seeing a different population than it was trained on, which changes the base rate of churn in the live traffic.

Answer

A training-serving skew exists between the training pipeline and the serving endpoint.

Answer

Concept drift has occurred, changing the relationship between features and churn.

Answer

Data leakage during training caused the model to overfit to historical patterns.

Question 6

A financial services company has deployed a classification model on Vertex AI to detect fraudulent transactions. The model is monitored using Vertex AI Model Monitoring for skew and drift detection, and also logs predictions to BigQuery for analysis. After a month, the monitoring alerts show a significant drift in one feature (transaction_amount). Which TWO actions should the team take to diagnose and address this issue?

Accepted Answer

Compare the feature distribution in the training data with the recent serving data using statistical tests.. Option A is correct because comparing the feature distribution of the training data with recent serving data using statistical tests (e.g., Kolmogorov-Smirnov or Jensen-Shannon divergence) is the standard first step to quantify the drift and confirm it is statistically significant. This diagnostic action helps the team understand the nature and magnitude of the drift before deciding on remediation steps. Vertex AI Model Monitoring already performs such comparisons, but the team should independently verify the results in BigQuery to ensure accuracy.

Answer

Increase the frequency of model monitoring checks to every hour.

Answer

Increase the sampling rate for prediction logging to ensure full data capture.

Answer

Reduce the alert threshold to minimize false positives.

Question 7

Drag and drop the steps to set up a distributed training job on Vertex AI using a custom container in the correct order.

Question 8

Match each Google Cloud storage option to its best use case.

Question 9

A company deploys a classification model on Vertex AI for loan approval. After a month, they notice the precision has dropped significantly. What should they do first?

Accepted Answer

Check for data drift using Vertex AI Model Monitoring. Option C is correct because a sudden drop in precision indicates that the model's predictions are no longer aligning with the ground truth, which is a classic symptom of data drift. Vertex AI Model Monitoring can automatically detect drift in feature distributions or prediction output compared to a baseline, allowing you to identify the root cause before taking corrective action. Retraining or reverting without first diagnosing the drift could waste resources or mask the underlying issue.

Answer

Retrain the model with more data

Answer

Increase the number of prediction nodes

Answer

Revert to the previous model version

Question 10

A team uses custom training and deploys a TensorFlow model using Vertex AI Endpoints. They set up Cloud Monitoring alerts for online prediction latency. However, they notice the latency metric shows a spike every hour, but the actual user experience is fine. What could be the cause?

Accepted Answer

The metric includes prediction time plus log writing time. Option A is correct because Vertex AI Endpoints' latency metric includes both the model inference time and the time taken to write prediction logs to Cloud Logging. This log writing occurs asynchronously but can cause periodic spikes in the reported latency metric when log buffers flush, even though the actual user-facing prediction latency remains unaffected. The spike every hour aligns with log rotation or buffer flush intervals, not with actual prediction performance degradation.

Answer

The alert threshold is too low

Answer

The metric is being sampled every hour

Answer

A monitoring agent on the VM is causing additional latency

Question 11

A machine learning engineer wants to monitor model performance on Vertex AI for a regression model. Which metric is most appropriate to track the average prediction error?

Accepted Answer

RMSE. RMSE (Root Mean Squared Error) is the most appropriate metric for tracking average prediction error in a regression model because it measures the standard deviation of residuals (prediction errors) in the same units as the target variable. On Vertex AI, RMSE is a built-in evaluation metric for regression models, directly quantifying how far predictions deviate from actual values on average.

Answer

F1 score

Answer

Precision

Answer

Accuracy

Question 12

A company uses Vertex AI Model Monitoring to detect data drift. They have a model that predicts house prices. Which dataset should they compare against the training data to detect drift?

Accepted Answer

The latest batch of predictions. Option C is correct because Vertex AI Model Monitoring compares the training data (serving as the baseline) against the latest batch of predictions to detect data drift. This batch represents the most recent inference requests, allowing the monitoring service to compute statistical distribution differences (e.g., Jensen-Shannon divergence) and trigger alerts when drift exceeds a configured threshold. Using the latest batch ensures timely detection of shifts in the production data distribution.

Answer

The entire historical prediction data

Answer

A random sample of recent predictions

Answer

The validation data used during training

Question 13

After setting up model monitoring on Vertex AI for a classification model, the engineer sees a high number of anomaly alerts for the "age" feature. Upon investigation, the age distribution in recent predictions is similar to training data. What might be the cause?

Accepted Answer

The monitoring threshold for age is too low. Option C is correct because the high number of anomaly alerts despite the age distribution being similar to training data indicates that the monitoring threshold for the 'age' feature is set too low. In Vertex AI Model Monitoring, anomaly detection compares recent prediction distributions against a baseline using statistical tests (e.g., the Kolmogorov-Smirnov test for numerical features). If the threshold is too sensitive, even minor, statistically insignificant deviations can trigger alerts, leading to false positives even when the distribution is essentially unchanged.

Answer

The feature importance of age has changed

Answer

The monitoring baseline was incorrectly set

Answer

The model is overfitting to age

Question 14

A data scientist wants to log prediction inputs and outputs for model monitoring. Which Google Cloud service is best suited for this?

Accepted Answer

Cloud Logging. Cloud Logging is the best choice because it is designed to ingest, store, and analyze log data, including custom log entries from applications. The data scientist can use the Cloud Logging API to write structured log entries containing prediction inputs and outputs, then query them using Logs Explorer or export them for further analysis. This aligns with the requirement to log prediction inputs and outputs for model monitoring, as Cloud Logging provides a centralized, scalable, and queryable log management service.

Answer

Cloud Monitoring

Answer

Cloud Storage

Answer

BigQuery

Question 15

A team deploys a model using Vertex AI and wants to monitor for concept drift. What should they track?

Accepted Answer

Changes in the relationship between inputs and outputs. Concept drift refers to a change in the underlying relationship between the input features and the target variable over time, which degrades model performance. In Vertex AI, monitoring this requires tracking the statistical relationship between inputs and outputs (e.g., via prediction residuals or model performance metrics), not just the input distribution alone. Option D correctly identifies this need, as concept drift is fundamentally about the input-output mapping shifting, even if the input distribution remains stable.

Answer

Number of prediction requests

Answer

Model prediction latency

Answer

Changes in input data distribution