PMLE Monitoring ML solutions — Timed 20 Questions

Question 1

You have deployed a regression model that predicts house prices. Over the past month, the model's predictions have been consistently too high. You suspect data drift in the input features. Which monitoring metric should you prioritize to confirm this?

Accepted Answer

Monitor feature distribution drift using a divergence metric like Jensen-Shannon divergence. Option B is correct because the question describes a scenario where predictions are consistently too high, which is a symptom of data drift—a change in the distribution of input features. Monitoring feature distribution drift using a divergence metric like Jensen-Shannon divergence directly measures whether the input data has shifted from the training distribution, which would cause the model to make biased predictions. This is the most direct way to confirm data drift in the input features.

Answer

Monitor prediction drift (prediction distribution)

Answer

Monitor feature attribution drift using SHAP values

Answer

Monitor residual distribution drift

Question 2

Your team has deployed a text classification model on Vertex AI Endpoints. You notice that the model's latency has increased significantly over the last week, but the request rate has remained stable. Which of the following is the most likely cause?

Accepted Answer

A change in the preprocessing logic that now includes a computationally expensive step. A computationally expensive preprocessing step directly increases per-request latency on the inference path, even when request rate is stable. Vertex AI Endpoints execute user-provided preprocessing code before model inference, so adding a heavy operation (e.g., large regex, image resizing, or external API call) will linearly increase response time for every prediction.

Answer

A sudden increase in the number of prediction requests

Answer

The model was replaced with a larger version without updating the endpoint

Answer

A misconfiguration in the autoscaling policy

Question 3

You are monitoring a classification model that predicts loan default. The model was trained on data from 2020-2022. In 2023, the economic conditions changed, and the model's accuracy dropped significantly. Which monitoring approach would best help you detect this issue early?

Accepted Answer

Monitor the prediction distribution for significant shift from training distribution. Option C is correct because monitoring the prediction distribution for a significant shift from the training distribution directly detects changes in the model's output behavior, which is the earliest indicator of concept drift or data drift caused by economic changes. Unlike accuracy monitoring, this approach does not require labeled data, enabling real-time detection of performance degradation before ground truth labels become available.

Answer

Monitor the accuracy of the model on the latest batch of labeled data

Answer

Monitor feature distribution drift using KS test

Answer

Monitor the freshness of the training data

Question 4

You are responsible for monitoring a batch prediction pipeline that runs daily. Recently, the pipeline started failing intermittently with out-of-memory errors. The input data volume has not changed. What is the most likely cause?

Accepted Answer

A recent code change that loads the entire dataset into memory before processing. Option A is correct because a code change that loads the entire dataset into memory before processing would directly cause out-of-memory (OOM) errors, even if the input data volume remains unchanged. In batch prediction pipelines, data is typically streamed or processed in chunks to manage memory efficiently. A change that bypasses this pattern and loads all data at once can exceed the available heap or container memory, leading to intermittent failures depending on data characteristics or concurrent loads.

Answer

Increase in model size due to retraining

Answer

Decrease in the number of worker machines

Answer

Increase in input data size

Question 5

You need to set up monitoring for a Vertex AI model that serves predictions in real-time. The model is expected to have a latency SLA of under 100ms. Which metric should you configure an alert on to ensure the SLA is met?

Accepted Answer

p99 latency of prediction requests. Option C is correct because p99 latency measures the worst-case latency experienced by 99% of requests, which is the standard metric for enforcing a strict SLA like under 100ms. Monitoring p99 ensures that even the slowest 1% of requests do not violate the threshold, providing a robust guarantee for real-time predictions.

Answer

p50 latency of prediction requests

Answer

Prediction drift score

Answer

Number of prediction requests per second

Question 6

Your company uses a custom container for model serving on Vertex AI. After a recent update, the model returns predictions but they are clearly wrong (e.g., negative probabilities for a classification model). The logs show no errors. What is the most likely cause?

Accepted Answer

The preprocessing code in the container was updated but the model was not retrained on the new preprocessing. Option A is correct because the most likely cause of a model returning predictions without errors, but with clearly wrong outputs like negative probabilities, is a mismatch between the preprocessing logic used during training and inference. If the preprocessing code in the container was updated (e.g., scaling, normalization, or feature engineering steps changed) but the model was not retrained on data processed with that new logic, the model receives inputs that are out of distribution, leading to nonsensical outputs. Vertex AI containers run inference with the deployed code, so any change in preprocessing directly affects the input tensor values without raising runtime errors.

Answer

The model file is corrupted

Answer

The model file was accidentally replaced with a different model

Answer

The container is using an incompatible version of the serving framework

Question 7

You are monitoring a machine learning pipeline that runs on Vertex AI Pipelines. The pipeline occasionally fails with a 'ResourceExhausted' error when attempting to read data from BigQuery. Which action should you take to resolve this issue?

Accepted Answer

Reduce the complexity of the BigQuery query or increase the reservation size. The 'ResourceExhausted' error when reading from BigQuery indicates that the query is consuming more resources than the BigQuery reservation allows. Option C is correct because reducing query complexity (e.g., using fewer JOINs, aggregations, or partitions) or increasing the reservation size directly addresses the root cause by either lowering resource demand or allocating more capacity. Other options like switching to Cloud Storage or adjusting pipeline memory do not fix the BigQuery-specific quota or slot exhaustion.

Answer

Switch from BigQuery to Cloud Storage for data source

Answer

Increase the memory allocated to the pipeline step

Answer

Reduce the batch size of the data being read

Question 8

You have an online prediction model that is showing increasing prediction latency. You have already verified that the request rate and input data size are unchanged. Which of the following should you investigate next?

Accepted Answer

Check if the model was recently updated to a larger version. If request rate and input data size are unchanged, increased prediction latency often points to a change in the model itself. A larger model (e.g., deeper neural network, more parameters) requires more computation per inference, directly increasing latency. This is a common root cause when monitoring ML pipelines, as model version updates can silently alter performance characteristics.

Answer

Check the monitoring dashboard configuration

Answer

Check if the feature engineering logic was changed

Answer

Check the geographic location of the endpoint

Question 9

Which TWO metrics should you monitor to detect data drift in a batch prediction pipeline?

Accepted Answer

Feature distribution drift (e.g., KS test). Feature distribution drift (C) is correct because it directly measures changes in the input data distribution over time using statistical tests like the Kolmogorov-Smirnov (KS) test, which compares the cumulative distribution of a feature in the current batch against a reference baseline. This is a primary indicator of data drift, as shifts in feature distributions can degrade model performance even if labels are not immediately available.

Answer

Model accuracy on recent labeled data

Answer

Model prediction latency

Answer

Training data size

Question 10

Which THREE components should you include in a comprehensive model monitoring dashboard for a production ML system?

Accepted Answer

System resource utilization (CPU, memory, latency). Option B is correct because system resource utilization metrics (CPU, memory, latency) are essential for monitoring the health and performance of the production infrastructure hosting the ML model. These metrics help detect resource bottlenecks, scaling issues, or degradation that could impact inference latency and throughput, which are critical for maintaining service-level objectives (SLOs).

Answer

Team member roles and responsibilities

Answer

Training pipeline code version

Question 11

Which TWO actions are appropriate when you detect that a production model's prediction distribution has shifted significantly from the training distribution?

Accepted Answer

Retrain the model using the most recent data. Option C is correct because retraining the model on the most recent data directly addresses the distribution shift by adapting the model to the new data patterns. This is a standard practice in MLOps when the shift is confirmed and the cause is understood, ensuring the model remains accurate and reliable in production.

Answer

Immediately roll back to the previous model version

Answer

Increase logging for future predictions

Answer

Reduce the traffic to the model to minimize impact

Question 12

You are the ML engineer for a financial services company. You have deployed a fraud detection model on Vertex AI Endpoints using a custom container. The model is a gradient boosting model trained on transactional data. Over the past week, the model's precision has dropped from 95% to 80%, while recall has remained stable. The input data volume and distribution have not changed significantly. The model is served on a single endpoint with autoscaling enabled (min replicas=2, max replicas=10). You notice that the average CPU utilization of the serving containers has increased from 40% to 90%, and the p99 latency has increased from 50ms to 200ms. The model is retrained weekly using the latest data, and the last retraining was 3 days ago. The logs show no errors, and the model version is unchanged. Given these symptoms, what is the most likely cause of the precision drop?

Accepted Answer

A recent change in the preprocessing code in the container transformed features differently than what the model expects, causing incorrect predictions.. Option C is correct because the precision drop without a change in input distribution or recall strongly indicates a systematic error in predictions, not a data shift. A preprocessing code change in the custom container would cause the model to receive features transformed differently than during training, leading to incorrect probability estimates. The increased CPU utilization and latency are consistent with the container performing additional or different preprocessing steps, not with autoscaling issues or model version changes.

Answer

The autoscaling policy is not scaling up fast enough, causing increased latency and prediction errors.

Answer

The model is overfitting to recent transaction patterns due to weekly retraining.

Answer

The model was replaced with a different version without updating the endpoint.

Question 13

A data science team deploys a regression model to predict house prices. After one month, the mean absolute error (MAE) on the serving data increases by 20% compared to the test set. Which monitoring strategy should the team implement first to diagnose the issue?

Accepted Answer

Monitor prediction residuals and compute serving-time MAE over sliding windows.. Option B is correct because the first step in diagnosing a 20% MAE increase on serving data is to monitor prediction residuals over sliding windows. This directly tracks how model errors evolve in production, allowing the team to detect whether performance degradation is sudden or gradual, and to correlate it with specific time windows or data slices. Computing serving-time MAE on sliding windows provides an immediate, interpretable signal of model health without assuming the root cause.

Answer

Retrain the model daily with the latest data to adapt to changing patterns.

Answer

Compare the distribution of training labels with serving labels using a two-sample t-test.

Answer

Monitor input feature distributions for drift using the Kolmogorov-Smirnov test.

Question 14

An e-commerce company uses a recommendation model deployed on Vertex AI Endpoints. The model's latency increases gradually over two weeks, causing timeouts. The model is served using a custom container. What is the most likely root cause and corrective action?

Accepted Answer

The custom container has a memory leak; implement memory monitoring and set container resource limits.. A gradual increase in latency over two weeks, without a sudden spike, strongly indicates a memory leak in the custom container. As the leak accumulates, the container's garbage collection becomes less effective, leading to increased GC pauses and eventual timeouts. Setting resource limits and monitoring memory usage can prevent the container from exhausting host memory and causing performance degradation.

Answer

The model is receiving more traffic; scale the number of replicas.

Answer

The Vertex AI endpoint has changed its URL; update the client application.

Answer

The model file size has grown due to feature engineering; reduce feature count.

Question 15

A financial services firm deploys a binary classification model for fraud detection. The model's precision is 0.95 and recall is 0.60 on the test set. After deployment, the fraud rate in production is 0.5% compared to 5% in the test set. The model shows good calibration on the test set (Brier score 0.02) but poor calibration in production (Brier score 0.15). What is the most likely explanation for the calibration degradation?

Accepted Answer

The relationship between features and the target has changed (concept drift), causing the model's probability estimates to be misaligned with the true probabilities.. The model's calibration degrades in production despite being well-calibrated on the test set, which had a 5% fraud rate, while production has a 0.5% fraud rate. This shift in class imbalance (prior probability shift) directly affects the model's probability estimates because the model's predicted probabilities are conditional on the training distribution. Option D is correct because concept drift—specifically a change in the base rate of fraud—causes the model's probability estimates to no longer reflect the true posterior probabilities in production, leading to a higher Brier score.

Answer

The distribution of input features has shifted significantly, causing the model to produce incorrect probabilities.

Answer

The model overfits to noise in the training data, leading to poor generalization.

Answer

The production data has a different class imbalance than the training data, causing the model to be biased toward the majority class.

Question 16

A company implements an ML pipeline using Vertex AI Pipelines. The pipeline trains a model using custom training jobs and then deploys it to an endpoint. The team notices that the endpoint occasionally serves an older model version for a few minutes after a new pipeline run completes. What is the most likely cause?

Accepted Answer

The endpoint is configured with a canary traffic split, and the old model is still receiving a fraction of traffic during the rollout.. D is correct because Vertex AI endpoints can be configured with a canary (gradual) traffic rollout strategy. When a new model is deployed, traffic is shifted incrementally from the old model to the new one over a specified duration. During this rollout window, the old model continues to serve a fraction of requests, which explains why users occasionally see the older model version for a few minutes after the pipeline completes.

Answer

The new model artifact is temporarily unavailable, so the endpoint falls back to the previous version.

Answer

The prediction cache is returning cached results from the old model.

Answer

The pipeline failed to update the endpoint with the new model ID.

Question 17

A team has deployed a model on Vertex AI Prediction and wants to monitor for data drift. Which TWO metrics should they use to detect drift in numerical features?

Accepted Answer

Jensen-Shannon divergence (JSD). Jensen-Shannon divergence (JSD) is a symmetric, bounded (0 to 1) measure of the difference between two probability distributions, making it ideal for detecting drift in numerical features by comparing the training distribution to the serving distribution. It is a smoothed and normalized version of Kullback-Leibler divergence, and Vertex AI Prediction's Model Monitoring natively supports JSD for numerical feature drift detection.

Answer

Pearson correlation coefficient

Answer

Chi-squared statistic

Answer

Population Stability Index (PSI)

Question 18

A company uses Vertex AI Model Monitoring to detect training-serving skew. They have a categorical feature 'product_category' with high cardinality. The monitoring job alerts for skew, but the data scientists believe the model performance is still acceptable. Which THREE actions should the team take to investigate and resolve the alert?

Accepted Answer

Examine which categories have the largest distribution changes to understand the nature of the shift.. Option A is correct because examining which categories have the largest distribution changes allows the team to pinpoint the root cause of the training-serving skew. In Vertex AI Model Monitoring, the skew alert is based on statistical distance metrics (e.g., Jensen-Shannon divergence) between training and serving distributions. By drilling down into the specific categories driving the divergence, the team can assess whether the shift is benign (e.g., seasonal) or problematic, rather than relying on aggregate model performance alone.

Answer

Remove the drifted categories from the feature set to eliminate the alert.

Answer

Ignore the alert because the model is performing well; monitoring alerts are often false positives.

Question 19

You are an ML engineer at a logistics company. You have deployed a deep learning model on Vertex AI Endpoints using a custom container with GPU acceleration. The model predicts delivery times based on route features. After one week, you notice that the endpoint's GPU utilization is consistently at 10%, but the prediction latency has increased by 50%. The number of prediction requests per second has remained stable. You check the container logs and see no errors. The model is served using TensorFlow Serving with batching enabled (batch size: 32, batch timeout: 100ms). The custom container uses a single NVIDIA T4 GPU. You have also set the Vertex AI endpoint to use autoscaling with minReplicaCount: 1 and maxReplicaCount: 5, and the CPU utilization target is 60%. Which action should you take to reduce latency?

Accepted Answer

Increase the batch size to 64 and batch timeout to 200ms to improve GPU utilization.. The core issue is low GPU utilization (10%) despite increased latency, indicating that the GPU is underutilized and the bottleneck is likely in batching or data pipeline overhead. Increasing the batch size to 64 and batch timeout to 200ms allows TensorFlow Serving to accumulate more requests per batch, improving GPU throughput and reducing per-request latency by better leveraging GPU parallelism. This directly addresses the mismatch between low GPU utilization and high latency.

Answer

Increase the minReplicaCount to 3 to handle requests in parallel.

Answer

Reduce the CPU utilization target to 40% to trigger more aggressive autoscaling.

Answer

Quantize the model to FP16 to reduce compute time per inference.

Question 20

A company deploys a custom ML model on Vertex AI to predict customer churn. The model retrains weekly, and predictions are served via a Vertex AI endpoint. After a recent retraining, the monitoring dashboard shows a sudden increase in prediction requests but a decrease in predicted churn probabilities. The model's accuracy on the validation set remains stable. What is the most likely cause of the observed behavior?

Accepted Answer

The incoming data distribution has changed, e.g., due to a new marketing campaign attracting different customers.. Option C is correct because a sudden increase in prediction requests alongside a decrease in predicted churn probabilities, while validation accuracy remains stable, indicates a shift in the incoming data distribution (covariate shift). This is typical when a new marketing campaign attracts a different customer segment that inherently has lower churn risk. The model itself hasn't degraded; it's simply seeing a different population than it was trained on, which changes the base rate of churn in the live traffic.

Answer

A training-serving skew exists between the training pipeline and the serving endpoint.

Answer

Concept drift has occurred, changing the relationship between features and churn.

Answer

Data leakage during training caused the model to overfit to historical patterns.