PMLE Monitoring ML Solutions Practice Test 3 — 15 Questions

Question 1

A media company uses a Vertex AI Model Monitoring job to detect drift on a recommendation model deployed to an endpoint. The model's output is a list of 100 item IDs. The MLOps team notices that the drift metrics for the output are not being computed, even though input feature drift is working. They have configured the monitoring job to monitor both input and output. What is the most likely reason for the missing output drift metrics?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Vertex AI Model Monitoring does not support monitoring drift for output when the output is a list or array.. Vertex AI Model Monitoring can only compute drift for scalar features, not for list or array outputs. Since the recommendation model returns a list of item IDs, the service cannot calculate distribution-based drift metrics for that output. Input feature drift works because those are scalar features. To monitor output drift, the team would need to transform the output into scalar values or monitor alternative metrics.

Answer

The monitoring job's sampling rate is too low to capture enough output values for drift analysis.

Answer

The monitoring frequency is set to daily, but output drift requires hourly monitoring.

Answer

The model's output schema was not defined in the endpoint's explanation metadata.

Question 2

A company has deployed a recommendation model to a Vertex AI Endpoint and wants to monitor the model's prediction quality over time. They have ground truth labels available with a delay of up to 48 hours. They need to compute a daily metric that compares predictions to actual outcomes. Which approach should they use?

Accepted Answer

Configure Vertex AI Model Monitoring with a custom monitoring job that ingests ground truth labels and computes the metric.. Vertex AI Model Monitoring allows custom monitoring jobs where you can define your own logic to compute metrics. Since ground truth labels are delayed, a custom job can be scheduled to run after labels arrive, join predictions with actuals, and compute metrics like accuracy or RMSE. The results can be logged to Cloud Monitoring for alerting and visualization.

Answer

Enable request-response logging and analyze the logs using BigQuery to compute the metric.

Answer

Use the Vertex AI Endpoint's built-in prediction drift detection to infer actual outcomes.

Answer

Use Vertex AI Model Monitoring's training-serving skew detection to compare prediction distributions.

Question 3

A healthcare analytics team has deployed a patient readmission risk model on a Vertex AI Endpoint. They want to monitor the model for training-serving skew and receive email alerts when skew exceeds a threshold. They have configured Model Monitoring with a training dataset and set a skew threshold. However, they are not receiving any email alerts even though skew metrics in the Vertex AI console show values above the threshold. What should they do to receive alerts?

Accepted Answer

Configure a Cloud Monitoring notification channel and link it to the Model Monitoring job.. Vertex AI Model Monitoring writes skew metrics to Cloud Monitoring but does not send alerts by itself. To receive email alerts, you must create a notification channel in Cloud Monitoring and define an alerting policy that fires when the skew metric exceeds the threshold. The Model Monitoring threshold alone does not trigger alerts. Thus, configuring Cloud Monitoring alerting is the necessary step.

Answer

Increase the skew threshold so that it is higher than the current skew values.

Answer

Enable email notifications in the Vertex AI Model Monitoring configuration page.

Answer

Set the monitoring frequency to every hour to trigger alerts more frequently.

Question 4

A company has a Vertex AI Endpoint serving predictions for an image classification model. They want to monitor the model's prediction quality in near real-time without waiting for ground truth labels, which are delayed by several days. Which approach should they use to detect potential degradation in prediction quality?

Accepted Answer

Set up Vertex AI Model Monitoring to compute prediction drift on the output probabilities.. Prediction drift monitoring in Vertex AI Model Monitoring tracks changes in the distribution of model outputs without needing ground truth. For image classification, monitoring the distribution of predicted classes or confidence scores can signal degradation. This enables near real-time detection, allowing teams to respond before delayed labels are available. It is a proactive approach to maintain model performance.

Answer

Configure the endpoint to return prediction confidence scores and set up alerts when average confidence drops below a threshold.

Answer

Implement a Cloud Function that logs all predictions and manually reviews a sample daily.

Answer

Use Vertex AI Model Monitoring to compute training-serving skew on the input images.

Question 5

A team has deployed a recommendation model on a Vertex AI Endpoint and enabled Vertex AI Model Monitoring. They now need to investigate a sudden drop in recommendation click-through rate. They want to determine whether the drop is due to changes in the input feature distribution or changes in the relationship between features and predictions. Which TWO monitoring signals should they examine? (Choose two.)

Accepted Answer

Prediction drift metrics for the model's output scores.. To diagnose a drop in click-through rate, the team should look at signals that reveal changes in input data and model outputs. Training-serving skew shows if live features differ from training, indicating input distribution changes. Prediction drift shows if the distribution of predicted scores has shifted, which can reflect altered feature-prediction relationships. Together, these signals help determine the root cause. Infrastructure and audit logs do not provide the needed statistical insights.

Answer

Endpoint CPU utilization metrics.

Answer

Vertex AI Pipelines execution history.

Answer

Cloud Audit Logs for model deployment events.

Question 6

A team operates a Vertex AI Endpoint with Vertex AI Model Monitoring enabled for feature drift. They want to reduce monitoring costs without losing the ability to detect sudden distribution shifts in production traffic. (Choose two.)

Accepted Answer

Decrease the sampling rate so that fewer prediction requests are logged and analyzed.. Monitoring cost in Vertex AI Model Monitoring is driven by how often analysis runs and how many requests are logged and scored. Reducing the monitoring frequency and lowering the sampling rate both cut cost while still allowing drift detection, though with trade-offs in alert latency and statistical confidence. The other options either increase cost or are unrelated to the goal.

Answer

Set the drift threshold to zero so that any deviation triggers an alert.

Answer

Increase the number of monitored features to improve detection sensitivity.

Answer

Disable prediction drift monitoring and keep only feature drift monitoring.

Question 7

A data scientist has deployed a model on Vertex AI Endpoints and wants to monitor the model's predictions for any drift over time. Which Vertex AI service should they use?

Accepted Answer

Vertex AI Model Monitoring. Vertex AI Model Monitoring is the purpose-built service for detecting drift in deployed models on Vertex AI Endpoints. It continuously compares incoming prediction requests against a training baseline and computes statistical drift metrics (e.g., Jensen-Shannon divergence) for features and, optionally, predictions. This is exactly the capability the data scientist needs to detect prediction drift over time.

Answer

Vertex AI Feature Store

Answer

Vertex AI Predictions

Answer

Vertex AI Explainable AI

Question 8

An MLOps engineer needs to collect ground truth labels for a deployed classification model to compare predictions against actuals. Where should the engineer store the ground truth data to enable Vertex AI model quality monitoring?

Accepted Answer

BigQuery. Vertex AI Model Monitoring for model quality requires ground truth labels to be stored in BigQuery. The monitoring job compares the model's online predictions (logged to BigQuery via prediction logging) against the actual observed labels, which must also reside in BigQuery so the service can join them on a shared key column. BigQuery is the only supported sink for ground truth data in Vertex AI model quality monitoring.

Answer

Firestore

Answer

Cloud Spanner

Answer

Cloud Storage

Question 9

An ML team wants to automatically retrain a model when data drift is detected. They have set up a Cloud Monitoring alert on drift. What service should they use to trigger a retraining pipeline in response to the alert?

Accepted Answer

Cloud Functions. Cloud Functions is the correct service to trigger a retraining pipeline in response to a Cloud Monitoring alert. Cloud Monitoring can publish alerts to a Pub/Sub topic, and a Cloud Function subscribed to that topic can invoke the Vertex AI pipeline or trigger a Cloud Scheduler job that starts retraining. This event-driven pattern is the standard way to automate retraining on drift detection.

Answer

Cloud Scheduler

Answer

Vertex AI Feature Store

Answer

Vertex AI Model Monitoring

Question 10

A company has a model serving predictions on Vertex AI Endpoints and wants to monitor for prediction drift. They enable Vertex AI Model Monitoring but also need to see a confusion matrix over time. How should they set up the confusion matrix monitoring?

Accepted Answer

Upload ground truth data to BigQuery and use Vertex AI Model Monitoring's model quality monitoring. Vertex AI Model Monitoring's model quality monitoring requires ground truth labels to compute metrics like confusion matrices, precision, recall, and F1. By uploading ground truth to BigQuery and configuring the monitoring job to join predictions with labels, Vertex AI can calculate and display confusion matrices over time. Prediction drift monitoring alone cannot produce a confusion matrix because it lacks label data.

Answer

Use Cloud Monitoring to create a custom dashboard with a confusion matrix chart

Answer

Export predictions to Cloud Storage and run a Dataflow job to compute confusion matrices

Answer

Enable Vertex AI Explainable AI and configure it to output confusion matrices

Question 11

A team is monitoring a model and observes that the error rate (prediction failures) has increased. They have enabled request/response logging on the Vertex AI Endpoint. How can they set up a metric and alert for prediction error rate?

Accepted Answer

Create a log-based metric in Cloud Logging for error logs and set up an alert in Cloud Monitoring. To monitor prediction error rate, you can create a log-based metric in Cloud Logging that counts error logs from the Vertex AI Endpoint, and then set up an alert in Cloud Monitoring based on that metric. This leverages the request/response logging already enabled.

Answer

Configure Cloud Monitoring to pull error rate from Cloud Endpoints

Answer

Use Vertex AI Model Monitoring to monitor error rate directly

Answer

Enable Vertex AI Pipelines to track errors

Question 12

A company has deployed a model for image classification and wants to monitor for feature drift using XRAI attributions. However, they notice that the XRAI attribution maps are too large and are causing high latency in the monitoring pipeline. What is the most effective way to reduce the overhead of explainability monitoring for image models?

Accepted Answer

Reduce the sampling rate for the explainability feature. Vertex AI Explainable AI supports XRAI for image models, but generating XRAI attributions can be computationally expensive. Sampling a subset of predictions reduces the number of explanations generated, lowering latency and cost.

Answer

Disable XRAI and use integrated gradients instead

Answer

Use a smaller image size for the model

Answer

Increase the number of replicas on the endpoint

Question 13

A company wants to monitor the cost of their Vertex AI prediction endpoint. They are charged per hour per replica and per request for GPU instances. Which approach should they use to track these costs?

Accepted Answer

Set up Cloud Billing budget alerts and export billing data to BigQuery for analysis. Cloud Billing budget alerts plus BigQuery billing export is the standard GCP approach for tracking and analyzing Vertex AI endpoint costs. Budget alerts notify when spend crosses thresholds, and the detailed billing export to BigQuery enables granular analysis by SKU, label, and resource — including per-replica and per-request GPU charges.

Answer

Use Vertex AI Model Monitoring to track cost metrics

Answer

Enable Cloud Monitoring dashboards for cost metrics

Answer

Use Vertex AI Pipelines to track cost per job

Question 14

An MLOps engineer is setting up monitoring for a deployed model on Vertex AI Endpoints. Which TWO actions are required to enable Vertex AI Model Monitoring for feature skew and drift? (Choose two.)

Accepted Answer

Enable request/response logging on the Vertex AI Endpoint. To enable model monitoring, you must enable request/response logging on the endpoint (to capture serving data) and create a monitoring job with the desired configuration.

Answer

Export ground truth labels to Cloud Storage

Answer

Enable Vertex AI Pipelines to run scheduled monitoring

Answer

Deploy the model with an explanation spec

Question 15

An ML team has set up automated retraining triggered by Cloud Monitoring alerts. When a feature drift alert fires, a Cloud Function publishes to Pub/Sub, which triggers a Vertex AI Pipeline. However, the retraining pipeline is failing because the training data is not updated. What is the most likely cause?

Accepted Answer

The training data in the pipeline input is stale or not refreshed. The pipeline is triggering correctly (the alert fires, the Cloud Function runs, Pub/Sub delivers, and the Vertex AI Pipeline starts), so the failure is downstream of orchestration — it is the data itself. When a drift alert fires, the pipeline's input dataset or feature table must be refreshed from the source system before training; if the pipeline references a static/stale snapshot or a BigQuery view that is not re-materialized, training runs on old data and fails validation or produces a useless model. The most likely cause is therefore that the training data in the pipeline input is stale or not refreshed.

Answer

The Cloud Function does not have permission to start the pipeline

Answer

The Pub/Sub topic is incorrectly configured

Answer

The model endpoint is overloaded