Monitoring ML Solutions

A data scientist has deployed a model on Vertex AI Endpoints and wants to monitor the model's predictions for any drift over time. Which Vertex AI service should they use?

An MLOps engineer needs to collect ground truth labels for a deployed classification model to compare predictions against actuals. Where should the engineer store the ground truth data to enable Vertex AI model quality monitoring?

A team is monitoring a deployed model and notices that the prediction distribution has changed significantly over the last week. They want to detect which features are contributing most to the drift. Which tool should they use?

An engineer wants to configure alerting when the data distribution of a serving feature deviates from the training data distribution. The model is deployed on Vertex AI Endpoints. Which divergence metric should they use to compare the training and serving distributions?

A team is monitoring a model on Vertex AI Endpoints and wants to track the p99 latency of online predictions. Which approach should they use to set up latency monitoring and alerting?

An ML team wants to automatically retrain a model when data drift is detected. They have set up a Cloud Monitoring alert on drift. What service should they use to trigger a retraining pipeline in response to the alert?

A company has a model serving predictions on Vertex AI Endpoints and wants to monitor for prediction drift. They enable Vertex AI Model Monitoring but also need to see a confusion matrix over time. How should they set up the confusion matrix monitoring?

An MLOps engineer is configuring Vertex AI Model Monitoring for a deployed model. They want to monitor for feature skew between training and serving data, but only for a subset of features. The training data has 100 features, and they want to monitor only the top 10 most important features to reduce cost and noise. How can they achieve this?

A team is monitoring a model and observes that the error rate (prediction failures) has increased. They have enabled request/response logging on the Vertex AI Endpoint. How can they set up a metric and alert for prediction error rate?

A company has deployed a model for image classification and wants to monitor for feature drift using XRAI attributions. However, they notice that the XRAI attribution maps are too large and are causing high latency in the monitoring pipeline. What is the most effective way to reduce the overhead of explainability monitoring for image models?

An organization is deploying a loan approval model and wants to monitor for fairness across demographic subgroups. They have ground truth labels stored in BigQuery. How can they use Vertex AI to evaluate performance disparities between groups?

A company wants to monitor the cost of their Vertex AI prediction endpoint. They are charged per hour per replica and per request for GPU instances. Which approach should they use to track these costs?

An MLOps engineer is setting up monitoring for a deployed model on Vertex AI Endpoints. Which TWO actions are required to enable Vertex AI Model Monitoring for feature skew and drift? (Choose two.)

A team is using Vertex AI Model Monitoring and wants to set up automated retraining when drift is detected. Which THREE services are needed to implement this pipeline? (Choose three.)

An ML engineer wants to monitor a deployed model for fairness across different age groups and genders. Which TWO Vertex AI services should they use together to achieve this? (Choose two.)

An ML engineer has deployed a model on Vertex AI Endpoints and wants to detect when the serving data distribution differs from the training data distribution. Which monitoring feature should they enable?

A data scientist notices that the model's prediction latency has increased over the last week. They need to investigate the root cause by examining request and response logs for the Vertex AI Endpoint. What is the recommended way to capture these logs?

A company has deployed a model to Vertex AI Endpoints and wants to monitor for feature drift using Jensen-Shannon divergence. They have set a threshold of 0.1. After one week, the monitoring job reports a divergence of 0.15 for a feature. What should the engineer do next to diagnose which features are contributing to the drift?

An ML team has set up automated retraining triggered by Cloud Monitoring alerts. When a feature drift alert fires, a Cloud Function publishes to Pub/Sub, which triggers a Vertex AI Pipeline. However, the retraining pipeline is failing because the training data is not updated. What is the most likely cause?

A machine learning engineer wants to monitor the fairness of a credit approval model across demographic subgroups. They have ground truth labels in BigQuery. Which approach should they use to evaluate performance disparities?

An ML engineer needs to monitor the online prediction latency of a Vertex AI Endpoint. Which metrics should they look at in Cloud Monitoring?

A company wants to be alerted when the prediction error rate on their Vertex AI Endpoint exceeds 5% in any 5-minute window. What is the best way to set up this alert?

A model deployed on a Vertex AI Endpoint uses an image model with XRAI explainability. The team notices that the prediction distributions are shifting over time. They want to monitor prediction drift. However, the explainability feature is not enabled. What must the engineer do to enable monitoring prediction drift?

An ML engineer needs to track the costs incurred by Vertex AI prediction endpoints. Which tool should they use to set budget alerts and monitor spending?

A data scientist has deployed a model with Vertex AI Endpoints and enabled request/response logging to BigQuery. They want to compute a confusion matrix over time to monitor model quality. What should they do?

An ML team is using Population Stability Index (PSI) to monitor feature drift on a Vertex AI Endpoint. The PSI value for a feature is 0.25, which exceeds the alert threshold of 0.2. The feature has high SHAP importance. The team wants to automatically retrain the model. What is the correct end-to-end setup?

An engineer is configuring Vertex AI Model Monitoring for a model deployed on an endpoint. They want to monitor feature skew using the training dataset as a baseline. The training dataset is large (10 TB). What is the most efficient way to provide the baseline distribution?

An ML engineer needs to set up automated retraining triggered by data drift. They have decided to use Cloud Monitoring alerts to detect drift. Which TWO additional services are required to complete the retraining pipeline? (Choose 2)

A data science team wants to monitor model quality by comparing predictions against ground truth labels. They have deployed a model on Vertex AI Endpoints and enable request/response logging to BigQuery. Which THREE actions should they take to set up model quality monitoring? (Choose 3)

An ML engineer wants to monitor the performance of a Vertex AI Endpoint. Which TWO metrics are available in Cloud Monitoring for Vertex AI Endpoints? (Choose 2)

A data scientist has deployed a classification model on a Vertex AI Endpoint and wants to monitor for feature drift in the serving data compared to the training data. Which Vertex AI service should be used?

An ML engineer has set up Vertex AI Model Monitoring on an endpoint with a sampling rate of 0.1 (10%). They notice that the monitoring job runs hourly but the reported drift metrics seem inconsistent. What is the most likely cause?

A team wants to collect ground truth labels for their model deployed on Vertex AI Endpoint to perform model quality monitoring. They have a process that generates actual outcomes within 24 hours of prediction. What is the recommended approach for storing these labels?

A financial services company deploys a model on Vertex AI Endpoints with GPU acceleration. They notice that the p99 latency for predictions has increased from 200ms to 1.2s over the past week. CPU utilisation is low, but GPU utilisation is high. Which action should they take to reduce latency?

An ML engineer needs to monitor the error rate of prediction jobs on a Vertex AI Endpoint. Where can they view the number of failed prediction requests over time?

A company wants to implement a retraining trigger for their ML model. They have set up Cloud Monitoring alerts that fire when drift exceeds a threshold. What should be the target of the alert to automatically start a Vertex AI Pipeline for retraining?

A team uses Vertex AI Explainable AI with integrated gradients for a deep learning model. They want to reduce the computational cost of explanations without significantly reducing explanation quality. Which configuration change should they make?

An organisation wants to monitor fairness of their loan approval model across demographic subgroups. They have predictions stored in BigQuery along with ground truth. Which GCP service can evaluate model performance for each subgroup and identify disparities?

A company wants to log all prediction requests and responses from a Vertex AI Endpoint to BigQuery for auditing and debugging. How can they achieve this?

An ML engineer is monitoring a model on Vertex AI Endpoint and sees that feature 'age' has a training distribution of (mean=45, std=10) but the serving distribution over the last hour shows (mean=30, std=15). JS divergence is 0.12, but the alert threshold is 0.1. The engineer suspects this is due to a temporary campaign targeting younger users. What should they do first?

A team wants to monitor prediction drift on a Vertex AI Endpoint for a classification model. They have configured Vertex AI Model Monitoring with default settings. Which metric will be used to detect prediction drift?

A company wants to track the cost of their Vertex AI prediction endpoint. They use a custom machine type with 1 n1-standard-4 (4 vCPU, 15 GB memory) and 1 NVIDIA T4 GPU. The endpoint is configured for automatic scaling with min=1, max=5 replicas. Which cost monitoring approach should they use?

A data science team is configuring Vertex AI Model Monitoring for a deployed model. They want to detect both feature skew and feature drift. Which TWO configurations must they set?

An ML engineer is troubleshooting why a Vertex AI Endpoint is returning high prediction latency. They have enabled request/response logging and see that some requests take >1 second while most are fast. Which THREE actions should they take to diagnose the issue?

A company wants to automatically retrain their model when data drift is detected. Which THREE components are needed to implement this pipeline?

An MLOps team has deployed a model on Vertex AI Endpoints and wants to monitor for skew between training and serving data distributions. Which Vertex AI service should they use?

A data scientist notices that the prediction distribution of a deployed model has changed significantly over the past week. They want to identify which features are contributing most to the drift. Which approach should they use?

A company deploys a model on Vertex AI Endpoints and configures Vertex AI Model Monitoring with a sampling rate of 0.1 and monitoring frequency of every hour. They notice that the monitoring alert fires only after several hours of drift. What is the most likely cause?

An ML team wants to automatically retrain a model when prediction drift is detected on the deployed endpoint. They have Vertex AI Model Monitoring configured to send alerts to Cloud Monitoring. Which minimal set of additional services should they use to trigger a retraining pipeline?

Which algorithm does Vertex AI Model Monitoring use by default to detect feature drift in a categorical feature?

An ML engineer wants to monitor the latency of online predictions from a Vertex AI Endpoint. They need to track p50, p95, and p99 latency over time and set up alerts if p99 exceeds 1 second. Which approach should they take?

A team is using Vertex AI Explainability with a deployed model. They need to generate explanations for image classification predictions. Which explanation method should they configure in the ExplanationSpec?

A company wants to monitor fairness of a model by evaluating performance metrics across demographic subgroups. They have ground truth labels stored in BigQuery. Which Vertex AI service should they use?

An ML engineer is configuring Vertex AI Model Monitoring for drift detection on a deployed endpoint. Which TWO settings directly affect the frequency and accuracy of drift detection? (Choose 2)

An organization wants to collect ground truth labels for model quality monitoring and store them in BigQuery. They also want to compute and visualize a confusion matrix over time. Which TWO actions should they take? (Choose 2)

A company is experiencing high prediction costs on Vertex AI Endpoints. They want to monitor and optimize costs. Which THREE actions should they take? (Choose 3)

An engineer wants to set up request/response logging for a Vertex AI Endpoint to analyze prediction behavior. Which TWO resources must be configured? (Choose 2)

A team uses Vertex AI Pipelines for continuous training triggered by model drift. They want to monitor the pipeline execution cost and optimize resource usage. Which THREE metrics should they track? (Choose 3)

An ML engineer is monitoring a Vertex AI Endpoint and notices a spike in 5xx error rates. Which TWO metrics should they examine to diagnose the issue? (Choose 2)