PMLE Monitoring ML Solutions Practice Test 6 — 15 Questions

Question 1

You are an MLOps engineer at a retail company. Your team has deployed a demand forecasting model to a Vertex AI Endpoint. The model uses 12 numerical features and outputs a single numeric value representing predicted units sold. You have configured Vertex AI Model Monitoring with a training dataset that includes the full feature schema and prediction distribution. After a week, you observe that the feature 'promotion_flag' has a Jensen-Shannon divergence of 0.15, while all other features remain below 0.05. The model's prediction distribution has also shifted. Which action should you take first to diagnose the cause of the drift?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Examine the distribution of the 'promotion_flag' feature in the recent serving data and compare it with the training distribution to determine if the feature's meaning or data collection process has changed.. The feature 'promotion_flag' has the highest drift, so examining its recent distribution compared to training helps identify if the feature's meaning or data collection has changed. This diagnostic step is crucial before retraining or adjusting monitoring, as it reveals whether the drift is due to a data pipeline issue or a genuine change in the underlying pattern.

Answer

Increase the monitoring frequency to 15 minutes and lower the drift threshold for 'promotion_flag' to 0.05 to get more granular alerts.

Answer

Ignore the drift because a Jensen-Shannon divergence of 0.15 is below the typical threshold of 0.2 used for numerical features.

Answer

Immediately trigger a retraining pipeline using the most recent data to update the model and reduce the drift.

Question 2

An ML engineer is monitoring a deployed model on Vertex AI Endpoints and wants to detect anomalies in the distribution of a categorical feature 'product_category' which has 50 possible values. The engineer configures Vertex AI Model Monitoring with training-serving skew detection. After a week, they receive an alert that the feature is skewed. Upon investigation, they find that a new product category was introduced in the serving data that was not present in the training data. What is the most likely reason for the alert?

Accepted Answer

The monitoring system treats the new category as an unknown value and flags it as skew.. Vertex AI Model Monitoring's skew detection compares the distribution of feature values between training and serving. When a new category appears in serving that was absent in training, the distribution diverges, and the system flags it as skew. This is because the model has never seen that category and may not handle it well, so monitoring alerts the team to investigate.

Answer

The model's predictions are incorrect for the new category, causing the monitoring system to flag the feature.

Answer

The training data already contained the new category but it was not properly encoded.

Answer

The monitoring configuration has a bug that misinterprets the new category as a numerical value.

Question 3

A credit-risk team runs a tabular model on a Vertex AI Endpoint. They configured Vertex AI Model Monitoring with a training dataset and skew detection using the default threshold. After a week, they receive alerts that many features have high training-serving skew, but the model's business metrics (approval rate, default rate) are unchanged. They suspect the alerts are false positives due to a recent change in an upstream data pipeline that shifted feature distributions. What should they do to reduce these false alerts while still monitoring for real skew?

Accepted Answer

Recreate the monitoring job with a new training dataset that reflects the recent pipeline change, and adjust the skew threshold based on observed variance.. Training-serving skew compares live data to the training baseline. When an upstream pipeline changes distributions legitimately, the old baseline becomes invalid and triggers false skew alerts. Updating the monitoring configuration with a representative training dataset and a threshold based on observed variance realigns detection with current production behavior. Disabling monitoring or making it more sensitive does not solve the underlying baseline mismatch.

Answer

Disable skew detection and rely only on prediction drift monitoring.

Answer

Switch from skew detection to outlier detection and set the outlier threshold to a very low value.

Answer

Increase the monitoring frequency and lower the skew threshold to capture more data points.

Question 4

An ML engineer is using Vertex AI Model Monitoring on a deployed model that predicts customer churn. The model's input features include a mix of numerical and categorical features. The engineer wants to detect changes in the distribution of a specific categorical feature, 'contract_type', which has 20 possible values. Which approach should be used to effectively monitor drift for this feature?

Accepted Answer

Configure drift detection specifically for 'contract_type' and set an appropriate threshold based on its cardinality.. Vertex AI Model Monitoring allows per-feature drift detection with customizable thresholds. For categorical features with many categories, the default threshold may not be optimal; tuning the threshold based on cardinality helps balance sensitivity and false positives. By explicitly monitoring 'contract_type', the engineer can detect shifts in its distribution, which is crucial for model performance if that feature is important.

Answer

Use Vertex AI Explainable AI to compute feature attributions and monitor changes in attributions for 'contract_type'.

Answer

Convert 'contract_type' into a numerical feature using one-hot encoding and then monitor it as a numerical feature.

Answer

Enable drift detection for all features and use the default threshold.

Question 5

Your team has deployed a tabular model to a Vertex AI Endpoint and enabled Vertex AI Model Monitoring with training-serving skew detection. You configured the monitoring job to run hourly and store statistics in a Cloud Storage bucket. After the first run, you notice that the job did not produce any drift metrics. You want to determine the cause of the missing metrics. What should you do first?

Accepted Answer

Check that the endpoint's deployed model has a reference dataset specified and that the monitoring job's input schema matches the endpoint's prediction schema.. Vertex AI Model Monitoring requires a reference dataset and a matching schema to compute training-serving skew. Without these, the job cannot generate metrics. Checking the configuration of the reference dataset and schema alignment is the logical first step to resolve missing metrics.

Answer

Increase the monitoring job's sampling rate to 1.0 to ensure that all prediction requests are analyzed.

Answer

Re-deploy the model to the endpoint with a larger machine type to handle the monitoring workload.

Answer

Verify that the Cloud Storage bucket has the correct IAM permissions for the Vertex AI service account to write objects.

Question 6

A financial institution has deployed a fraud detection model on a Vertex AI Endpoint. The model uses both numerical and categorical features. They have enabled Vertex AI Model Monitoring with training data and configured drift thresholds. After several weeks, they notice that the model's recall has dropped significantly, but the overall prediction distribution remains stable. They suspect that a specific subgroup of transactions is being misclassified. Which approach should they use to identify the subgroup and diagnose the issue?

Accepted Answer

Export the serving data and predictions to BigQuery, then use slice-based evaluation to compare performance metrics across different subgroups.. Exporting serving data and predictions to BigQuery enables slice-based evaluation, which can reveal performance disparities across subgroups. Since Vertex AI Model Monitoring does not provide subgroup-level metrics, this manual analysis is necessary to identify the problematic subgroup and diagnose the recall drop.

Answer

Use Vertex AI Model Monitoring's feature attribution analysis to identify which features contribute most to the misclassifications.

Answer

Configure Vertex AI Model Monitoring to monitor the 'transaction_amount' feature with a lower threshold to detect subtle drifts that might affect recall.

Answer

Retrain the model immediately using the most recent data to improve recall across all subgroups.

Question 7

A team is using Vertex AI Model Monitoring to detect prediction drift on a deployed model. They have configured the monitoring job to run every 6 hours. After a week, they notice that the drift metric has been consistently high but no alerts have been triggered. They have verified that the alerting policy in Cloud Monitoring is correctly configured. What is the most likely cause?

Accepted Answer

The drift threshold is set higher than the observed drift metric values.. Alerts are triggered when the drift metric exceeds the configured threshold. If the threshold is set higher than the observed drift values, no alert will fire. The team should review and lower the threshold to match their tolerance for drift, ensuring alerts are generated when drift is significant.

Answer

The sampling rate is too low, causing the drift metric to be inaccurate.

Answer

The monitoring job is not writing the drift metrics to Cloud Monitoring.

Answer

The monitoring frequency is too high, causing the system to ignore the drift.

Question 8

An ML engineer manages a Vertex AI Endpoint serving a fraud detection model. Compliance requires that every prediction be logged with its input features for audit, but the team also wants to minimize storage costs. They decide to enable request-response logging on the endpoint. Which configuration should they use to meet both requirements?

Accepted Answer

Enable request-response logging with a sampling rate of 1.0 and set the logging destination to a Cloud Storage bucket with a lifecycle policy to delete objects after 30 days.. Compliance requires that every prediction be logged with its input features, so the sampling rate must be 1.0. To minimize storage costs, logs should be stored in a cost-effective location such as Cloud Storage, with a lifecycle policy to delete after the required retention period. Sampling below 1.0 or using higher-cost logging services would either miss predictions or increase expenses, failing one of the stated requirements.

Answer

Enable request-response logging with a sampling rate of 0.1 and send logs to Cloud Logging.

Answer

Enable request-response logging with a sampling rate of 1.0 and send logs to Cloud Logging with a default retention of 30 days.

Answer

Enable request-response logging with a sampling rate of 0.5 and configure a BigQuery sink for analysis.

Question 9

A logistics company has a Vertex AI Model Monitoring job that detects feature drift on a deployed route optimization model. The model uses 50 features, and the monitoring job is configured to monitor all features. The MLOps team notices that the monitoring job is incurring high costs and taking a long time to complete. They want to reduce monitoring overhead while still detecting significant drift on the most important features. What should they do?

Accepted Answer

Select a subset of features to monitor based on feature importance or business relevance.. Monitoring all 50 features increases cost and runtime. To reduce overhead while still detecting significant drift, the team should select a subset of features to monitor based on importance or business relevance. This reduces the computational load per run. Reducing frequency lowers total runs but not per-run cost. Increasing sampling rate would increase cost. Disabling numerical features arbitrarily could miss critical drift.

Answer

Disable drift detection for numerical features and only monitor categorical features.

Answer

Increase the sampling rate to reduce the volume of prediction requests analyzed.

Answer

Reduce the monitoring frequency from daily to weekly.

Question 10

You are using Vertex AI Model Monitoring to detect prediction drift on a deployed model that serves online predictions. You want to ensure that the monitoring job can correctly compute drift metrics for numerical features. Which two configurations are required for the monitoring job to compute drift for a numerical feature? (Choose two.)

Accepted Answer

Specify the feature's distribution as a numerical distribution and provide the training dataset as the reference.. To compute drift for numerical features, Vertex AI Model Monitoring needs a reference dataset to establish the baseline distribution and a correct input schema to parse the feature as numerical. These two configurations are essential; sampling rate, explainability, and thresholds are not required for the computation itself.

Answer

Enable explainability on the endpoint to generate feature attributions for drift analysis.

Answer

Configure the monitoring job to use a default threshold for drift detection, such as 0.3 for the Jensen-Shannon distance.

Answer

Set the monitoring job's sampling rate to at least 0.5 to ensure sufficient data for statistical tests.

Question 11

A healthcare company has deployed a diagnostic model on Vertex AI Endpoints. They use Vertex AI Model Monitoring to detect drift in features and predictions. The model's input features include patient age, which is a numerical feature. The monitoring job has been running for a month and has generated several alerts for age drift. However, the model's performance has not degraded. The team wants to reduce false alerts without missing critical drifts. What should they do?

Accepted Answer

Analyze the distribution of age in the serving data over time to determine if the drift is due to expected seasonal variation or a data pipeline issue, then adjust the monitoring configuration accordingly.. Analyzing the age distribution over time helps determine if the drift is due to expected variability or a data issue. This allows the team to adjust monitoring configuration, such as setting a more appropriate threshold or excluding the feature if it's not performance-critical, thereby reducing false alerts without missing important drifts.

Answer

Switch from Vertex AI Model Monitoring to a custom monitoring solution using Cloud Monitoring to have more control over alert thresholds.

Answer

Disable drift monitoring for the age feature entirely, since it is not causing performance degradation.

Answer

Increase the drift threshold for the age feature to reduce sensitivity, and monitor other features more closely.

Question 12

A company has deployed a model to a Vertex AI Endpoint and wants to receive an alert when the model's prediction latency exceeds a threshold. They have configured Cloud Monitoring to track the endpoint's latency metrics. They now need to create a notification channel to send alerts to their on-call team. Which Cloud Monitoring resource should they use to define the condition that triggers the alert?

Accepted Answer

An alerting policy with a condition based on the endpoint's latency metric.. Cloud Monitoring alerting policies are used to define conditions that trigger alerts. To alert on prediction latency, you create an alerting policy with a condition based on the endpoint's latency metric and attach a notification channel to it. Dashboards and notification channels do not define conditions.

Answer

A notification channel that sends emails to the on-call team.

Answer

A dashboard that displays the endpoint's latency over time.

Answer

A log-based metric that counts the number of high-latency requests.

Question 13

A media streaming company uses a recommendation model deployed on Vertex AI Endpoints. The model predicts whether a user will click on a recommended item. They have set up Vertex AI Model Monitoring with a training dataset and configured drift detection for both features and predictions. Recently, they observed that the prediction drift metric (Jensen-Shannon divergence) for the 'click' prediction has exceeded the threshold, but feature drifts are within normal ranges. What is the most likely cause of this prediction drift?

Accepted Answer

A change in the relationship between features and the target variable (concept drift) that is not captured by feature drift alone.. Prediction drift with stable feature distributions indicates concept drift, where the relationship between features and the target has changed. This is common in dynamic environments like media streaming, where user preferences evolve. Detecting this early allows for timely model updates.

Answer

A bug in the Vertex AI Model Monitoring configuration that incorrectly computes the prediction drift metric.

Answer

An increase in the overall volume of prediction requests, leading to a higher variance in the prediction distribution.

Answer

A data pipeline error that has corrupted the feature values, causing the model to produce different predictions.

Question 14

An ML engineer is configuring Vertex AI Model Monitoring for a deployed model that receives a mix of numerical and categorical features. The engineer wants to ensure that the monitoring job can detect both data drift and training-serving skew. Which two configurations are required to enable both types of detection? (Choose two.)

Accepted Answer

Enable skew detection and specify the training dataset used to train the model.. To detect both data drift and training-serving skew in Vertex AI Model Monitoring, you must enable each detection type and provide the appropriate baseline. Skew detection requires the training dataset to compare against serving data. Drift detection requires a baseline dataset, often a sample of recent serving data. These two configurations are independent and both necessary to achieve the desired monitoring.

Answer

Set the monitoring objective to 'drift' and provide a training dataset for baseline.

Answer

Set the monitoring frequency to at least once per hour to capture both types of drift.

Answer

Configure alerting via Cloud Monitoring to notify when either drift or skew exceeds thresholds.

Question 15

You have deployed a model to a Vertex AI Endpoint and enabled Vertex AI Model Monitoring with a monitoring frequency of every 24 hours. The model serves predictions with a feature called 'transaction_amount' that has a skewed distribution. After several days, you notice that the drift metrics for this feature are consistently below the threshold, but you suspect that the feature distribution has actually shifted. You want to improve the sensitivity of drift detection for this feature. What should you do?

Accepted Answer

Adjust the drift threshold for 'transaction_amount' to a lower value to trigger alerts more easily.. To increase sensitivity to drift for a specific feature, you can lower the drift threshold for that feature. This makes the monitoring job trigger alerts at smaller deviations from the reference distribution. Changing frequency or sampling rate affects data volume or timeliness, not the sensitivity of the drift metric itself.

Answer

Enable explainable AI on the endpoint to get feature attributions for 'transaction_amount'.

Answer

Change the monitoring frequency to every hour to capture more data points.

Answer

Increase the sampling rate to analyze a larger proportion of prediction requests.