PMLE Monitoring ML solutions Practice Test 4 — 15 Questions

Question 1

You have deployed a text classification model using Vertex AI Endpoints. The model is performing well, but the operations team wants to be alerted if the endpoint returns an excessive number of HTTP 503 errors. What is the simplest way to achieve this?

Accepted Answer

Create a Cloud Monitoring alert based on the metric 'prediction/failed_request_count' with a condition on 5xx errors.. Option B is correct because Vertex AI Endpoints automatically export the 'prediction/failed_request_count' metric to Cloud Monitoring, which includes a label for HTTP status codes. By creating an alert on this metric with a filter for 5xx errors, you can directly monitor excessive 503 responses without additional infrastructure or custom code.

Answer

Configure a Cloud Monitoring uptime check on the endpoint URL.

Answer

Add a logging statement in the custom prediction routine to count errors manually.

Answer

Export Cloud Logging to BigQuery and run a scheduled query for 503s.

Question 2

A recommendation system model is updated daily via a retraining pipeline. After each update, the online prediction latency increases significantly for about 30 minutes before returning to normal. What is the most likely cause and solution?

Accepted Answer

The new model version causes cold start in the serving infrastructure; pre-warm the model by sending a dummy request after deployment.. Option A is correct because the cold start due to model version change causes initial slow inference while caches warm up, and pre-warming with traffic can mitigate. Option B is wrong because GKE is not directly involved. Option C is wrong because GPU switching is not needed. Option D is wrong because the issue is not resource contention.

Answer

The Vertex AI endpoint autoscaling policy is too aggressive, causing scale-down during retraining.

Answer

The retraining pipeline runs on a GKE cluster that shares resources with the serving endpoint.

Answer

The model is being switched from CPU to GPU at deployment.

Question 3

Your team manages multiple ML models on Vertex AI. You need to implement a centralized monitoring solution to track model performance over time. Which TWO approaches should you consider? (Choose two.)

Accepted Answer

Create Cloud Monitoring dashboards and alerts based on Vertex AI metrics.. Option C is correct because Cloud Monitoring provides centralized dashboards and alerting for Vertex AI metrics such as prediction latency, request count, and error rates, enabling you to track model performance over time without additional infrastructure. Option D is correct because Vertex AI Model Monitoring is purpose-built to detect training-serving skew and feature drift by comparing serving data distributions to training data, which is essential for maintaining model performance in production.

Answer

Store all prediction logs in BigQuery and analyze using SQL.

Answer

Use Cloud Source Repositories to track model code versions.

Answer

Enable Cloud Billing budgets to track cost per model.

Question 4

You are monitoring a production model that is experiencing gradual decay in AUC. Which THREE metrics should you set up alerts for to diagnose the root cause? (Choose three.)

Accepted Answer

Concept drift score measured by comparing predicted vs actual outcomes.. Option A is correct because concept drift directly measures the degradation of model performance by comparing predicted probabilities against actual outcomes over time. A gradual AUC decay indicates that the relationship between features and the target is shifting, and tracking concept drift via metrics like the PSI or distribution of residuals helps isolate whether the model's predictive power is eroding due to changing data patterns.

Answer

Average prediction latency over the past hour.

Answer

Model staleness (days since last retraining).

Question 5

Your team deploys a model using Vertex AI Endpoints with autoscaling. Which TWO metrics are most important to monitor in order to optimize cost and performance? (Choose two.)

Accepted Answer

Number of requests per minute.. Option B is correct because the number of requests per minute directly drives autoscaling behavior in Vertex AI Endpoints. Monitoring this metric allows you to right-size the number of serving nodes to match traffic patterns, avoiding over-provisioning (cost) or under-provisioning (performance). Option C is correct because CPU utilization of the serving containers indicates whether the model is compute-bound or idle; high CPU suggests the need for more nodes, while low CPU suggests over-provisioning, directly impacting both cost and latency.

Answer

Number of active nodes in the endpoint.

Answer

Error rate (HTTP 4xx/5xx).

Answer

P99 prediction latency.

Question 6

A data science team has deployed a model on Vertex AI and wants to automatically detect when the distribution of a specific feature shifts significantly from the training data. Which service should they use?

Accepted Answer

Vertex AI Model Monitoring. Vertex AI Model Monitoring is the correct service because it is specifically designed to detect feature distribution drift (skew) between training and serving data for deployed models. It continuously monitors the input features and alerts when statistical metrics like the Jensen-Shannon divergence or the L-infinity distance exceed a configured threshold, enabling proactive model retraining.

Answer

Cloud Data Loss Prevention

Answer

Vertex AI Explainable AI

Answer

Cloud Composer

Question 7

A machine learning engineer notices that the online prediction latency for a custom TensorFlow model deployed on Vertex AI has increased significantly over the past week. Cloud Monitoring shows that the CPU utilization of the endpoints remains below 40%, but the number of concurrent requests has doubled. What is the most likely cause of the latency increase?

Accepted Answer

Insufficient number of replicas for autoscaling. Option C is correct because the CPU utilization remains below 40% while concurrent requests have doubled, indicating that the existing replicas are not saturated on CPU but are bottlenecked by request queuing or thread contention. Vertex AI autoscaling scales based on CPU utilization by default; if the threshold is not crossed, new replicas are not provisioned, causing requests to queue and latency to spike. The engineer should verify the autoscaling configuration and consider scaling on request count or reducing the CPU utilization target.

Answer

Data skew causing longer inference time

Answer

Memory leak in the serving container

Answer

Model overfitting

Question 8

A large enterprise has multiple ML models deployed in production across different regions. They want to implement a centralized monitoring dashboard that tracks key performance indicators such as prediction accuracy, latency, and error rates for all models, with the ability to drill down into individual model versions. Which approach best meets these requirements?

Accepted Answer

Use Cloud Monitoring with custom metrics reported by each model deployment, and create a unified dashboard with filterable resources. Option D is correct because Cloud Monitoring with custom metrics allows each model deployment to report key performance indicators (e.g., prediction accuracy, latency, error rates) as metric time series. These custom metrics can be aggregated into a single unified dashboard, and the dashboard can be configured with filterable resources (e.g., region, model version) to enable drill-down into individual model versions. This approach provides centralized, real-time monitoring without relying on log-based or batch analytics.

Answer

Use Vertex AI Experiments to log metrics and compare across runs

Answer

Use Cloud Logging to search logs from each model and create a dashboard

Answer

Use BigQuery to store prediction logs and then visualize in Looker

Question 9

An ML team is using Vertex AI Pipelines to run automated retraining workflows. They want to monitor pipeline execution and receive alerts when a pipeline run fails. Which Google Cloud service should they use to set up such alerts?

Accepted Answer

Cloud Monitoring. Cloud Monitoring (formerly Stackdriver Monitoring) is the correct service because it provides alerting policies that can be triggered based on pipeline run status metrics, such as failure counts or run state changes. Vertex AI Pipelines automatically exports execution metrics to Cloud Monitoring, allowing you to define conditions (e.g., metric 'pipeline/run_count' with filter 'status=FAILED') and configure notifications via channels like email, Pub/Sub, or PagerDuty.

Answer

Vertex AI Metadata

Answer

Cloud Logging

Answer

Cloud Scheduler

Question 10

A company has deployed a model that predicts customer churn. The model's performance, as measured by AUC, has been declining over the past month. The team suspects data drift. They have enabled Vertex AI Model Monitoring, but no alerts have been triggered. What is a possible reason for the lack of alerts?

Accepted Answer

The monitoring is only sampling 10% of the serving data. If the sampling rate is low (e.g., 10% of serving data), Model Monitoring may not capture enough data to detect drift, leading to no alerts even if drift exists. A low threshold would create more alerts, not fewer. Daily retraining might correct drift, but would still likely trigger alerts if drift occurred between retraining runs. Restricting to categorical features only would miss continuous feature drift, but that would still trigger alerts for categorical features.

Answer

The drift detection threshold is set too low

Answer

The model is being retrained daily

Answer

The drift detection focuses on categorical features only

Question 11

A team is monitoring a production ML system that includes multiple models and data processing pipelines. They want to set up a comprehensive alerting strategy that minimizes false positives while ensuring critical issues are promptly addressed. Which approach is the most effective?

Accepted Answer

Use AIOps with anomaly detection to dynamically adjust thresholds. Option D is correct because AIOps with anomaly detection uses machine learning to dynamically adjust alert thresholds based on real-time system behavior, reducing false positives while ensuring critical issues are detected promptly. This approach adapts to changing data distributions and traffic patterns, unlike static thresholds that require manual tuning and often miss subtle anomalies. It is the most effective strategy for complex ML production systems where multiple models and pipelines interact, as it can correlate signals across components to identify genuine incidents.

Answer

Set up alerts for all possible error conditions

Answer

Use static thresholds based on historical data

Answer

Rely on manual monitoring during business hours

Question 12

A machine learning model deployed on Vertex AI is returning erroneous predictions. The team needs to investigate the root cause by examining the prediction request and response details. Which Google Cloud tool is best suited for this?

Accepted Answer

Cloud Logging. Cloud Logging is the correct tool because it captures detailed logs of prediction requests and responses, including input features, model outputs, and any errors. By examining these logs, the team can trace the exact data flow and identify discrepancies causing erroneous predictions, such as data preprocessing issues or model version mismatches.

Answer

Cloud Monitoring

Answer

Cloud Debugger

Answer

Cloud Trace

Question 13

A team is using Vertex AI Feature Store to manage features for training and serving. They want to monitor the freshness of the features (i.e., how recently each feature was updated). Which approach should they take?

Accepted Answer

Create a custom Cloud Monitoring metric based on feature ingestion timestamps. Vertex AI Feature Store does not provide a built-in monitoring dashboard for feature freshness. To track how recently each feature was updated, you must create a custom Cloud Monitoring metric based on feature ingestion timestamps, which allows you to define alerting thresholds and visualize freshness over time.

Answer

Use Cloud Logging to track feature updates

Answer

Use Vertex AI Feature Store's monitoring dashboard

Answer

Use Cloud Audit Logs to monitor API calls

Question 14

A company has deployed a machine learning model that uses a large input tensor. They notice that the prediction latency varies significantly between requests of the same size. Cloud Monitoring shows that the serving endpoint's CPU utilization is consistently below 50%, but memory utilization fluctuates between 70% and 95%. What is the most likely cause?

Accepted Answer

The model is performing garbage collection cycles. The correct answer is A because the described symptoms—low CPU utilization (below 50%) and high, fluctuating memory utilization (70%–95%) with variable latency—are classic indicators of garbage collection (GC) pauses in a managed runtime like Python or Java. When the model processes large input tensors, it allocates significant memory; as memory pressure builds, the garbage collector runs more frequently, causing stop-the-world pauses that increase latency unpredictably, even though CPU is not fully utilized.

Answer

The model is using excessive memory due to a memory leak

Answer

The prediction latency is being affected by CPU throttling

Answer

The model is hitting a cold start due to autoscaling

Question 15

A team is deploying a new model version. They want to ensure that they can quickly roll back if the new version performs poorly in production. Which TWO actions should they take? (Choose 2.)

Accepted Answer

Use traffic splitting to gradually shift traffic. Option C is correct because traffic splitting allows you to gradually shift a percentage of inference requests from the old model version to the new one. If the new version performs poorly, you can immediately revert the split to 0% for the new version, providing a fast and controlled rollback without redeploying or disrupting service.

Answer

Keep the old model version deployed alongside the new one

Answer

Configure Vertex AI Model Monitoring to compare predictions

Answer

Set up Cloud Monitoring alerts on model performance