CCNA Monitoring ML Solutions Questions

Vertex AI Model Monitoring continuously evaluates deployed endpoint predictions against a training baseline, detecting training-serving skew and prediction drift. It satisfies the requirement to monitor predictions over time, unlike feature-level logging or scheduled batch jobs, which capture data but perform no drift computation.

Cloud Monitoring collects and visualizes metrics from Google Cloud services, including Vertex AI Endpoints. It automatically ingests metrics such as prediction request count and latency. You can create dashboards and alerts based on these metrics. This is the standard service for monitoring operational metrics of deployed models, providing near real-time visibility into endpoint performance.

Vertex AI Model Monitoring detects training-serving skew and drift by comparing incoming prediction request distributions against a baseline schema and statistics. It satisfies the requirement to monitor distributional skew on deployed Endpoints without custom tooling.

Error count is exported automatically to Cloud Monitoring for every Vertex AI Endpoint, aggregated per deployed model and response code. It satisfies the stem's monitoring requirement by surfacing failed prediction requests, letting the engineer alert on serving errors without configuring custom logging or additional instrumentation.

Vertex AI Model Monitoring only analyzes features explicitly listed in the monitoring configuration. If average session duration was omitted from the monitored feature list, the service will not compute drift for it regardless of the baseline or frequency. This is the most likely reason no alerts fired despite an obvious shift, because unlisted features are silently ignored.

Confusion matrices need both predictions and ground truth labels. Vertex AI request/response logging writes predictions to BigQuery, so joining uploaded ground truth labels there and computing the matrix in a scheduled query satisfies the stem's monitoring requirement.

Cloud Billing budget alerts notify on spend thresholds, and exporting cost data to BigQuery enables granular analysis by label, including the custom machine type and GPU replicas. This satisfies the stem's requirement to track prediction endpoint costs with autoscaling replicas.

Vertex AI Model Monitoring computes drift by comparing live feature distributions against a baseline derived from a training dataset or a saved baseline. Pointing the monitor at the original training data fixes the reference distribution to the model's training period, so drift alerts reflect deviation from that stable baseline rather than from rolling production windows.

The pipeline executes correctly but trains on an unchanged dataset, so drift persists. The alert and Pub/Sub trigger fire as designed; the failure lies in the input source, which is not being refreshed from the updated feature store or data location before the pipeline runs.

Training-serving skew compares live serving inputs to the statistics of the training dataset. If the upstream pipeline legitimately changed feature distributions, the original training baseline is stale and will flag normal data as skewed. Updating the baseline to include recent representative data and setting a threshold that accounts for observed variance restores accurate detection. This aligns monitoring with the current production reality without disabling protection.

Feature skew compares the live prediction input distribution to the training data distribution. Inspecting the training dataset baseline reveals whether the baseline itself has shifted or is unrepresentative, which directly explains the reported skew. Without verifying the baseline, you cannot determine if the skew is genuine or an artifact of a stale or incorrect training set.

Percentile latencies (p50, p95, p99) expose the tail behaviour that averages hide, which matters for online prediction where a minority of slow requests breach user-facing SLAs. Vertex AI publishes these under `prediction/latencies` in Cloud Monitoring, directly satisfying the requirement to monitor endpoint latency distribution rather than throughput or resource saturation.

Prediction drift measures changes in the distribution of the model's outputs over time. An alert on prediction drift alone, with no feature skew, suggests that the model's predictions are shifting even though inputs appear stable. This can happen due to concept drift or model degradation, where the relationship between inputs and outputs changes. It signals a potential need to investigate model performance or retrain.

Sampling rate determines what proportion of prediction requests are logged and analysed, directly governing how much data feeds the drift baseline. A higher rate improves detection accuracy for low-traffic endpoints, while a lower rate reduces cost but risks missing subtle distribution shifts.

A Cloud Function provides the event-driven compute layer that reacts to a drift alert and triggers the Vertex AI Pipeline, satisfying the requirement for automatic retraining without manual intervention. It bridges drift detection to pipeline execution.

Vertex AI Model Monitoring computes prediction drift independently of explainability; XRAI only affects feature attribution. Enabling the monitoring job with a training dataset baseline and drift thresholds satisfies the stem's requirement, since drift detection compares incoming prediction distributions against the reference distribution without needing explanations enabled.

Using feature-specific thresholds accounts for differences in natural variability and importance. A feature like age might fluctuate more than a binary flag, so a single global threshold could cause false alerts for age or miss drift in the flag. Tailoring thresholds improves detection accuracy and reduces noise, making alerts more meaningful.

Monitoring requires a reference dataset (the baseline) and a matching schema to compute skew. If the reference dataset is missing or the schema does not align with the endpoint's inputs, the job cannot compute metrics and may silently produce no output. Verifying these configurations is the correct first step to diagnose the issue.

For numerical features, Vertex AI Model Monitoring requires a reference dataset to establish the baseline distribution. The monitoring job compares the live distribution to this baseline to compute drift. Without a reference, drift cannot be calculated. Thus, specifying the numerical distribution and providing the training dataset are essential.

Training-serving skew detection compares the distribution of feature values. If a new category appears in serving data that was not in training, it is considered an unknown value. The monitoring system detects this as a divergence from the training distribution, triggering a skew alert. This is a common occurrence when categorical features have evolving vocabularies.

High-cardinality categorical features often have categories present in live data that were not in the training data, or vice versa. This mismatch in category sets leads to large divergence metrics, such as Jensen-Shannon, because the distributions have disjoint support. Even if the overall distribution seems stable, the presence of unseen categories can cause high skew. Therefore, this is the most likely cause.

Exporting serving data and predictions to BigQuery allows you to perform slice-based analysis, such as comparing recall across different subgroups defined by categorical features. This approach can identify which subgroup is experiencing performance degradation. Vertex AI Model Monitoring itself does not provide subgroup-level performance metrics, so this manual analysis is necessary to diagnose the issue.

Training-serving skew detection requires comparing the feature distributions of the training data to the serving data. Therefore, you must enable skew detection and provide the training dataset as the baseline. This allows Vertex AI Model Monitoring to compute the training distribution and detect any significant differences in serving data, which is essential for identifying skew.

Training-serving skew occurs when the feature distribution in production differs from the training data. If the model was trained on an older customer base, the 'customer_age' distribution would naturally be lower than the current serving population, causing the skew alert. This is a classic example of data drift due to changing population.

Model quality monitoring computes confusion matrices, but it requires labelled ground truth to compare against predictions. Uploading ground truth to BigQuery supplies that reference data, satisfying the stem's need for confusion matrix metrics over time on the Vertex AI Endpoint.

Vertex AI Model Evaluation computes performance metrics and slices results by configured demographic columns, exposing fairness disparities across subgroups. This directly satisfies the stem's need to evaluate each subgroup using BigQuery predictions and ground truth labels.

Feature skew detection compares live input features to the training data distribution. Without specifying a training dataset, Vertex AI cannot compute the skew metric. Therefore, providing a training dataset is a required configuration for feature skew alerts. This is a fundamental step in setting up skew monitoring.

Correct: This is the recommended architecture for automated retraining.

Request/response logging writes prediction failures to Cloud Logging, so a log-based metric counts matching error entries. Cloud Monitoring then alerts on that metric, satisfying the need to detect a rising prediction error rate from the endpoint's existing logs.

Cloud Billing export to BigQuery captures per-replica hourly GPU charges and per-request costs as granular line items, letting you query and attribute spend by endpoint. Budget alerts alone only notify on thresholds; BigQuery analysis satisfies the requirement to track both billing dimensions.

XRAI attribution maps are generated per prediction, so reducing the sampling rate lowers how many explanations are computed, cutting pipeline latency. This satisfies the overhead constraint while still providing statistically useful drift signal from the sampled subset.

Cloud Monitoring alerting policies define conditions that trigger alerts. To alert on prediction latency, you create an alerting policy with a condition that monitors the endpoint's latency metric (e.g., aiplatform.googleapis.com/prediction/latencies). This is the correct resource to define the threshold and trigger notifications.

The 'promotion_flag' feature shows the highest divergence, indicating a significant shift. Comparing recent serving data distribution to the training distribution helps identify whether the feature's meaning or data collection process has changed. This diagnostic step is appropriate before considering retraining or other actions, as it pinpoints the root cause of drift and informs subsequent mitigation steps.

Google Cloud Budgets & Alerts scopes budgets to projects, folders, or labels, and triggers notifications when actual or forecast spend crosses thresholds. Labelling the Vertex AI endpoints lets the engineer isolate their prediction costs, satisfying the requirement to set budget alerts and monitor spending on those endpoints specifically.

If the drift threshold is set higher than the actual drift values, the alert will not fire even though drift is present. The team sees high drift, but it may not exceed the threshold. Adjusting the threshold to a lower value would trigger alerts appropriately.

Vertex AI Model Monitoring compares serving-time feature distributions against the training baseline and raises drift alerts when skew or drift thresholds are exceeded. It is the native service for detecting feature drift on a deployed Endpoint, satisfying the monitoring requirement directly.

Checking CPU and GPU utilisation metrics directly exposes resource saturation on the endpoint's underlying nodes, which is the most likely cause of sporadic >1-second latency spikes amid otherwise fast responses. Vertex AI publishes these per-replica metrics to Cloud Monitoring, letting the engineer correlate spikes with the slow requests seen in the request/response logs.

Vertex AI Model Monitoring supports feature-level drift detection. For categorical features with multiple values, you can set a specific threshold that accounts for the feature's cardinality. By focusing on 'contract_type' and tuning the threshold, the engineer can effectively detect meaningful distribution shifts without being overwhelmed by noise from other features. This targeted approach is best for monitoring a specific high-cardinality categorical feature.

Vertex AI Model Evaluation computes performance metrics, including fairness and bias slices, by comparing model predictions against ground truth labels. Pointing it at the BigQuery label data lets the team evaluate metrics across demographic subgroups, satisfying the fairness monitoring requirement.

Vertex AI Model Monitoring publishes drift metrics to Cloud Monitoring. To receive email alerts, you create an alerting policy on the relevant drift metric and attach an email notification channel. This is the standard integration path and directly satisfies the requirement without custom code.

Vertex AI model quality monitoring reads ground truth labels from a BigQuery table, joining them against logged predictions to compute accuracy, precision and recall. Storing labels in BigQuery satisfies this requirement because the monitoring pipeline expects that source, unlike Cloud Storage objects or local files.

Prediction drift alerts fire when the distribution of model outputs changes beyond a threshold. A major news event can cause a legitimate shift in user behavior and thus in prediction distributions, without indicating model degradation. Raising the threshold based on observed historical variance reduces false alerts while still catching significant deviations that could signal real problems. This balances sensitivity and specificity for the current environment.

Cloud Functions provides an event-driven, serverless trigger that fires when the Cloud Monitoring drift alert publishes to a Pub/Sub topic, invoking the retraining pipeline. This satisfies the requirement to react automatically to drift alerts without managing servers.

Analyzing the age distribution helps distinguish between expected variation (e.g., seasonal changes in patient demographics) and genuine issues like data pipeline errors. Based on this analysis, the team can adjust thresholds or exclude the feature from drift monitoring if it's not predictive of performance. This targeted approach reduces false alerts while maintaining vigilance for critical drifts.

Required to define how often drift is computed.

Lowering the threshold for the feature's drift metric makes the monitoring job more sensitive to deviations from the reference distribution. If the feature's distribution has shifted but the drift value is below the default threshold, reducing the threshold will cause alerts to fire when the shift occurs, improving sensitivity.

Prediction drift can occur even when feature distributions remain stable if the underlying relationship between features and the target changes. This is known as concept drift. For example, user preferences may shift such that the same feature values now lead to different click behavior. Monitoring prediction distribution helps detect such changes, which may require model retraining or adaptation.

Monitoring fewer features reduces the computational load and cost per run. By focusing on the most important features, the team can still detect significant drift while lowering overhead. Vertex AI Model Monitoring allows you to specify which features to monitor. This targeted approach is the most effective way to reduce cost and runtime without sacrificing detection of critical drift.

Logging requests to BigQuery and running scheduled queries to compute drift metrics is a lightweight, serverless approach. It avoids the overhead of managing a Vertex AI Model Monitoring job. Cloud Monitoring can trigger alerts when computed drift exceeds a threshold. This meets the need for a simple solution that compares live data to a reference distribution and sends alerts, without full monitoring infrastructure.

Training-serving skew compares live prediction feature distributions against the training data distribution. If no baseline dataset was supplied when configuring the monitoring job, or if baseline statistics were not generated, Vertex AI cannot calculate skew. The console will show no skew metrics. Providing the training data or enabling baseline generation is required for skew detection.

GPU exhaustion can lead to prediction failures.

A 0.1 sampling rate means only 10% of prediction requests feed the drift calculation, so hourly windows may contain too few samples for statistically reliable distribution comparisons. Raising the sampling rate satisfies the need for sufficient data volume behind each reported drift metric.

BigQuery stores ground truth with prediction timestamp and model version columns, letting Vertex AI join labels to logged predictions for skew and drift analysis. The 24-hour outcome delay fits batch upload, and the schema keys each label to the exact prediction it evaluates.

Pipeline execution duration exposes wall-clock time each run consumes, revealing whether drift-triggered retraining is becoming slower and therefore costlier. Tracking it against a baseline highlights inefficient steps or resource contention, directly supporting the optimisation goal alongside billing and GPU-hour metrics.

A sampling rate of 1.0 logs every prediction, which satisfies the audit requirement that every prediction be recorded with its input features. Directing logs to Cloud Storage and applying a lifecycle policy to delete after 30 days controls storage costs while meeting the retention need. This combination ensures completeness for compliance and cost efficiency, unlike sampling or shorter retention that would violate the audit mandate.

Vertex AI Model Monitoring allows you to set the monitoring window and a minimum number of samples required before computing skew. By using a longer window, you accumulate more instances, improving statistical power. Setting a minimum sample size prevents the job from running on insufficient data, thus reducing false positives and ensuring valid skew detection even with low traffic.

Model Monitoring computes skew and drift by comparing live serving traffic against the training baseline, so it needs sampled request and response payloads. Enabling request/response logging on the endpoint supplies that input, satisfying the stem's requirement for drift detection.

Cloud Billing provides cost breakdowns by service and resource.

Enabling request/response logging on the Vertex AI Endpoint captures the full prediction payloads, satisfying the audit requirement for both inputs and outputs. Routing those logs to BigQuery via a sink then makes them queryable for debugging. Sampling settings must be set to capture all traffic, since default sampling is partial.

XRAI integrates gradients over regions rather than individual pixels, producing saliency maps grouped into coherent areas. For image classification, this regional attribution satisfies the need to highlight which image regions drove the predicted class, which pixel-level methods like Integrated Gradients express less intuitively.

Cloud Monitoring captures Vertex AI Endpoint request metrics, including the count of failed prediction requests, and plots them over time. This directly satisfies the requirement to view error rate trends, since prediction failures are surfaced as monitored metrics rather than logs.

Vertex AI Pipelines logs pipeline execution events to Cloud Logging. By creating a log-based metric that filters for error-level logs from the pipeline, you can then create an alerting policy that triggers when the metric exceeds a threshold (e.g., >0 errors). This is the standard way to alert on pipeline failures using logs.

Vertex AI Model Monitoring computes drift scores and writes them as metrics, but alerts only fire if a Cloud Monitoring alerting policy exists and its condition evaluates the correct metric. If the policy is missing or misconfigured, no notification is sent even when the drift score exceeds the threshold. This is the most likely reason for the discrepancy.