PMLE Quick Quiz — 10 Questions

Question 1

A data science team deploys a regression model to predict house prices. After one month, the mean absolute error (MAE) on the serving data increases by 20% compared to the test set. Which monitoring strategy should the team implement first to diagnose the issue?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Monitor prediction residuals and compute serving-time MAE over sliding windows.. The first step in diagnosing a 20% MAE increase on serving data is to monitor prediction residuals over sliding windows. This directly tracks how model errors evolve in production, allowing the team to detect whether performance degradation is sudden or gradual, and to correlate it with specific time windows or data slices. Computing serving-time MAE on sliding windows provides an immediate, interpretable signal of model health without assuming the root cause.

Answer

Retrain the model daily with the latest data to adapt to changing patterns.

Answer

Compare the distribution of training labels with serving labels using a two-sample t-test.

Answer

Monitor input feature distributions for drift using the Kolmogorov-Smirnov test.

Question 2

You are an ML engineer at a logistics company. You have deployed a deep learning model on Vertex AI Endpoints using a custom container with GPU acceleration. The model predicts delivery times based on route features. After one week, you notice that the endpoint's GPU utilization is consistently at 10%, but the prediction latency has increased by 50%. The number of prediction requests per second has remained stable. You check the container logs and see no errors. The model is served using TensorFlow Serving with batching enabled (batch size: 32, batch timeout: 100ms). The custom container uses a single NVIDIA T4 GPU. You have also set the Vertex AI endpoint to use autoscaling with minReplicaCount: 1 and maxReplicaCount: 5, and the CPU utilization target is 60%. Which action should you take to reduce latency?

Accepted Answer

Increase the batch size to 64 and batch timeout to 200ms to improve GPU utilization.. The core issue is low GPU utilization (10%) despite increased latency, indicating that the GPU is underutilized and the bottleneck is likely in batching or data pipeline overhead. Increasing the batch size to 64 and batch timeout to 200ms allows TensorFlow Serving to accumulate more requests per batch, improving GPU throughput and reducing per-request latency by better leveraging GPU parallelism. This directly addresses the mismatch between low GPU utilization and high latency.

Answer

Increase the minReplicaCount to 3 to handle requests in parallel.

Answer

Reduce the CPU utilization target to 40% to trigger more aggressive autoscaling.

Answer

Quantize the model to FP16 to reduce compute time per inference.

Question 3

Your company runs a high-traffic web application that serves the same machine learning model prediction for many identical requests (e.g., product recommendations for the same user profile). You want to reduce latency and load on the prediction endpoint by caching responses. Which Google Cloud service should you use?

Accepted Answer

Cloud Memorystore. Cloud Memorystore (B) is correct because it provides a managed in-memory cache (Redis or Memcached) that can store the results of identical prediction requests, reducing latency and load on the prediction endpoint. By caching responses keyed on the user profile or request parameters, subsequent identical requests can be served directly from Memorystore in microseconds, avoiding redundant model inference.

Answer

Cloud CDN

Answer

Cloud Spanner

Answer

BigQuery

Question 4

Match each model evaluation metric to its use case.

Accepted Answer

Accuracy: Correct when classes are balanced and errors have equal cost. Accuracy is correctly matched with balanced classes and equal error cost; Precision with minimizing false positives; Recall with minimizing false negatives; F1 Score with balancing precision and recall for imbalanced classes. Option E incorrectly pairs Accuracy with 'when false positives are costly' which is actually the domain of Precision. The main trap is confusing accuracy with precision when dealing with asymmetric costs.

Answer

Accuracy: When false positives are costly

Question 5

A company uses Vertex AI Model Monitoring. Which two configuration options can be set to reduce false positive drift alerts?

Accepted Answer

Use a sample percentage of predictions. Using a sample percentage of predictions reduces the volume of data analyzed for drift, which lowers the chance of detecting statistically insignificant fluctuations that could trigger false positive alerts. This is a common technique to filter out noise in high-throughput production systems.

Answer

Set a shorter alerting window

Answer

Decrease the drift threshold

Answer

Enable feature attribution monitoring

Question 6

Which TWO actions should be taken to ensure reproducibility of ML experiments when collaborating across teams on Vertex AI?

Accepted Answer

Lock dependency versions in a container image used for training. Locking dependency versions in a container image ensures that the exact same software environment (e.g., Python packages, CUDA libraries, system tools) is used every time a training job runs. This eliminates variability from package updates or OS patches, which is a fundamental requirement for reproducibility across teams. Vertex AI supports custom containers for training, making this a direct and reliable method.

Answer

Share notebooks via Colab Enterprise with real-time editing

Answer

Allow each team to use their own preferred environment

Answer

Always use random seeds for all random operations

Question 7

Which TWO strategies can help reduce the cost of running ML pipelines on Vertex AI?

Accepted Answer

Use Vertex AI managed datasets to reduce storage costs. Options B and D are correct. Option B is correct because using Vertex AI managed datasets can reduce storage costs by eliminating duplicate copies and providing efficient storage. Option D is correct because preemptible VMs are significantly cheaper than regular VMs, making them cost-effective for fault-tolerant training tasks. Option A is incorrect because hyperparameter tuning with a large search space can increase cost due to many trials. Option C is incorrect because manual scaling is less efficient and can lead to higher costs compared to automated scaling. Option E is incorrect because using a larger machine type for training usually increases cost, even if it reduces training time.

Answer

Run hyperparameter tuning jobs with a large search space

Answer

Manually scale up resources during peak times and scale down during off-peak

Answer

Use a larger machine type for training to complete faster

Question 8

A company has developed a prototype fraud detection model using a small sample of transactions. The prototype runs on a single VM and uses a Random Forest classifier. They want to scale to the full dataset of 50 million transactions. The data is stored in BigQuery. The team wants to use Vertex AI for training. After moving the code to a custom training container and using Vertex AI Training with a single n1-standard-4 machine, the training job fails with an error: "Process terminated with exit code 1". The logs show: "java.lang.OutOfMemoryError: Java heap space". The model uses a scikit-learn RandomForest. Which course of action is most appropriate?

Accepted Answer

Increase the machine type to n1-highmem-8 to provide more memory.. Option B because increasing memory (n1-highmem-8) directly addresses the Java heap space error. Random Forest memory usage scales with data size, so more memory is needed. Option A is not appropriate because scikit-learn does not natively support distributed training, and setting up distributed Random Forest is complex. Option C is unnecessary; switching to a linear model may reduce performance. Option D is wrong because high-CPU machines have less memory per core, which does not help with memory issues.

Answer

Use a distributed training strategy with multiple workers.

Answer

Switch from Random Forest to a linear model to reduce memory usage.

Answer

Switch to a high-CPU machine type like n1-highcpu-16.

Question 9

An ML team is converting a prototype model to a production pipeline using Vertex AI. They want to ensure model versioning and lineage. Which two practices should they adopt? (Select TWO)

Accepted Answer

Use Vertex AI Model Registry to manage model versions.. Options A and C are correct. Using Vertex AI Model Registry allows you to manage and track model versions, while storing model artifacts in Cloud Storage with unique versioned directories ensures lineage and reproducibility. Option B is wrong because keeping only the latest version loses the history and lineage. Option D is wrong because training without tracking violates versioning principles. Option E is wrong because using separate GCP projects per version is unnecessarily complex and does not directly address versioning and lineage.

Answer

Only keep the latest model version to save storage.

Answer

Train models directly in production without tracking.

Answer

Use a separate GCP project for each model version.

Question 10

A data science team deploys a large language model (LLM) on Vertex AI Prediction using an NVIDIA A100 GPU. The end-to-end latency is acceptable, but the cost is high due to low GPU utilization. The model is stateless and requests are independent. Which strategy would most effectively reduce cost per prediction?

Accepted Answer

Implement dynamic batching in the serving container to aggregate multiple requests into a single inference call.. Dynamic batching aggregates multiple independent requests into a single inference call, increasing GPU utilization and reducing cost per prediction without significantly impacting latency if batch size is tuned. Option A is wrong because migrating to TPUs would require model changes and may not improve GPU utilization. Option B is wrong because using a smaller GPU (T4) may increase latency or reduce throughput, and increasing replicas adds cost. Option C is wrong because reducing min replicas to 0 causes cold starts, increasing latency for each request.

Answer

Migrate the model to Cloud TPU using TensorFlow to benefit from higher throughput.

Answer

Use a smaller GPU, such as NVIDIA T4, and increase the number of replicas to maintain throughput.

Answer

Reduce the number of min replicas to 0 and scale from 0 on each request.