PDE Operationalizing machine learning models Practice Test 12 — 15 Questions

Question 1

A company is designing a CI/CD pipeline for their ML models using Cloud Build and Vertex AI. Which TWO practices should they adopt to ensure reliable and reproducible deployments?

Accepted Answer

Use immutable container images with version tags for each model deployment. Options B and D are correct. B (unit tests for preprocessing) ensures data consistency. D (immutable versioned images) ensures reproducibility. Option A (manual approval for all changes) slows down CI/CD. Option C (staging endpoint) is good but not required for reproducibility. Option E (single source of truth) is important but not a specific CI/CD practice.

Answer

Require manual approval for every model change before deployment

Answer

Store all model artifacts in a single Cloud Storage bucket without versioning

Answer

Deploy every model version directly to production for immediate use

Question 2

A team monitors a deployed Vertex AI model and notices an increasing number of prediction errors with status code 413 (Request Entity Too Large). Which TWO actions should they consider to resolve this issue?

Accepted Answer

Implement client-side pre-processing to compress or downsample input data. Option A is correct because status code 413 indicates the HTTP request payload exceeds the server's size limit. Implementing client-side pre-processing to compress or downsample input data reduces the payload size before it reaches the Vertex AI endpoint, directly addressing the root cause. This approach is efficient because it shifts the computational burden to the client and avoids hitting the server-imposed request size cap, which is typically 1.5 MB for online predictions in Vertex AI.

Answer

Switch the model to batch prediction to handle large payloads offline

Answer

Increase the number of replicas to handle load

Answer

Decrease the machine type to reduce resource consumption

Question 3

During a Vertex AI training pipeline, the training job fails with an error: 'Out of memory: Killed process'. The model is a large deep learning model using TensorFlow. Which THREE steps should the team take to resolve this issue?

Accepted Answer

Enable memory growth configuration in TensorFlow. Option B is correct because TensorFlow by default allocates all available GPU memory, which can cause out-of-memory (OOM) errors when other processes or the system itself need memory. Enabling memory growth with `tf.config.experimental.set_memory_growth` allows TensorFlow to allocate memory incrementally, reducing the risk of OOM kills. This is a direct mitigation for the 'Killed process' error caused by memory exhaustion.

Answer

Change to a distributed training strategy

Answer

Switch the training from GPU to TPU accelerator

Question 4

Your company deploys a classification model on Vertex AI for online predictions. The model is an XGBoost model trained on tabular data with 500 features. The endpoint uses a single n1-standard-4 node. After deployment, users report that predictions take 8-10 seconds on average, while the required SLA is under 2 seconds. You have already verified that the model is not large (under 100 MB) and the input data size is small. The endpoint does not scale automatically. Which action should you take to reduce latency to meet the SLA? A) Change the machine type to n1-highcpu-4 to prioritize compute over memory. B) Enable autoscaling by setting min replicas to 2 and max replicas to 5. C) Switch to a custom container that preloads the model into memory. D) Reduce the number of features by half.

Accepted Answer

Enable autoscaling by setting min replicas to 2 and max replicas to 5.. Option B is correct because the current single node is overloaded; autoscaling distributes traffic across multiple nodes, reducing latency for each request. Option A (CPU-optimized machine) may not help if the bottleneck is not CPU. Option C (preloading) is already default for Vertex AI. Option D (feature reduction) could degrade model accuracy and is not necessary.

Answer

Change the machine type to n1-highcpu-4 to prioritize compute over memory.

Answer

Reduce the number of features by half.

Answer

Switch to a custom container that preloads the model into memory.

Question 5

A retail company uses Vertex AI Pipelines to automate monthly retraining of a recommendation model. The pipeline consists of three steps: (1) extract data from BigQuery, (2) train a TensorFlow model on Vertex AI Training, (3) upload the model to Vertex AI Model Registry and deploy to an endpoint if performance metrics improve. Recently, the pipeline has been failing at step 2 with the error: 'The job was cancelled by the system because it exceeded the maximum training time of 3600 seconds.' You have confirmed that the training code is correct and the data size has not changed significantly. What should you do to fix this pipeline failure? A) Reconfigure the pipeline to use a larger machine type for training. B) Set the training timeout to 7200 seconds in the pipeline configuration. C) Reduce the training dataset size by sampling fewer rows. D) Switch from TensorFlow to a simpler model framework.

Accepted Answer

Set the training timeout to 7200 seconds in the pipeline configuration.. Option B is correct because the default timeout for a training job in Vertex AI Pipelines is 3600 seconds; increasing the timeout allows the job to complete. Option A (larger machine) may help but is not a direct fix for timeout. Option C (reducing data) degrades model quality. Option D (changing framework) is drastic and unnecessary.

Answer

Reduce the training dataset size by sampling fewer rows.

Answer

Switch from TensorFlow to a simpler model framework.

Answer

Reconfigure the pipeline to use a larger machine type for training.

Question 6

A healthcare company deploys a model for diagnosing medical images on Vertex AI using a custom container with a TensorFlow model. The model uses a mixture of GPUs (NVIDIA T4) and CPUs. After deployment, you notice that prediction latency is highly variable: sometimes under 100ms, sometimes over 10 seconds. Investigation shows that the variability correlates with the number of concurrent requests. The endpoint has a min replicas of 1 and max replicas of 3, with target CPU utilization set to 80%. You also observe that GPU utilization remains low (<20%) even during high load. What is the most likely cause of the latency variability? A) The model is not fully utilizing GPUs due to inefficient data loading from CPU. B) The autoscaling metric (CPU utilization) is not appropriate for a GPU-bound workload; the endpoint does not scale based on GPU utilization. C) The GPU machine type is too small for the model. D) The container is not configured to use the GPU correctly.

Accepted Answer

The autoscaling metric (CPU utilization) is not appropriate for a GPU-bound workload; the endpoint does not scale based on GPU utilization.. Option B is correct because Vertex AI scales based on CPU utilization by default, but GPU-bound workloads may have low CPU utilization, causing autoscaling not trigger. Thus, during high load, the single replica is overwhelmed, causing high latency. Option A (inefficient data loading) could contribute but is not the primary cause. Option C (GPU too small) would cause consistently high latency. Option D (GPU not configured) would cause continuous errors, not variable latency.

Answer

The model is not fully utilizing GPUs due to inefficient data loading from CPU.

Answer

The container is not configured to use the GPU correctly.

Answer

The GPU machine type is too small for the model.

Question 7

An e-commerce company uses Vertex AI to serve a real-time personalization model. The model is updated daily via a retraining pipeline that uploads a new version to the same endpoint. Recently, after a model update, the online prediction responses have been returning anomalous results (e.g., recommending irrelevant products). The previous version performed well. The team suspects that the new model is undercooked or has a bug. They have already checked the training code and the pipeline logs, which show no errors. The pipeline deploys the new model version to the endpoint by updating the traffic split to route 100% of traffic to the new version. Which course of action should the team take to quickly mitigate the issue while diagnosing the root cause? A) Roll back the endpoint to the previous model version by setting traffic split to 0% for the new version. B) Delete the current endpoint and recreate it with the previous model version. C) Tweak the training hyperparameters and retrain immediately. D) Increase the number of replicas on the endpoint to handle load.

Accepted Answer

Roll back the endpoint to the previous model version by setting traffic split to 0% for the new version.. Option A is correct because rolling back traffic to the previous known-good version immediately restores correct predictions, while the team investigates the new model. Option B (deleting endpoint) is excessive and causes downtime. Option C (retrain) takes time and may not fix the bug. Option D (more replicas) does not address the incorrect model output.

Answer

Tweak the training hyperparameters and retrain immediately.

Answer

Increase the number of replicas on the endpoint to handle load.

Answer

Delete the current endpoint and recreate it with the previous model version.

Question 8

A data science team has deployed a custom TensorFlow model on Vertex AI Prediction. They notice increasing prediction latency and a growing number of 503 errors during peak traffic hours. The model is served using a single regional endpoint with min replica count of 2 and max replica count of 10. Which TWO actions should the team take to address these issues?

Accepted Answer

Enable autoscaling with a higher max replica count and configure a CPU utilization target of 60%.

Answer

Use a larger machine type (e.g., n1-highmem-8) instead of the current n1-standard-4 to improve per-replica throughput.

Answer

Reduce the min replica count to 0 to allow the service to scale down to zero when not in use.

Answer

Deploy the model as a batch prediction job and move all online predictions to batch.

Question 9

An MLOps team manages a pipeline that retrains an XGBoost classifier weekly using BigQuery data. The pipeline is orchestrated with Cloud Composer and deploys the new model to Vertex AI Endpoint if validation metrics (AUC > 0.9) are met. Over the past month, the deployed model's AUC has dropped from 0.95 to 0.88, despite the training pipeline consistently reporting AUC > 0.9. Which THREE steps should the team take to diagnose and fix this issue?

Accepted Answer

Add a canary deployment step where new model version receives a small percentage of traffic before full rollout.

Answer

Review the training pipeline's hyperparameter tuning configuration to ensure it is not overfitting to stale data.

Answer

Retrain the model using a longer training history to include older data that may still be relevant.

Question 10

Your company has deployed a machine learning model on Vertex AI Endpoint to serve real-time predictions for a mobile application. The model was trained using TensorFlow and the prediction requests include raw images that are preprocessed by the client before sending. Recently, the application developers reported that the predictions are becoming less accurate over time. They suspect the issue is related to changes in the client-side preprocessing code. You need to verify this hypothesis and monitor for future regressions. What should you do?

Accepted Answer

Enable Vertex AI Model Monitoring for feature attribution and set up alerting on skew detection.

Answer

Retrain the model using the latest client data to adapt to any changes in preprocessing.

Answer

Roll back to a previous model version that was known to work well and disable automatic retraining.

Answer

Ask the developers to provide the exact preprocessing code and manually compare it with the training pipeline's preprocessing.

Question 11

Your team is responsible for operationalizing a series of machine learning models that are trained and deployed using Vertex AI Pipelines. The pipeline consists of several steps including data preprocessing, training with hyperparameter tuning, model evaluation, and deployment to an endpoint. Recently, the pipeline has been failing intermittently at the model evaluation step with an error indicating insufficient memory. The evaluation step uses a custom container with a memory limit of 4 GB. The training step uses 8 GB and completes successfully. You need to resolve the failure without drastically increasing costs. What should you do?

Accepted Answer

Optimize the evaluation code to use streaming or incremental processing to reduce peak memory usage.

Answer

Increase the memory limit for the evaluation custom container to 8 GB to match the training step.

Answer

Reduce the batch size used in the evaluation step to lower memory consumption.

Answer

Use a smaller machine type for the evaluation step to force lower memory usage.

Question 12

You manage a large-scale machine learning system that recommends products to users. The model is a deep neural network trained on TensorFlow and deployed on Vertex AI Endpoint with global load balancing. The model receives over 10,000 requests per second. Recently, the team added a new feature: the user's current geographic location (latitude/longitude). After deploying the updated model, you notice that the average prediction latency has doubled, and the error rate has increased, particularly for requests from regions far from the model's primary training data (North America). You suspect the location feature is causing issues. What should you do to diagnose and mitigate the problem?

Accepted Answer

Examine the latency breakdown using Cloud Monitoring to see if the location feature is causing computationally expensive operations, then consider feature engineering like bucketing coordinates.

Answer

Remove the location feature from the model and retrain without it to restore performance.

Answer

Increase the number of replicas for the endpoint to handle the increased latency.

Answer

Switch to a regional endpoint in North America to reduce latency for the majority of users.

Question 13

A startup is using Cloud Build to automate the training and deployment of their machine learning models. The workflow is defined in cloudbuild.yaml and includes steps to: 1) Run a training job on AI Platform Training, 2) Build a custom prediction container, 3) Deploy the container to Cloud Run for serving. The deployment step fails intermittently with the error: 'Cloud Run service already exists and is not owned by the calling user.' You need to fix this so that deployments are reliable. What should you do?

Accepted Answer

Ensure the Cloud Build service account has the 'run.services.update' permission on the Cloud Run service.

Answer

Delete the existing Cloud Run service manually before each build.

Answer

Use 'gcloud run deploy --replace' in the build step to force replace the existing service.

Answer

Use Cloud Run for Anthos instead of fully managed Cloud Run to avoid ownership issues.

Question 14

Your organization deploys multiple versions of the same model to Vertex AI Endpoint for A/B testing. You have a production model (v1) serving 90% of traffic and a candidate model (v2) serving 10%. After one week, you observe that v2 has a slightly lower AUC but significantly higher business metrics like click-through rate. The product team wants to gradually increase v2's traffic. However, you need to ensure that the overall prediction latency remains under 200 ms. Currently, the endpoint has 10 replicas for v1 and 2 replicas for v2. What is the best approach to roll out v2 while maintaining latency SLO?

Accepted Answer

Increase v2's traffic split by 10% each day while also adding replicas for v2 based on CPU utilization.

Answer

Merge v2's model into v1 by retraining v1 with v2's architecture and deploy as a single model.

Answer

Immediately set v2 to serve 100% traffic and monitor latency; if it exceeds 200 ms, roll back.

Answer

Use a separate endpoint for v2 and route traffic at the load balancer level.

Question 15

A financial services company uses a custom container on Vertex AI Prediction to serve a fraud detection model. The container runs a Flask app that loads a large feature engineering library (~2 GB) at startup. The model is updated weekly. For the past two weeks, the new model version has been failing health checks and showing 'Container failed to start' errors in the logs. The previous versions worked fine. You inspect the container image and confirm it is built correctly using Cloud Build. The only change in the latest build is an updated version of the feature engineering library. What is the most likely cause and how should you fix it?

Accepted Answer

The new library version increased memory consumption during startup, exceeding the health check timeout. Increase the startup probe initial delay.

Answer

The Cloud Build step that pushes the image is misconfigured. Rebuild using a different approach.

Answer

The Vertex AI endpoint machine type is too small for the new container. Upgrade to a larger machine type.

Answer

The new library has a dependency conflict that causes the Flask app to crash. Roll back to the previous library version.