PMLE Hard Questions — 25 Difficult Questions

Question 1

Which TWO actions should be taken to ensure reproducibility of ML experiments when collaborating across teams on Vertex AI?

Accepted Answer

Lock dependency versions in a container image used for training. Locking dependency versions in a container image ensures that the exact same software environment (e.g., Python packages, CUDA libraries, system tools) is used every time a training job runs. This eliminates variability from package updates or OS patches, which is a fundamental requirement for reproducibility across teams. Vertex AI supports custom containers for training, making this a direct and reliable method.

Answer

Share notebooks via Colab Enterprise with real-time editing

Answer

Allow each team to use their own preferred environment

Answer

Always use random seeds for all random operations

Question 2

Which TWO strategies can help reduce the cost of running ML pipelines on Vertex AI?

Accepted Answer

Use Vertex AI managed datasets to reduce storage costs. Options B and D are correct. Option B is correct because using Vertex AI managed datasets can reduce storage costs by eliminating duplicate copies and providing efficient storage. Option D is correct because preemptible VMs are significantly cheaper than regular VMs, making them cost-effective for fault-tolerant training tasks. Option A is incorrect because hyperparameter tuning with a large search space can increase cost due to many trials. Option C is incorrect because manual scaling is less efficient and can lead to higher costs compared to automated scaling. Option E is incorrect because using a larger machine type for training usually increases cost, even if it reduces training time.

Answer

Run hyperparameter tuning jobs with a large search space

Answer

Manually scale up resources during peak times and scale down during off-peak

Answer

Use a larger machine type for training to complete faster

Question 3

A data science team deploys a large language model (LLM) on Vertex AI Prediction using an NVIDIA A100 GPU. The end-to-end latency is acceptable, but the cost is high due to low GPU utilization. The model is stateless and requests are independent. Which strategy would most effectively reduce cost per prediction?

Accepted Answer

Implement dynamic batching in the serving container to aggregate multiple requests into a single inference call.. Dynamic batching aggregates multiple independent requests into a single inference call, increasing GPU utilization and reducing cost per prediction without significantly impacting latency if batch size is tuned. Option A is wrong because migrating to TPUs would require model changes and may not improve GPU utilization. Option B is wrong because using a smaller GPU (T4) may increase latency or reduce throughput, and increasing replicas adds cost. Option C is wrong because reducing min replicas to 0 causes cold starts, increasing latency for each request.

Answer

Migrate the model to Cloud TPU using TensorFlow to benefit from higher throughput.

Answer

Use a smaller GPU, such as NVIDIA T4, and increase the number of replicas to maintain throughput.

Answer

Reduce the number of min replicas to 0 and scale from 0 on each request.

Question 4

A retail company deployed a demand forecasting model using TensorFlow on Vertex AI Batch Prediction. The model runs weekly on a large dataset stored in BigQuery. Over the past month, the prediction accuracy has degraded significantly. The ML engineer reviews the monitoring dashboard and sees that the feature distribution for 'product_price' has shifted from a mean of $50 to $55, and the new product category 'electronics' now represents 20% of the data, whereas it was only 5% in training. The model was never retrained after initial deployment six months ago. The engineer also notices that the Vertex Explainable AI feature importance scores have changed: 'product_price' used to be the top feature (importance 0.35) but now ranks third (importance 0.20). The company requires minimal downtime and wants to improve accuracy as quickly as possible without incurring high costs from excessive retraining. Which course of action should the ML engineer take?

Accepted Answer

Retrain the model using the most recent 3 months of data, including all new product categories, and deploy the updated model via a new Vertex AI endpoint.. The correct action is to retrain the model with the latest data because the feature distributions and data composition have changed significantly, indicating covariate shift and concept drift. Retraining on the most recent 3 months of data (including the new 'electronics' category) will adapt the model to the current data distribution, improving accuracy with minimal downtime by deploying via a new Vertex AI endpoint. Option A (switching to a more complex model without retraining) would not address the drift and may overfit. Option B (routing predictions to human reviewers) is costly, not scalable, and introduces latency. Option D (adjusting prediction thresholds) is insufficient because the model's internal representations are outdated due to the distribution shift, and thresholds alone cannot correct inaccurate predictions.

Answer

Increase the complexity of the model by switching from a feedforward neural network to a gradient boosted tree ensemble, and then deploy without retraining.

Answer

Route all predictions to human reviewers until the model can be re-evaluated, and then manually correct the outputs.

Answer

Adjust the prediction threshold for the 'product_price' feature to account for the price shift, and monitor for another month.

Question 5

A company runs batch predictions on a large dataset using Vertex AI Batch Prediction. They want to reduce costs without significantly increasing processing time. Which three actions should they take? (Choose three.)

Accepted Answer

Use preemptible VMs for the batch prediction job.. Options A, C, and E are correct. A uses preemptible VMs which are cheaper. C uses custom machine types to avoid overprovisioning and reduce costs. E tunes batch size to maximize throughput per worker, reducing the number of workers needed. Option B increases machine size, which may increase cost per worker. Option D uses TPUs, which are more expensive and may not be beneficial for all model types.

Answer

Use a larger machine type to reduce the number of workers.

Answer

Use TPUs instead of GPUs to accelerate processing.

Question 6

A company needs to maintain an audit trail of model changes for compliance. Multiple teams will be updating models. What is the best approach to track who created, modified, or deployed each model version?

Accepted Answer

Use Vertex AI Experiments and Metadata to track model lineage and audit logs. Vertex AI Experiments and Metadata provide a built-in way to track model lineage, including who created, modified, or deployed each version, with automatic audit logging. Option A is wrong because Cloud Storage audit logs only capture access to storage objects, not model-specific metadata or changes. Option B is wrong because Cloud Logging requires manual correlation of log entries to model versions, which is error-prone and not designed for lineage tracking. Option D is wrong because a shared spreadsheet lacks automation, access control, and auditability, making it unsuitable for compliance.

Answer

Enable Cloud Storage audit logs and require all model files to be stored in a bucket

Answer

Use Cloud Logging to collect logs from all services and search for model names

Answer

Ask team members to maintain a shared spreadsheet of changes

Question 7

A team is scaling their prototype inference model to handle high-throughput requests with low latency. They use a custom container on Vertex AI Prediction. They notice that latency spikes occur under heavy load. What is the most effective strategy?

Accepted Answer

Optimize model serving with batching and model warm-up.. Optimizing model serving with batching combines multiple requests into a single inference call, improving throughput, and model warm-up pre-loads models into memory, reducing cold-start latency spikes. Option A is wrong because auto-scaling with higher minimum replicas helps with availability but does not address per-request latency spikes; it can add replicas only after load increases. Option C is wrong because adding more CPUs may not reduce latency if the bottleneck is model inference computation or memory bandwidth; batching and warm-up are more targeted. Option D is wrong because switching to GPUs can improve throughput for compute-heavy models but may not specifically reduce latency spikes caused by request bursts; batching and warm-up directly mitigate transient spikes.

Answer

Enable auto-scaling with a higher minimum number of replicas.

Answer

Use a larger machine type with more CPUs.

Answer

Use a GPU-based machine.

Question 8

You are using Vertex AI Prediction with a custom container that requires a large model file (5 GB). Deployment takes 10 minutes to start. You want to reduce cold start latency. Which action would be MOST effective?

Accepted Answer

Set minReplicas to 1 to keep at least one instance always running.. Cold start latency occurs when a new instance is started and must load the 5 GB model from disk, which can take 10 minutes. Setting minReplicas to 1 ensures that at least one instance is always running and serving, so the model is already loaded in memory, completely avoiding cold start on prediction requests. While other options (compressing the model, using local SSD, or switching to batch prediction) may reduce load time or avoid online serving, they do not eliminate the cold start penalty as effectively as maintaining a warm replica.

Answer

Compress the model file and decompress on startup.

Answer

Use a machine type with local SSD to speed up model loading.

Answer

Switch to batch prediction to avoid online cold start.

Question 9

You are fine-tuning a large language model (LLM) from Vertex AI Model Garden using a custom dataset. You need to minimize training cost while maintaining reasonable throughput. Which THREE strategies should you combine?

Accepted Answer

Use spot VM instances for training. Spot VM instances are significantly cheaper than on-demand instances, reducing training cost. They can be preempted, but for fine-tuning tasks that can checkpoint and resume, this trade-off is acceptable for cost savings.

Answer

Use full fine-tuning of all model parameters

Answer

Use TPU v4 pods for training

Question 10

A team is building a CI/CD pipeline for an ML model. They want to automatically trigger a Vertex AI pipeline for retraining whenever new training data arrives in a Cloud Storage bucket, but only if a specific Pub/Sub notification is published by a data ingestion process. Which approach meets these requirements with minimal operational overhead?

Accepted Answer

Use Eventarc to route the Pub/Sub notification to a Cloud Function that calls the Vertex AI pipeline creation API.. Eventarc can directly listen to a Pub/Sub topic and route matching messages to a Cloud Function, which then calls the Vertex AI pipeline creation API. This serverless approach triggers the pipeline only when the specific Pub/Sub notification is published, meeting the requirement with zero infrastructure to manage and no polling overhead.

Answer

Use Cloud Scheduler to run a job every hour that checks for new files in Cloud Storage and starts the pipeline if new files exist.

Answer

Configure a Cloud Build trigger that listens to the Pub/Sub topic and executes a build step that submits the pipeline run.

Answer

Create a Dataflow streaming pipeline that reads from Pub/Sub and triggers the Vertex AI pipeline via a custom sink.

Question 11

Your team is serving a large language model on Vertex AI using a custom container. The endpoint experiences intermittent 502 errors during traffic spikes. The autoscaling configuration uses a CPU utilization target of 60% and the model is deployed on n1-standard-4 instances. The model requires significant memory. Which combination of changes is most likely to resolve the issue?

Accepted Answer

Switch to a machine type with more memory, e.g., n1-highmem-8, and increase min_replica_count.. The 502 errors likely indicate the instances are overwhelmed or timing out. Increasing the machine type to a high-memory instance reduces memory pressure, and adding more replicas through a lower target scaling metric or higher min replicas provides capacity. Tuning batch size helps but is secondary. GPU may not help if the issue is memory.

Answer

Increase the target CPU utilization to 90% to allow more requests per instance.

Answer

Enable canary traffic splitting to reduce load on the main endpoint.

Answer

Reduce the model batch size from 32 to 1 to lower memory per request.

Question 12

Your team deploys a multi-model endpoint on Vertex AI with two models: Model A (small, low latency) and Model B (large, high latency). You configure traffic splitting so that 90% goes to Model A and 10% to Model B. However, you notice that the latency for Model A increases when Model B receives traffic. What is the most likely cause?

Accepted Answer

The models are collocated on the same instances, leading to resource contention.. In a multi-model endpoint, all models share the underlying infrastructure. When Model B handles requests, it consumes resources (CPU/memory), causing contention that degrades Model A's latency. Collocation of models on the same instance is the issue.

Answer

Model A is being overloaded because autoscaling is based on aggregate traffic.

Answer

The traffic split is misconfigured, causing requests to be routed incorrectly.

Answer

Model B's logging is generating too much output, slowing down the predictor.

Question 13

A data scientist deployed a model to Vertex AI Prediction. When making a prediction request as shown in the exhibit, they receive a 400 error. What is the most likely cause?

Accepted Answer

The model was trained on 2 features, but the request provides 3 features.. The 400 error indicates a malformed request, typically due to a mismatch between the input features the model expects and what is provided. Since the model was trained on 2 features but the request includes 3 features, Vertex AI rejects the prediction as invalid input shape mismatch. This is the most common cause of 400 errors in Vertex AI Prediction when the instance structure does not match the model's signature.

Answer

The request JSON is malformed due to a missing comma between instances.

Answer

The endpoint path is incorrect; it should include the model version.

Answer

The request is sending 3 separate instances but the model expects only 1.

Question 14

A data scientist uses Vertex AI Pipelines to orchestrate an ML workflow. They want to reuse a component from Google's curated repository. What is the recommended way to incorporate it?

Accepted Answer

Use prebuilt components from the Google Cloud Pipeline Components repository. Google provides a curated set of prebuilt components in the Google Cloud Pipeline Components repository, which are designed to be directly imported and used within Vertex AI Pipelines. These components encapsulate common ML tasks (e.g., model training, deployment) and are maintained by Google, ensuring compatibility and reducing custom code. Using them is the recommended approach to avoid reinventing the wheel and to leverage Google's best practices.

Answer

Import the component from Google Cloud Build

Answer

Use the 'aiplatform' Python SDK to define the component

Answer

Copy the component code into the pipeline definition

Question 15

Which TWO actions are recommended to detect and mitigate data drift in a production ML system on Vertex AI?

Accepted Answer

Automatically retrain the model when drift exceeds thresholds. Vertex AI's automated retraining pipeline can be triggered when data drift exceeds a predefined threshold, ensuring the model adapts to distribution changes without manual intervention. Option D is correct because Vertex AI Model Monitoring continuously tracks feature distribution statistics (e.g., using Jensen-Shannon divergence or L-infinity distance) and sends alerts when drift is detected, enabling proactive mitigation.

Answer

Deploy multiple models and use an ensemble to average predictions

Answer

Manually review model predictions daily

Answer

Monitor prediction errors and flag when confidence is low

Question 16

You are a machine learning engineer at a financial technology company. You have deployed a complex ensemble model consisting of three sub-models (XGBoost, TensorFlow, and PyTorch) for real-time fraud detection. The model is served on Vertex AI online prediction with a custom container that orchestrates the three models sequentially. The endpoint currently uses n1-highmem-8 machines with no accelerators. You are experiencing high latency (avg 500ms) during peak trading hours (9:30 AM - 4:00 PM EST), exceeding the 200ms SLA. The container is CPU-bound, and memory usage is around 60%. The model weights total 500 MB. You have already tried increasing the batch size per request from 1 to 4, which reduced latency slightly but not enough. The traffic pattern is very spiky, with sudden bursts of up to 1000 requests per second. Your goal is to meet the latency SLA without significantly increasing cost. Which action should you take?

Accepted Answer

Add a NVIDIA T4 GPU accelerator to the existing machine type.. Adding a GPU accelerator (e.g., NVIDIA T4) to the instances can significantly speed up the TensorFlow and PyTorch components, which are deep learning models. The XGBoost part runs on CPU but the overall latency bottleneck is likely the deep learning models. GPU will accelerate inference of those models, reducing total latency. Increasing CPUs will help only marginally as the main bottleneck is compute. Reducing min replicas may increase cold start and queue. Switching to batch prediction changes the model from real-time to batch, which does not meet the latency requirement.

Answer

Reduce the min_replica_count to 0 to allow scaling down aggressively and add more replicas during spikes.

Answer

Increase the machine type to n1-highmem-16 with more vCPUs.

Answer

Switch the model to Vertex AI batch prediction and run predictions every hour.

Question 17

A company runs a Vertex AI pipeline that uses a container component to preprocess data. The component downloads a large file from a public URL and saves the output to Cloud Storage. The pipeline fails intermittently with a 'timeout' error. Which THREE steps should the team take to improve reliability? (Choose three.)

Accepted Answer

Make the component idempotent by checking for existing output before processing.. Making the component idempotent by checking for existing output before processing prevents redundant work and avoids timeout failures when the file has already been downloaded. In Vertex AI pipelines, idempotent components can safely skip processing if the output already exists in Cloud Storage, reducing the risk of hitting timeout limits on subsequent pipeline runs.

Answer

Reduce the size of the file being downloaded.

Answer

Increase the machine type for the component.

Question 18

You are designing a batch prediction pipeline using Vertex AI. The input data is 100 TB of images stored in Cloud Storage. The model is a custom TensorFlow model that expects TFRecord format. The pipeline must be cost-effective and run within a time window of 2 hours. Which THREE steps should you include?

Accepted Answer

Create a Vertex AI batch prediction job with input from GCS (TFRecord files).. Vertex AI batch prediction jobs natively accept TFRecord files stored in Cloud Storage as input, which aligns with the requirement for a custom TensorFlow model. This approach is cost-effective and can complete within 2 hours by leveraging Vertex AI's managed infrastructure, avoiding the need to spin up and manage compute resources manually.

Answer

Store batch prediction results in BigQuery.

Answer

Use Cloud Functions to convert images to TFRecord.

Question 19

A company uses Vertex AI Prediction with a custom container for a TensorFlow model. They notice that after deploying a new model version, requests still go to the old version. What is the most likely cause?

Accepted Answer

Traffic is not split to the new model version. In Vertex AI Prediction, when you deploy a new model version to an existing endpoint, you must explicitly allocate traffic to it. By default, the new version receives 0% traffic, so all requests continue to be served by the old version. The correct fix is to update the endpoint's traffic split, for example via the console or the `gcloud ai endpoints update` command with the `--traffic-split` flag.

Answer

The custom container is not compatible with Vertex AI

Answer

The model is cached and needs cache invalidation

Answer

The new model version was not deployed to the same endpoint

Question 20

A retail company deploys a new recommendation model alongside the current champion on Vertex AI Endpoints. They want to gradually shift traffic to the challenger while monitoring business metrics (conversion rate). Which two steps are required? (Choose 2)

Accepted Answer

Deploy the challenger model to the same endpoint as the champion with a separate deployed model.. Vertex AI Endpoints support deploying multiple model versions (champion and challenger) to the same endpoint, each as a separate deployed model. This allows the endpoint to serve both models simultaneously, enabling traffic splitting without requiring separate endpoints or infrastructure.

Answer

Use Vertex AI Experiments to track the traffic split percentages.

Answer

Enable Cloud Memorystore to cache identical requests for both models.

Answer

Use Cloud Monitoring to track custom metrics like conversion rate per model version.