PMLE Scaling Prototypes into ML Models Practice Test 10 — 15 Questions

Question 1

You are training a scikit-learn model on Vertex AI using a custom training job. The training dataset is a 2 TB CSV file stored in Cloud Storage, and the job must run on a single CPU-only VM. Loading the entire file into memory fails because the machine has only 32 GB of RAM. You need to train the model without increasing the VM size and without rewriting the training code to use a distributed framework. What should you do?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Use the scikit-learn partial_fit method on an estimator that supports incremental learning, reading the CSV in chunks with pandas.read_csv and feeding each chunk to partial_fit.. Incremental learning with partial_fit is the correct approach because it lets a scikit-learn estimator update its parameters one chunk at a time. Reading the CSV in chunks with pandas.read_csv bounds memory to the chunk size rather than the full 2 TB. This keeps the single-model semantics, avoids distributed training, and fits within the 32 GB VM, directly satisfying all the stated constraints.

Answer

Use Vertex AI Pipelines to split the CSV into many smaller files, then run a separate training job for each file and average the model weights.

Answer

Convert the CSV to TFRecord format and use tf.data to stream batches, then train the scikit-learn model on the streamed batches.

Answer

Mount the Cloud Storage bucket as a local filesystem on the training VM and call pandas.read_csv on the mounted path, relying on the OS page cache to keep memory usage low.

Question 2

A team is using Vertex AI Pipelines to orchestrate a machine learning workflow. They want to ensure that the pipeline can be reproduced and that artifacts are tracked. Which two of the following practices should they follow? (Choose two.)

Accepted Answer

Version all pipeline components and store them in a version control system.. Versioning pipeline components and using Vertex AI Metadata are key practices for reproducibility and artifact tracking. Versioning ensures that the exact code and dependencies are captured, while Metadata automatically records parameters, metrics, and artifacts for each run, providing lineage and enabling reproducibility.

Answer

Use the same container image tags for all pipeline steps to ensure consistency.

Answer

Manually copy all artifacts to a Cloud Storage bucket after each run.

Answer

Disable caching for all pipeline steps to ensure fresh executions.

Question 3

A team is training a TensorFlow model on Vertex AI using a custom container. The training script writes checkpoints to a local directory inside the container. The job runs for 14 hours, and when it completes, the team cannot find the checkpoints in Cloud Storage. They need the checkpoints to be persisted so they can resume training and deploy the best model. What should they do?

Accepted Answer

Configure the training job to write checkpoints to a Cloud Storage URI by passing a gs:// path to the checkpoint directory in the training script, and ensure the Vertex AI service account has storage.objectAdmin on the bucket.. Checkpoints must be written to a durable location, and in Vertex AI custom training the durable location is Cloud Storage. Passing a gs:// URI to the checkpoint directory makes TensorFlow persist each checkpoint through its GCS filesystem, and granting storage.objectAdmin to the training service account authorizes those writes. This allows both resuming training and retrieving the best model for deployment.

Answer

Enable Vertex AI TensorBoard integration and configure the training script to log checkpoints as TensorBoard artifacts, then retrieve them from the TensorBoard instance.

Answer

Set the training job's base output directory to a local path and rely on Vertex AI to automatically upload everything under that path to the job's Cloud Storage output directory at the end of training.

Answer

Increase the boot disk size of the training VM and re-run the job, then copy the checkpoints from the boot disk to Cloud Storage after training finishes.

Question 4

An ML engineer is using Vertex AI Training to fine-tune a large image classification model on a dataset stored in Cloud Storage. The training job uses a custom container and runs on a single NVIDIA V100 GPU. The engineer notices that GPU utilization is consistently low (around 20%) and training is slow. The data is stored as many small JPEG files. What should the engineer do to improve GPU utilization and training speed?

Accepted Answer

Convert the dataset to TFRecord format and use the tf.data API with parallel interleaving and prefetching.. Low GPU utilization during training often indicates that the GPU is starved for data. When data is stored as many small files, the I/O overhead dominates. Converting to TFRecord, a binary format optimized for TensorFlow, and using tf.data with parallel reads and prefetching can dramatically speed up data loading. This ensures the GPU receives data as fast as it can process it, improving utilization and reducing training time.

Answer

Enable Vertex AI Vizier to automatically tune hyperparameters and improve training speed.

Answer

Use a larger GPU instance with more memory, such as an NVIDIA A100.

Answer

Increase the batch size to the maximum that fits in GPU memory.

Question 5

You are scaling a prototype ML model to production on Vertex AI. The model is trained with a custom training job and you want to ensure reproducibility and traceability of each training run. Which two practices should you implement? (Choose two.)

Accepted Answer

Store the training code in a Git repository and pass the commit hash as a hyperparameter or label to the Vertex AI training job.. To ensure reproducibility and traceability, you need to record the exact code version and the training parameters and metrics. Storing the Git commit hash with the training job links the model to the code, while Vertex AI Experiments logs parameters, metrics, and artifacts for each run. Together, these practices allow you to reproduce any model and trace its lineage. Model monitoring, hardware consistency, and timestamped folders do not provide the necessary metadata linkage.

Answer

Store the trained model artifacts in a Cloud Storage bucket with a unique timestamped folder for each run.

Answer

Use the same machine type and accelerator for every training run to ensure identical hardware.

Answer

Enable Vertex AI Model Monitoring on the endpoint to detect training-serving skew.

Question 6

You are training a large tabular model on Vertex AI using a custom training job. The dataset is stored in BigQuery and is several terabytes in size. Training reads the same data for many epochs, and reading directly from BigQuery each epoch is slow and expensive. You want to maximize training throughput while keeping the data accessible to the training container. What should you do?

Accepted Answer

Export the BigQuery table to Cloud Storage in TFRecord or Parquet format, then read the exported files from the training container using the appropriate data loader.. The most efficient approach is to materialize the BigQuery data in Cloud Storage once and then have the training container read those files for every epoch. This removes repeated BigQuery scans, reduces cost and latency, and lets the training job use high-throughput file readers. The other choices either keep the expensive repeated reads or add complexity without fixing the core bottleneck.

Answer

Enable Vertex AI Pipelines caching so that the BigQuery read step is not re-executed on subsequent training runs.

Answer

Increase the number of worker replicas in the training job so that each worker reads a smaller portion of the BigQuery table in parallel.

Answer

Keep reading directly from BigQuery each epoch and rely on the BigQuery Storage Read API to stream rows to the training container.

Question 7

You are deploying a scikit-learn model to Vertex AI for online prediction. The model was trained on a dataset with numerical features and expects input data in a specific JSON format. You have created a custom container that serves the model using a Flask app. After deploying the model to a Vertex AI Endpoint, you send a prediction request with a JSON payload, but the response is an error indicating that the input format is invalid. What is the most likely reason for this error?

Accepted Answer

The custom container's Flask app does not parse the incoming JSON correctly because it expects a different key or structure than what was sent.. The correct answer is that the custom container's Flask app does not parse the incoming JSON correctly. Vertex AI passes the raw request to the container, so the container must implement logic to extract and format the input as expected by the model. If the JSON structure sent by the client does not match what the container expects, an input format error occurs. Ensuring alignment between the client request and container preprocessing is essential.

Answer

The Vertex AI Endpoint requires the input data to be base64-encoded.

Answer

The Vertex AI Endpoint automatically converts the input to a TensorFlow tensor, which is incompatible with scikit-learn models.

Answer

The model artifact was not uploaded to Cloud Storage, so the container cannot load the model and returns an input format error.

Question 8

You are scaling a prototype ML model to production on Vertex AI. The model is a TensorFlow model that you want to train on a large dataset using distributed training across multiple nodes. You need to minimize training time and ensure the job can recover from node failures. Which approach should you take?

Accepted Answer

Use Vertex AI Training with a custom training job that uses the TensorFlow DistributionStrategy with MultiWorkerMirroredStrategy across multiple nodes, and configure the job to save checkpoints to Cloud Storage and resume from the latest checkpoint on failure.. MultiWorkerMirroredStrategy enables multi-node, multi-GPU distributed training in TensorFlow. Combined with checkpointing to Cloud Storage and automatic resume, it provides both scalability and fault tolerance. This approach minimizes training time and ensures recovery from node failures.

Answer

Use Vertex AI Training with a custom training job that uses the TensorFlow DistributionStrategy with CentralStorageStrategy, and enable automatic restart of the job on failure without checkpointing.

Answer

Use Vertex AI Training with a custom training job that uses the TensorFlow DistributionStrategy with MirroredStrategy across multiple GPUs on a single node, and enable checkpointing to Cloud Storage.

Answer

Use Vertex AI Training with a custom training job that uses the TensorFlow DistributionStrategy with TPUStrategy on a TPU pod, and rely on TPU's built-in fault tolerance to handle node failures.

Question 9

You are scaling a prototype recommendation model to production on Vertex AI. The model is trained with a custom container and uses a large embedding table that must be updated frequently. You need to serve predictions with low latency and also support online updates to the embeddings without redeploying the model. Which Vertex AI feature should you use?

Accepted Answer

Deploy the model to a Vertex AI Endpoint with a custom container that loads the embedding table from a shared store such as Cloud Storage or a database at prediction time, and update the shared store independently.. To serve low-latency predictions while allowing online embedding updates, the model should run in a custom container that can read and refresh the embedding table from an external store. This keeps the model deployed while the embeddings change independently. The other options either do not update internal embeddings, require redeployment, or are batch-oriented and therefore unsuitable for low-latency online serving.

Answer

Deploy the model to a Vertex AI Endpoint with a pre-built container and use Vertex AI Feature Store to serve the embeddings as features at prediction time.

Answer

Deploy the model to a Vertex AI Endpoint and use the endpoint's built-in support for model updates through Vertex AI Model Registry versioning.

Answer

Use Vertex AI Batch Prediction with a large batch size so that embeddings are computed in bulk and can be updated between batches.

Question 10

You are training a TensorFlow model on Vertex AI using a custom container. The training job uses a single node with 4 GPUs and a global batch size of 1024. You notice that the training is slower than expected and GPU utilization is low. You suspect the input pipeline is the bottleneck. Which of the following should you do to improve training throughput?

Accepted Answer

Use tf.data to prefetch data and parallelize data extraction and transformation with num_parallel_calls and prefetch.. Low GPU utilization in multi-GPU training often indicates the input pipeline cannot supply data fast enough. Using tf.data with prefetch and parallel calls overlaps data preprocessing with model training, keeping the GPUs busy. This is a standard optimization for TensorFlow input pipelines and directly targets the bottleneck without changing hardware or batch size.

Answer

Reduce the global batch size to 512 to decrease memory pressure on the GPUs.

Answer

Increase the number of GPUs to 8 and double the global batch size to 2048.

Answer

Switch to using TPUs instead of GPUs for the training job.

Question 11

An ML engineer is training a PyTorch model on Vertex AI using a custom training job. The dataset is stored in a Cloud Storage bucket with 500,000 small JPEG images. The training job is configured with an n1-standard-8 machine and a single NVIDIA T4 GPU. The engineer observes that GPU utilization is very low (around 15%) and training is slow. The model code is not the bottleneck. What is the most likely cause and the best solution?

Accepted Answer

The data loading pipeline is inefficient due to many small files; convert the dataset to TFRecord or WebDataset format and use a larger batch size with prefetching.. Low GPU utilization with many small files typically points to an I/O-bound data pipeline. Cloud Storage has high per-file access latency, so reading hundreds of thousands of small JPEGs individually creates a bottleneck. Converting to a sharded format like TFRecord or WebDataset enables efficient sequential reads, and combining with prefetching and larger batches ensures the GPU is continuously fed, improving utilization and training speed.

Answer

The NVIDIA T4 GPU is not powerful enough for this model; upgrade to an NVIDIA A100 GPU.

Answer

The training job is using a single worker and not distributing the load; switch to distributed training with multiple workers.

Answer

The Cloud Storage bucket is in a different region than the training job; move the bucket to the same region.

Question 12

You have a prototype model trained on a single machine using scikit-learn. You now need to scale training to a larger dataset that does not fit in memory on one machine. You want to use Vertex AI training with minimal changes to your existing scikit-learn code. What should you do?

Accepted Answer

Use Vertex AI Training with a custom container that runs your scikit-learn script, and use a distributed framework such as Dask or Ray to partition the data across multiple workers.. The best approach is to keep the scikit-learn code and run it in a custom container on Vertex AI, using a distributed framework like Dask or Ray to partition the data across workers. This scales beyond a single machine's memory while minimizing changes to the original training script. The other options either rely on incompatible distribution strategies, assume automatic distribution that does not exist, or require a full rewrite.

Answer

Convert the scikit-learn model to a TensorFlow model and use MirroredStrategy to train across multiple GPUs.

Answer

Use Vertex AI Training with the pre-built scikit-learn container and set the worker count to more than one; Vertex AI will automatically distribute the scikit-learn training.

Answer

Use Vertex AI distributed training with a ParameterServerStrategy and wrap your scikit-learn estimator in a TensorFlow estimator.

Question 13

You are deploying a custom model to Vertex AI for online prediction. The model requires a preprocessing step that normalizes input features. You want to ensure that the same preprocessing is applied during both training and serving to avoid training-serving skew. What should you do?

Accepted Answer

Export the preprocessing as a TensorFlow Transform (tf.Transform) function and include it in both training and serving graphs.. Using TensorFlow Transform (tf.Transform) to define preprocessing ensures that the same transformations are applied during both training and serving. The transform function is included in the model graph, so the serving container automatically applies the same logic. This eliminates manual replication and reduces the risk of training-serving skew.

Answer

Use Vertex AI Feature Store to serve features and apply preprocessing at training time only.

Answer

Perform preprocessing on the client side before sending data to the model for prediction.

Answer

Include the preprocessing logic in the training script and replicate it in the serving container.

Question 14

An ML engineer is preparing to train a large recommendation model on Vertex AI. The model uses a custom training loop in PyTorch and requires a multi-node cluster with 8 A100 GPUs per node. The engineer wants to minimize training time and ensure the job can recover from a node failure without restarting from scratch. Which combination of Vertex AI features should the engineer use?

Accepted Answer

Use a custom training job with a worker pool of 8 A100 GPUs per replica and multiple replicas, enable distributed training with the PyTorch distributed launcher, and write checkpoints to Cloud Storage periodically.. The requirement is a multi-node A100 cluster with fault tolerance. A Vertex AI custom training job with multiple replicas, each with 8 A100 GPUs, supplies the cluster, and the PyTorch distributed launcher coordinates training across nodes. Writing checkpoints to Cloud Storage at intervals lets the job resume after a node failure without losing all progress, which is the standard pattern for large distributed training on Vertex AI.

Answer

Use Vertex AI Pipelines to orchestrate the training as a series of steps, with each step training on a subset of data, and use pipeline caching to avoid retraining on failure.

Answer

Use a Vertex AI custom training job with a single replica of 8 A100 GPUs and enable automatic machine restart, then rely on the default checkpointing behavior of the PyTorch training loop.

Answer

Use a Vertex AI hyperparameter tuning job with a single replica of 8 A100 GPUs, relying on early stopping to reduce training time.

Question 15

A team is deploying a scikit-learn model to Vertex AI for online prediction. The model requires a custom preprocessing step that scales numerical features using statistics computed from the training set. The preprocessing must be identical between training and serving. The team wants to minimize latency and ensure consistency. What should they do?

Accepted Answer

Wrap the preprocessing and model into a single scikit-learn Pipeline object, and deploy that pipeline as a custom model on Vertex AI.. Wrapping preprocessing and the model into a single scikit-learn Pipeline ensures that the exact same transformations are applied during both training and serving. This eliminates training-serving skew and simplifies deployment. Vertex AI supports deploying custom models with custom prediction routines, but using a native scikit-learn Pipeline is more straightforward and less error-prone, as it encapsulates all steps and can be serialized as one artifact.

Answer

Use a Vertex AI pipeline to preprocess the data before training, and then deploy the model without preprocessing, assuming the input data at serving time will already be scaled.

Answer

Save the scikit-learn model and the scaler as separate artifacts, and in the prediction script, load both and apply the scaler before calling model.predict.

Answer

Use Vertex AI Feature Store to store the scaled features and serve them online, bypassing the need for preprocessing in the model.