PMLE Scaling Prototypes into ML Models Practice Test 9 — 15 Questions

Question 1

You are training a model on Vertex AI using a custom training job. The training data is stored in a Cloud Storage bucket in the us-central1 region, and the training job runs in the us-central1 region. You notice that the training job takes significantly longer than expected due to data loading. You want to improve data loading performance without changing the model architecture. What should you do?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Convert the training data to TFRecord format and use the tf.data.TFRecordDataset with parallel interleave and prefetching.. Converting data to TFRecord and using tf.data with parallel interleave and prefetching optimizes the input pipeline by reducing the number of read operations and overlapping data loading with computation. This is a standard best practice for improving training performance on Vertex AI when data is in Cloud Storage.

Answer

Copy the training data to the local SSD of the training VM before training begins, and read from there.

Answer

Enable streaming reads from Cloud Storage by using the tf.data.experimental.make_csv_dataset function with a large buffer size.

Answer

Use the Vertex AI Training reduction server to accelerate data loading.

Question 2

You are preparing to train a large image classification model on Vertex AI using a custom training job. You want to optimize the training job for cost and performance. The dataset is stored in Cloud Storage as TFRecords and is about 2 TB. You plan to use a machine with 4 NVIDIA V100 GPUs. Which two actions should you take to improve training efficiency? (Choose two.)

Accepted Answer

Use a larger batch size and scale the learning rate accordingly.. Mixed precision training and increasing batch size with learning rate scaling are both effective ways to improve training efficiency on V100 GPUs. Mixed precision leverages tensor cores for faster computation and reduced memory, while larger batches improve GPU utilization. These actions directly target performance and cost without altering the model architecture or data storage.

Answer

Use a smaller model architecture to reduce training time.

Answer

Store the TFRecords in a regional Cloud Storage bucket instead of a multi-regional bucket.

Answer

Increase the number of vCPUs on the machine to improve data loading throughput.

Question 3

You are training a TensorFlow model on Vertex AI using a custom container with a single Tesla T4 GPU. You notice that training is slower than expected, and GPU utilization is consistently below 20%. Profiling shows that the input pipeline is the bottleneck. Which change should you make to improve GPU utilization?

Accepted Answer

Use tf.data with prefetching and parallel data extraction to overlap data loading with GPU computation.. When the input pipeline is the bottleneck, the GPU waits for data. Using tf.data with prefetching and parallel extraction overlaps CPU preprocessing with GPU computation, ensuring the GPU is continuously fed. This is the standard TensorFlow optimization for such scenarios and directly improves GPU utilization.

Answer

Switch to a larger GPU instance with more memory to reduce data loading overhead.

Answer

Move the dataset to a local SSD on the training VM to reduce I/O latency.

Answer

Increase the batch size to fully utilize GPU memory.

Question 4

You are deploying a scikit-learn model to Vertex AI for online prediction. The model requires a custom preprocessing step that involves scaling numerical features and one-hot encoding categorical features. You have packaged the preprocessing and model into a single Python script that uses a custom prediction routine. You need to ensure that the online prediction service can handle varying input formats and provide low-latency responses. What should you do?

Accepted Answer

Use Vertex AI's custom prediction routine with a preprocessor that accepts raw input and transforms it, and deploy the model as a model artifact with the custom routine.. Using a custom prediction routine with a preprocessor allows you to encapsulate preprocessing and model logic in a single package, ensuring consistency between training and serving. Vertex AI manages the infrastructure, providing low-latency and scalability. This is the most efficient and maintainable approach for scikit-learn models with custom preprocessing.

Answer

Deploy the model as a custom container on Vertex AI, and implement the preprocessing logic inside the container's HTTP server.

Answer

Use Vertex AI's feature store to perform preprocessing, and then feed the features directly to the model.

Answer

Deploy the model using Vertex AI's built-in scikit-learn container, and perform preprocessing on the client side before sending requests.

Question 5

You are training a scikit-learn random forest model on a large tabular dataset using a Vertex AI custom training job. The training script reads a CSV file from a Cloud Storage bucket and writes the trained model artifact to the same bucket. You need to ensure the training job can access the Cloud Storage bucket without embedding long-lived credentials in the container. What should you do?

Accepted Answer

Attach a custom service account to the Vertex AI training job that has the Storage Object Viewer and Storage Object Creator roles on the bucket, and let the Vertex AI Training service use the service account's credentials automatically.. The correct approach is to attach a custom service account with least-privilege Cloud Storage roles to the Vertex AI training job. Vertex AI training jobs automatically use the credentials of the attached service account via the metadata server, so no key files are needed. This ensures secure, short-lived authentication and follows Google Cloud best practices for access control.

Answer

Grant the Vertex AI Service Agent role to the default Compute Engine service account and rely on the default credentials provided by the training environment to access the bucket.

Answer

Configure the training job to use a user-managed service account that has the Editor role on the project, ensuring it has sufficient permissions to read and write to any Cloud Storage bucket.

Answer

Create a service account key file, store it in Secret Manager, and mount it into the training container as a volume so the scikit-learn script can read the key and authenticate to Cloud Storage.

Question 6

You are preparing a Vertex AI custom training job that uses a custom Docker image built on top of the PyTorch pre-built training container. The image must be able to read training data from a Cloud Storage bucket without embedding credentials in the image. You want the job to run on a single NVIDIA T4 GPU. Which approach should you take?

Accepted Answer

Specify a user-managed service account with the Storage Object Viewer role when submitting the custom job, and let the container use Application Default Credentials to read from Cloud Storage.. A user-managed service account attached to the custom job provides the identity that the training container uses. Vertex AI injects credentials via the metadata server, so Application Default Credentials in the container can authenticate to Cloud Storage. Granting the least-privilege Storage Object Viewer role satisfies the data access requirement without embedding keys. The job can also be configured with a single T4 GPU accelerator to meet the compute requirement.

Answer

Mount the Cloud Storage bucket as a local file system using Cloud Storage FUSE inside the container and rely on the bucket's default IAM permissions for anonymous access.

Answer

Hard-code a service account JSON key file into the Docker image and set GOOGLE_APPLICATION_CREDENTIALS in the Dockerfile.

Answer

Grant the Vertex AI Custom Code Service Agent the Storage Object Viewer role and let the training container use the default credentials from the metadata server.

Question 7

You have a trained XGBoost model that you want to deploy on Vertex AI for online prediction. The model expects input features in a specific order and requires a custom preprocessing step that normalizes numerical features using statistics computed during training. You need to ensure that the same preprocessing is applied at serving time. What should you do?

Accepted Answer

Deploy the model using a custom container that includes the preprocessing code and the trained model artifacts, and implement the preprocessing in the container's prediction handler.. For custom preprocessing with non-TensorFlow models like XGBoost, a custom container is the most reliable way to ensure that the exact same preprocessing is applied at serving time. You can embed the training statistics and logic in the container, maintaining consistency and simplifying client requests.

Answer

Use Vertex AI's built-in preprocessing feature by specifying a preprocessing function in the model's metadata when uploading.

Answer

Export the model as a SavedModel and include the preprocessing logic in the model's serving signature using TensorFlow's preprocessing layers.

Answer

Preprocess the input data on the client side before sending requests to the Vertex AI endpoint, using the same statistics.

Question 8

An ML engineer is training a TensorFlow model on Vertex AI using a custom training job with a single worker and multiple GPUs. The training script uses tf.distribute.MirroredStrategy. After a few epochs, the job fails with a NCCL timeout error. The engineer confirms the GPUs are healthy and the batch size is reasonable. What should they do to resolve the error?

Accepted Answer

Set the environment variable NCCL_SOCKET_IFNAME to the correct network interface and ensure the container image includes a compatible NCCL version.. NCCL timeout errors in a single-worker, multi-GPU Vertex AI training job usually stem from NCCL being unable to select the correct network interface or from incompatible NCCL versions. Explicitly setting NCCL_SOCKET_IFNAME to the primary interface and using a container image with a validated NCCL build ensures that inter-GPU communication initializes correctly. MirroredStrategy does not require TF_CONFIG, so adding workers or TF_CONFIG would not help.

Answer

Increase the number of workers in the worker pool configuration to match the number of GPUs.

Answer

Set the environment variable NCCL_DEBUG=INFO and reduce the per-GPU batch size to lower memory pressure.

Answer

Ensure the training job is configured with a single replica and that the container sets the TF_CONFIG environment variable appropriately for a single-worker, multi-GPU setup.

Question 9

An ML engineer has a prototype that trains a TensorFlow model on a single CPU machine using Vertex AI custom training. The job now needs to train on a larger dataset and must use multiple GPUs on one machine. The training script already uses tf.distribute.MirroredStrategy. What change is required to scale the job?

Accepted Answer

Set the worker pool machine type to a GPU machine and specify the number of accelerators and accelerator type.. Scaling a single-machine TensorFlow job that already uses MirroredStrategy to multiple GPUs is primarily a resource configuration task. You set the machine type to a GPU-capable machine and specify the accelerator type and count in the worker pool spec. The strategy code detects the local GPUs automatically. Multi-worker strategies, TPU strategies, and disk or image changes are not required for this scenario.

Answer

Add a second worker pool with one replica and set the distribution strategy to MultiWorkerMirroredStrategy.

Answer

Enable TPU training by specifying a TPU machine type and changing the script to use TPUStrategy.

Answer

Increase the boot disk size and set the training container to use the GPU-enabled base image.

Question 10

You are deploying a scikit-learn model to Vertex AI for online predictions. The model requires a custom preprocessing step that transforms raw JSON input into a feature vector before calling predict. You want to minimize latency and avoid managing infrastructure. What should you do?

Accepted Answer

Create a custom container that includes the scikit-learn model and a Flask app that performs preprocessing and calls predict, then deploy it to a Vertex AI endpoint.. Vertex AI custom containers for prediction allow you to bundle the scikit-learn model, custom preprocessing code, and an HTTP server such as Flask. This keeps preprocessing and inference in the same process, reducing latency, and Vertex AI handles scaling and infrastructure management. Pre-built containers do not support arbitrary preprocessing, and batch prediction or model conversion would not meet the low-latency online requirement.

Answer

Export the scikit-learn model to a TensorFlow SavedModel and deploy it with the TensorFlow pre-built container, using a preprocessing layer in the model.

Answer

Deploy the scikit-learn model using the pre-built scikit-learn container and implement preprocessing in a Cloud Function that calls the endpoint.

Answer

Use Vertex AI Batch Prediction with a pre-built scikit-learn container and a custom preprocessing script.

Question 11

A team is training a large recommendation model on Vertex AI using a custom container. They need to log training metrics and visualize them in Vertex AI TensorBoard. The training code is written in PyTorch and runs on multiple worker nodes. Which of the following is the correct way to enable TensorBoard logging?

Accepted Answer

Use the Vertex AI SDK to create a TensorBoard instance and pass its resource name to the training job using the --tensorboard flag.. To integrate with Vertex AI TensorBoard, you must create a TensorBoard instance and pass its resource name to the training job. The training code should write logs to the directory specified by the AIP_TENSORBOARD_LOG_DIR environment variable. This allows Vertex AI to automatically collect and display metrics from all workers in the managed TensorBoard service.

Answer

Install TensorBoard in the custom container and run it as a sidecar process on each worker node.

Answer

Write training metrics to a Cloud Storage bucket and then manually upload them to Vertex AI TensorBoard after training.

Answer

Use the torch.utils.tensorboard.SummaryWriter to write logs to a local directory, and Vertex AI will automatically sync it to TensorBoard.

Question 12

You have a prototype model trained on a small sample of data. You now want to train on the full dataset using Vertex AI, but the dataset is stored in BigQuery and is several terabytes. You need to minimize data movement and avoid exporting the full dataset to Cloud Storage. What should you do?

Accepted Answer

Use the BigQuery reader in the training application to read data directly from BigQuery during training.. When training data lives in BigQuery and the goal is to avoid moving it, the training application should read from BigQuery directly using the available client libraries or Storage API. This keeps the data in place and avoids exporting terabytes to Cloud Storage. Exporting to CSV or Parquet, or routing through Feature Store, adds unnecessary copies or is not designed for bulk training reads.

Answer

Export the BigQuery table to Cloud Storage as CSV and pass the Cloud Storage path to the training job.

Answer

Create a Dataproc cluster to convert the BigQuery table to Parquet in Cloud Storage, then train from that path.

Answer

Load the BigQuery table into a Vertex AI Feature Store and train from the feature store online serving endpoint.

Question 13

You have a prototype ML model that you want to scale to production on Vertex AI. The model is a Python function that performs simple data preprocessing and then calls a pre-trained scikit-learn model. You need to deploy this as a batch prediction job that runs weekly on a large dataset stored in BigQuery. What is the most efficient way to accomplish this?

Accepted Answer

Create a custom container that includes the preprocessing code and the scikit-learn model, push it to Artifact Registry, and use it to run a Vertex AI Batch Prediction job with BigQuery as the input source.. For batch prediction with custom preprocessing, a custom container is required because pre-built containers do not include your code. Vertex AI Batch Prediction natively supports BigQuery as both input and output, so you can avoid data movement. This method is efficient, scalable, and integrates with Vertex AI's managed infrastructure.

Answer

Deploy the model to a Vertex AI Endpoint and write a script that sends all BigQuery rows as individual prediction requests, then store the results in BigQuery.

Answer

Use a Vertex AI Pipelines job that runs a Dataflow transform to preprocess the BigQuery data, then calls a pre-built scikit-learn model for prediction and writes results back to BigQuery.

Answer

Export the BigQuery data to Cloud Storage as CSV files, then run a Vertex AI Batch Prediction job using a pre-built scikit-learn container that reads from Cloud Storage.

Question 14

A machine learning engineer is training a model on Vertex AI using a custom container. The training job uses a large dataset stored in BigQuery. The engineer wants to minimize data transfer costs and maximize training speed. Which of the following approaches is most efficient?

Accepted Answer

Use the BigQuery Storage API to read data directly into the training program with parallel streams.. The BigQuery Storage API is designed for high-performance data reading, offering parallel streams and efficient data transfer. It avoids the overhead of exporting data to Cloud Storage or loading into memory all at once. This makes it the most efficient and cost-effective approach for training on large BigQuery datasets.

Answer

Run a query in BigQuery to extract the data, then save it to a local SSD on the training VM.

Answer

Export the BigQuery table to Cloud Storage in TFRecord format, then read it using the tf.data API.

Answer

Use the BigQuery client library to run a query and fetch results into memory using the to_dataframe() method.

Question 15

An ML engineer has a prototype scikit-learn model that must be served on Vertex AI. The model requires a custom preprocessing step that cannot be expressed in a scikit-learn Pipeline. The engineer wants to package the model with this preprocessing logic and deploy it to a Vertex AI Endpoint for online predictions. Which approach should they take?

Accepted Answer

Create a custom container that includes the scikit-learn model and the preprocessing code, push it to Artifact Registry, import it as a Vertex AI Model, and deploy to an Endpoint.. When a model requires custom preprocessing that is not part of the model artifact itself, the most reliable way to serve it on Vertex AI is to build a custom container. The container can include the model, the preprocessing code, and all dependencies, ensuring consistent behavior between training and serving. Vertex AI then deploys this container as a Model resource and makes it available via an Endpoint.

Answer

Use Vertex AI Model Monitoring to apply the preprocessing transformations before the model receives the request.

Answer

Save the model with joblib, upload it as a Vertex AI Model, and deploy it to an Endpoint; Vertex AI automatically applies the preprocessing.

Answer

Deploy the model as a batch prediction job, which automatically applies preprocessing from a saved scikit-learn Pipeline.