PMLE Scaling Prototypes into ML Models Practice Question

• ✓ A.Vertex AI Model Optimization
• B.Vertex AI Model Monitoring
• C.Vertex AI Matching Engine
• D.Vertex AI Continuous Training

Why A: Vertex AI Model Optimization is the correct feature because it provides model quantization, pruning, and distillation techniques specifically designed to reduce model size and improve inference latency on edge devices. This service applies post-training quantization (e.g., FP32 to INT8) and structured weight pruning to shrink the model footprint while maintaining accuracy within acceptable thresholds, directly addressing the need for efficient deployment on resource-constrained hardware.

• A.Pruning only
• ✓ B.Post-training quantization to INT8
• C.Knowledge distillation only
• D.Use full precision FP32 to maintain accuracy

Why B: Post-training quantization to INT8 reduces model weights and activations from 32-bit floats to 8-bit integers, cutting model size by roughly 4x and speeding up inference on edge hardware with minimal accuracy loss. It requires no retraining and works with most trained models. This makes it the best fit for limited-compute edge deployment.

• ✓ A.Post-training quantization using TensorFlow Lite.
• B.Knowledge distillation to train a smaller student model.
• C.Pruning the model weights to zero out unimportant connections.
• D.Use a larger VM for training.

Why A: Post-training quantization with TensorFlow Lite converts a trained model's weights from 32-bit floats to 8-bit integers, reducing model size by ~4x and speeding up inference on edge devices with minimal accuracy loss. TensorFlow Lite is purpose-built for edge deployment, making this the most direct and practical technique for the stated constraint.

• A.Use a larger pre-trained model as a starting point.
• ✓ B.Post-training quantization to INT8.
• C.Use half-precision (FP16) instead of INT8.
• ✓ D.Apply weight pruning to remove small weights.
• E.Increase the number of layers in the model.

Why B: Post-training quantization to INT8 reduces model size by converting 32-bit floating-point weights and activations to 8-bit integers, which also speeds up inference on edge devices with integer-optimized hardware. This technique typically maintains accuracy within 1-2% of the original model while significantly lowering memory footprint and latency.

• A.Pruning and distillation
• B.Quantization and distillation
• ✓ C.Quantization and pruning

Why C: Quantization reduces model size by converting weights from FP32 to INT8, achieving up to 75% size reduction. Pruning removes redundant weights (e.g., 50% pruning) to further shrink the model. Together, these two techniques can reduce model size by over 80% (e.g., 50% pruning + INT8 quantization yields ~87.5% reduction) while preserving acceptable accuracy on edge devices. Knowledge distillation also reduces size but typically requires a pre-trained teacher and may not achieve the extreme compression needed when combined with quantization alone.

• A.Use gcloud CLI to prune the model.
• B.Use Cloud TPU for faster inference.
• ✓ C.Use Vertex AI Model Optimization with TensorRT.
• D.Use TensorFlow.js converter to optimize the model for web.

Why C: Vertex AI Model Optimization with TensorRT is specifically designed to reduce model size and improve inference latency on NVIDIA GPUs by applying techniques like quantization, pruning, and graph optimizations. TensorRT optimizes the model for the target GPU architecture, enabling faster inference with minimal accuracy loss, which directly addresses the engineer's goals.

• A.Only apply weight pruning
• B.Apply quantization-aware training and knowledge distillation
• C.Use knowledge distillation to train a smaller student model
• ✓ D.Apply post-training quantization (INT8) and pruning

Why D: Post-training quantization (INT8) reduces model size ~4x and speeds inference by using 8-bit integers instead of 32-bit floats, while pruning removes redundant weights to further shrink the model and reduce compute. Together they can bring a 200 MB / 50 ms model under 50 MB and under 10 ms with minimal accuracy loss, making them the most practical combination for edge deployment. Quantization-aware training and knowledge distillation are heavier lifts that may not be necessary to hit these targets.

• ✓ A.Post-training quantization to INT8
• B.Using a larger batch size during inference
• ✓ C.Quantization-aware training
• D.Increasing the number of layers
• ✓ E.Pruning of weights with small magnitude

Why A: Post-training quantization to INT8 (A) is correct because it converts the model's FP32/FP16 weights to 8-bit integers after training, cutting memory usage roughly 2-4x with minimal accuracy loss, which is ideal for edge deployment. Quantization-aware training (C) is correct because it simulates quantization during fine-tuning so the Gemma 7B model learns to compensate for reduced precision, yielding better accuracy than post-training quantization alone at the same low-bit footprint. Pruning of weights with small magnitude (E) is correct because removing near-zero weights reduces parameter count and model size, directly lowering the memory footprint on edge devices. Using a larger batch size during inference (B) is incorrect because batch size affects throughput and activation memory, not the model's stored weight footprint, and larger batches actually increase memory use. Increasing the number of layers (D) is incorrect because adding layers enlarges the model and increases memory requirements, the opposite of compression.