Choosing the Right GPU Instance for SageMaker Real-Time Inference

Quick Answer

This question is testing right-sizing rather than raw capability: the model needs GPU acceleration and has a high memory footprint, but the goal is to meet that requirement at the lowest cost, not to provision the largest available GPU instance. The P3 instance family uses NVIDIA V100 GPUs, which deliver the high memory bandwidth needed to keep a large model's inference latency low, so the family itself is well matched to the requirement. Within that family, the 2xlarge size is the smallest configuration, giving a single GPU with enough memory to host the model without paying for additional GPUs or a larger multi-GPU configuration that this single real-time endpoint doesn't need. Choosing the smallest instance in the right family is the pattern to recognize: once you've identified the GPU family that satisfies the technical requirement, cost-minimization means picking the smallest configuration within that family rather than over-provisioning GPUs you won't use. This mirrors how right-sizing works with any AWS instance family, first narrow down to the family that meets the workload's core need, then select the smallest size in that family that still comfortably satisfies it. Whenever a scenario asks you to minimize cost while still meeting a GPU or memory requirement, expect the answer to be the smallest instance size within the appropriate instance family, not the largest or most powerful option available.