GPU Instance for CPU-Bound Model Latency

Quick Answer

The clue that should drive this answer is what happened when the team added more instances: latency didn't improve, which tells you the problem isn't a lack of parallel capacity to handle concurrent requests, it's that each individual request is slow because the model itself is compute-bound on the CPU. Horizontal scaling only helps when the bottleneck is throughput under concurrent load; it does nothing for a per-request latency problem rooted in how long a single inference takes to compute, since each additional instance just runs the same slow computation independently. Because the model is explicitly described as CPU-bound, switching to a GPU-backed instance type like ml.g4dn.xlarge addresses the actual bottleneck directly: GPUs are built for the kind of parallel numerical computation that machine learning inference relies on, so offloading that computation from the CPU to a GPU reduces the time each individual request takes to process, which is exactly the p99 latency metric the team is failing to meet. This is also framed as the most cost-effective fix because it resolves the root cause with a single instance-type change, rather than continuing to add more CPU-based instances that don't actually shrink per-request latency and would just accumulate cost without solving the problem. Whenever a scenario shows that adding instances doesn't improve latency and specifically identifies the workload as compute-bound, treat that as a strong signal to look at accelerating the compute itself, such as moving to a GPU instance, rather than continuing to scale out.