A company is using SageMaker to host a model for real-time inference. They notice that the endpoint's latency increases over time. The model is stateless and the inference code does not log any errors. What is the MOST likely cause?
Memory leaks cause slowdown over time.
Why this answer
A memory leak in the inference container causes the process's resident memory to grow over time as allocated memory is not freed. Since the model is stateless and no errors are logged, the leak is likely in the inference code or a dependency (e.g., a TensorFlow session or a Python list that grows unbounded). As memory pressure increases, the operating system may swap or the container may be OOM-killed, leading to increased garbage collection pauses and higher latency for each request.
Exam trap
The trap here is that candidates confuse a 'stateless model' with 'no memory issues' — but a stateless model means no state between requests, not that the container's memory usage is stable; a memory leak in the inference code or framework can still cause latency degradation over time.
How to eliminate wrong answers
Option B is wrong because a gradual increase in request payload size would cause a sudden or stepwise latency increase when the payload crosses a threshold, not a steady increase over time, and it would be observable in request logs. Option C is wrong because endpoint auto scaling adding new instances would reduce latency by distributing load, not increase it; new instances are warm and ready to serve. Option D is wrong because the model is explicitly stated as stateless, meaning it does not accumulate state from previous requests; if it did, that would contradict the given information and would likely cause errors or state corruption.