A company uses OCI Generative AI service with a Cohere Command model for a real-time chat application and experiences high latency. They have already set max_tokens to 50 and temperature to 0.2. Which further change would be most effective in reducing latency?
Smaller models have fewer parameters and are faster.
Why this answer
Switching to a smaller model variant (e.g., from Command to Command-Light) directly reduces the number of parameters and computational steps per token, which lowers inference latency. Since the company has already minimized max_tokens and temperature, the next most impactful change is to use a less resource-intensive model. This is a common optimization for real-time applications where response speed is critical.
Exam trap
The trap here is that candidates often confuse throughput optimization (asynchronous calls or more GPUs) with latency reduction, but for a single real-time request, model size is the dominant factor.
How to eliminate wrong answers
Option A is wrong because asynchronous invocation does not reduce the latency of a single request; it only decouples the client from waiting for the response, which is unsuitable for a real-time chat application that requires synchronous replies. Option C is wrong because disabling context caching would increase latency, as the model would have to reprocess the conversation history from scratch on every turn, negating the benefit of cached key-value states. Option D is wrong because increasing the number of GPUs does not reduce per-request latency for a single inference call; it improves throughput for concurrent requests but adds overhead for distributing the workload, which can actually increase latency for a single user.