NCP-GENL Model Deployment Practice Question
An engineer is deploying a LLM using NVIDIA Triton Inference Server with the TensorRT-LLM backend. They need to ensure that the model can handle a sudden surge in requests without increasing latency beyond a specified threshold. They have configured the model with a maximum batch size of 32 and dynamic batching with a preferred batch size of 16. However, during peak load, latency spikes are observed. Which Triton configuration parameter should they adjust to control the maximum time a request waits in the dynamic batching queue before being processed?
⚠ Common exam trap
The trap here is assuming that increasing batch size or instance count will solve latency spikes, when the root cause is the queue wait time controlled by max_queue_delay_microseconds.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
max_queue_delay_microseconds
The max_queue_delay_microseconds parameter in Triton's dynamic batching configuration specifies the maximum time a request can wait in the queue before being processed. Lowering it reduces latency spikes under surge conditions by preventing requests from waiting too long for a full batch. Other parameters like max_batch_size and preferred_batch_size affect batching but not the wait timeout.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✗
max_batch_size
Why it's wrong here
max_batch_size defines the maximum number of requests that can be batched together. It affects throughput and memory but does not control how long a request waits in the queue. Adjusting it would not directly address latency spikes caused by queuing delay; it might even increase latency if set too high.
- ✗
instance_group
Why it's wrong here
instance_group configures the number of model instances and their placement on GPUs. Increasing instances can improve concurrency but does not control the dynamic batching queue delay. It addresses resource allocation rather than the timing of batch formation, so it would not directly mitigate the latency spikes described.
- ✗
preferred_batch_size
Why it's wrong here
preferred_batch_size specifies the batch sizes that Triton tries to achieve before processing. It influences batching behavior but does not set a time limit for waiting. Without a queue delay limit, requests can still wait indefinitely if the preferred size is not reached, so it does not directly control latency spikes.
- ✓
max_queue_delay_microseconds
Why this is correct
max_queue_delay_microseconds sets the maximum time a request can wait in the dynamic batching queue before Triton processes it, even if the preferred batch size is not reached. Reducing this value limits latency spikes during peak load by forcing earlier execution, though it may reduce batching efficiency. It directly controls the trade-off between latency and throughput.
About these practice questions
This NCP-GENL question is part of Courseiva's 352-question bank — original exam-style content with full explanations and wrong-answer analysis, never real exam questions or exam dumps. Learn why practice questions differ from exam dumps →
JA
Written and reviewed by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
Last reviewed September 2026 · checked against the official NVIDIA exam blueprint
This NCP-GENL practice question is part of Courseiva's free NVIDIA certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the NCP-GENL exam.