NCP-GENL Model Deployment — 10 Questions

Question 1

Refer to the exhibit. An engineer configures Triton for dynamic batching. If requests arrive at 2ms intervals and the current queue is empty, what is the expected batching behavior?

Accepted Answer

It will wait 5ms and execute whatever requests are in the queue.. With a max delay of 5000 microseconds (5ms) and arrival intervals of 2ms, Triton will wait to collect requests until the delay limit is reached or the preferred batch size is met. Since the preferred batch size is 4, it will wait for at least four requests to arrive. The time taken to receive four requests at 2ms intervals is 6ms, which exceeds the max delay. Therefore, it will trigger a batch after 5ms.

Answer

It will execute a batch of 1 immediately upon arrival.

Answer

It will wait for exactly 8 requests before starting inference.

Answer

It will error out because the interval is smaller than the delay.

Question 2

Which THREE techniques are effective for managing memory in high-concurrency LLM deployments on NVIDIA GPUs? (Select THREE)

Accepted Answer

Implementing PagedAttention to reduce KV cache fragmentation.. Managing memory for LLMs is critical because they occupy significant VRAM. Techniques like PagedAttention effectively manage the KV cache, preventing fragmentation. Model parallelism (tensor or pipeline) allows splitting models across multiple GPUs when they exceed single-device capacity. Quantization directly reduces the memory footprint of weights, enabling higher concurrency. Collectively, these ensure memory is utilized efficiently without sacrificing performance or falling into fragmented memory states.

Answer

Increasing the batch size to the maximum theoretical hardware limit.

Answer

Disabling the CUDA graph capture to save VRAM overhead.

Question 3

When migrating an LLM deployment to a multi-GPU setup, what is the significance of 'Tensor Parallelism'?

Accepted Answer

It partitions model layers across GPUs for memory efficiency.. Tensor Parallelism splits the individual matrix operations of a single model layer across multiple GPUs. This is necessary when a model is too large to fit into the memory of a single GPU or when high-speed generation requires the combined compute power of multiple devices. By parallelizing the weights and computations, the model can execute at scale without exceeding the physical constraints of a single GPU's VRAM.

Answer

It allows multiple different models to run on one GPU.

Answer

It replicates the entire model on every available GPU.

Answer

It accelerates model training by caching gradients.

Question 4

An engineering team is deploying a Transformer-based model using NVIDIA TensorRT-LLM on an H100 GPU cluster. Which TWO factors are most critical to consider when configuring the In-flight Batching (IFB) feature for optimal performance?

Accepted Answer

The total VRAM allocated for the KV cache and the maximum number of concurrent sequences.. In-flight Batching allows new requests to join an existing batch as soon as slots become available, rather than waiting for the entire batch to complete. This is vital for autoregressive decoding where output lengths vary significantly. Understanding KV cache memory management and the maximum number of concurrent sequences is essential to prevent OOM errors and ensure that the H100's high compute density is fully utilized during inference.

Answer

The clock frequency settings for the GPU's high-bandwidth memory (HBM).

Answer

The specific activation function used within the model's feed-forward layers.

Answer

The network latency between the load balancer and the inference server instances.

Question 5

Refer to the exhibit. An engineer observes that the LLM deployment exhibits high latency and inefficient GPU utilization under moderate load. Based on the provided Triton configuration, what is the most likely cause?

Accepted Answer

The dynamic_batching parameter is incorrectly applied to the TensorRT-LLM backend.. The exhibit shows that dynamic batching is enabled for a tensorrtllm backend, which is fundamentally incompatible with the In-flight Batching requirements of modern LLM serving. TensorRT-LLM backends should manage batching through the engine itself rather than the Triton dynamic batcher. By using the dynamic batcher, the server introduces a queuing mechanism that interferes with the model's native ability to process tokens as they are generated.

Answer

The instance count is set to 1, which prevents multi-GPU parallelism.

Answer

The preferred_batch_size values are too high, causing memory allocation errors.

Answer

The backend name is case-sensitive and should be capitalized as TENSORRTLLM.

Question 6

A healthcare company runs an NVIDIA Triton Inference Server hosting a clinical-summarization LLM. Compliance requires that every generated summary be fully reproducible for audit: the same prompt must always yield byte-identical output, and token-level probabilities must be retrievable later. The team currently sends requests with default sampling parameters and stores only the final text. Which change to the deployment configuration best satisfies the audit requirement?

Accepted Answer

Switch the model to greedy decoding by setting beam_width to 1 and top_k to 1, and set return_log_probs to true in the Triton request so token-level probabilities are returned with each response.. Auditable generative AI requires deterministic token selection plus retention of the probability distribution used at each step. Greedy decoding with beam_width 1 and top_k 1 eliminates sampling randomness, guaranteeing that the same prompt maps to the same output. Setting return_log_probs to true makes Triton include per-token log probabilities in the response, giving auditors the quantitative evidence they need alongside the text.

Answer

Enable Triton's dynamic batching and set max_batch_size to 1, then log the raw prompt and final response text to an immutable object store.

Answer

Keep the default sampling parameters but lower temperature to 0.0 and increase top_p to 1.0 so the model becomes deterministic.

Answer

Deploy a second Triton instance with a fixed random seed and route audit requests to it, while production traffic continues using default sampling.

Question 7

A production LLM service on NVIDIA Triton Inference Server must handle bursty traffic while maintaining low latency. The team wants to cap the number of concurrent requests per model instance and queue excess requests. Which Triton configuration parameter should they tune?

Accepted Answer

Model instance concurrency limit via the backend's max concurrency setting.. Capping concurrent requests per model instance is achieved through the backend's concurrency limit, which queues excess requests rather than admitting them immediately. This protects GPU resources and keeps latency predictable under bursts. Parameters like max_batch_size, instance_group count, and queue delay affect batching and parallelism but do not enforce a per-instance concurrency ceiling.

Answer

instance_group count in the model configuration.

Answer

max_batch_size in the model configuration.

Answer

dynamic_batching max_queue_delay_microseconds.

Question 8

A financial services company is deploying a 13B-parameter LLM using NVIDIA TensorRT-LLM. Their security policy requires that model weights never leave the GPU memory unencrypted and that inference requests are processed with the lowest possible overhead. They have a single NVIDIA H100 GPU with 80 GB memory. Which TensorRT-LLM feature should they enable to meet these requirements while maximizing throughput?

Accepted Answer

TensorRT-LLM with NVIDIA confidential computing enabled. Confidential computing on NVIDIA H100 GPUs provides hardware-based encryption of data in use, ensuring model weights are protected while resident in GPU memory. This directly addresses the requirement that weights never leave GPU memory unencrypted. In-flight batching and quantization improve performance but do not provide encryption, and weight streaming moves weights to host memory, violating the policy.

Answer

Weight streaming from host memory

Answer

In-flight batching with paged KV cache

Answer

TensorRT-LLM's built-in weight-only quantization (INT8/INT4) with GPU-only weight residency

Question 9

A team is deploying a Llama-3-8B model with NVIDIA TensorRT-LLM on a single 24 GB A10G GPU. They configure the TensorRT-LLM build with a max batch size of 64 and the default paged KV cache. During a load test with 40 concurrent requests, the server begins returning CUDA out-of-memory errors after several minutes, even though the model weights alone fit comfortably in VRAM. Which configuration change most directly addresses the root cause?

Accepted Answer

Reduce the max batch size in the TensorRT-LLM build configuration. In TensorRT-LLM, the paged KV cache is provisioned according to the max batch size and maximum sequence length baked into the engine. With 40 concurrent sequences, the cache can grow beyond the VRAM left after weights, causing OOM. Reducing the max batch size directly caps that reservation and is the targeted fix.

Answer

Switch the deployment to a smaller context window by truncating prompts at the client

Answer

Enable FP8 quantization of the model weights in the TensorRT-LLM build

Answer

Increase the GPU memory utilization fraction in the TensorRT-LLM runtime

Question 10

A financial services company is deploying a Llama-2 13B model on NVIDIA Triton Inference Server. Compliance requires that the model must never be served with unvalidated weights. The MLOps team stores each retrained model in a new version directory under the model repository. They want Triton to automatically load the newest version as soon as it appears, but they also need the ability to immediately roll back to the previous version if a validation check fails. Which Triton model control mode should they configure?

Accepted Answer

Poll model control mode with a polling interval, so Triton detects new version directories and loads them automatically while keeping older versions available for rollback.. Poll mode is the Triton control mode that periodically scans the model repository and automatically loads newly discovered version directories. It preserves previously loaded versions, so an operator can route requests to an earlier version if validation fails. This matches the need for automatic adoption of a validated model plus a fast rollback path without writing a custom control plane.

Answer

None model control mode, because it disables all repository scanning and guarantees that only the first version loaded at startup is ever served.

Answer

Explicit model control mode with the model loaded via the repository API only after a validation webhook succeeds.

Answer

Ensemble model control mode, which schedules the new version behind the previous version and promotes it only after a health check passes.