NCP-GENL Model Deployment — Timed 20 Questions

Question 1

Refer to the exhibit. An engineer configures Triton for dynamic batching. If requests arrive at 2ms intervals and the current queue is empty, what is the expected batching behavior?

Accepted Answer

It will wait 5ms and execute whatever requests are in the queue.. With a max delay of 5000 microseconds (5ms) and arrival intervals of 2ms, Triton will wait to collect requests until the delay limit is reached or the preferred batch size is met. Since the preferred batch size is 4, it will wait for at least four requests to arrive. The time taken to receive four requests at 2ms intervals is 6ms, which exceeds the max delay. Therefore, it will trigger a batch after 5ms.

Answer

It will execute a batch of 1 immediately upon arrival.

Answer

It will wait for exactly 8 requests before starting inference.

Answer

It will error out because the interval is smaller than the delay.

Question 2

Which THREE techniques are effective for managing memory in high-concurrency LLM deployments on NVIDIA GPUs? (Select THREE)

Accepted Answer

Implementing PagedAttention to reduce KV cache fragmentation.. Managing memory for LLMs is critical because they occupy significant VRAM. Techniques like PagedAttention effectively manage the KV cache, preventing fragmentation. Model parallelism (tensor or pipeline) allows splitting models across multiple GPUs when they exceed single-device capacity. Quantization directly reduces the memory footprint of weights, enabling higher concurrency. Collectively, these ensure memory is utilized efficiently without sacrificing performance or falling into fragmented memory states.

Answer

Increasing the batch size to the maximum theoretical hardware limit.

Answer

Disabling the CUDA graph capture to save VRAM overhead.

Question 3

When migrating an LLM deployment to a multi-GPU setup, what is the significance of 'Tensor Parallelism'?

Accepted Answer

It partitions model layers across GPUs for memory efficiency.. Tensor Parallelism splits the individual matrix operations of a single model layer across multiple GPUs. This is necessary when a model is too large to fit into the memory of a single GPU or when high-speed generation requires the combined compute power of multiple devices. By parallelizing the weights and computations, the model can execute at scale without exceeding the physical constraints of a single GPU's VRAM.

Answer

It allows multiple different models to run on one GPU.

Answer

It replicates the entire model on every available GPU.

Answer

It accelerates model training by caching gradients.

Question 4

An engineering team is deploying a Transformer-based model using NVIDIA TensorRT-LLM on an H100 GPU cluster. Which TWO factors are most critical to consider when configuring the In-flight Batching (IFB) feature for optimal performance?

Accepted Answer

The total VRAM allocated for the KV cache and the maximum number of concurrent sequences.. In-flight Batching allows new requests to join an existing batch as soon as slots become available, rather than waiting for the entire batch to complete. This is vital for autoregressive decoding where output lengths vary significantly. Understanding KV cache memory management and the maximum number of concurrent sequences is essential to prevent OOM errors and ensure that the H100's high compute density is fully utilized during inference.

Answer

The clock frequency settings for the GPU's high-bandwidth memory (HBM).

Answer

The specific activation function used within the model's feed-forward layers.

Answer

The network latency between the load balancer and the inference server instances.

Question 5

Refer to the exhibit. An engineer observes that the LLM deployment exhibits high latency and inefficient GPU utilization under moderate load. Based on the provided Triton configuration, what is the most likely cause?

Accepted Answer

The dynamic_batching parameter is incorrectly applied to the TensorRT-LLM backend.. The exhibit shows that dynamic batching is enabled for a tensorrtllm backend, which is fundamentally incompatible with the In-flight Batching requirements of modern LLM serving. TensorRT-LLM backends should manage batching through the engine itself rather than the Triton dynamic batcher. By using the dynamic batcher, the server introduces a queuing mechanism that interferes with the model's native ability to process tokens as they are generated.

Answer

The instance count is set to 1, which prevents multi-GPU parallelism.

Answer

The preferred_batch_size values are too high, causing memory allocation errors.

Answer

The backend name is case-sensitive and should be capitalized as TENSORRTLLM.

Question 6

A healthcare company runs an NVIDIA Triton Inference Server hosting a clinical-summarization LLM. Compliance requires that every generated summary be fully reproducible for audit: the same prompt must always yield byte-identical output, and token-level probabilities must be retrievable later. The team currently sends requests with default sampling parameters and stores only the final text. Which change to the deployment configuration best satisfies the audit requirement?

Accepted Answer

Switch the model to greedy decoding by setting beam_width to 1 and top_k to 1, and set return_log_probs to true in the Triton request so token-level probabilities are returned with each response.. Auditable generative AI requires deterministic token selection plus retention of the probability distribution used at each step. Greedy decoding with beam_width 1 and top_k 1 eliminates sampling randomness, guaranteeing that the same prompt maps to the same output. Setting return_log_probs to true makes Triton include per-token log probabilities in the response, giving auditors the quantitative evidence they need alongside the text.

Answer

Enable Triton's dynamic batching and set max_batch_size to 1, then log the raw prompt and final response text to an immutable object store.

Answer

Keep the default sampling parameters but lower temperature to 0.0 and increase top_p to 1.0 so the model becomes deterministic.

Answer

Deploy a second Triton instance with a fixed random seed and route audit requests to it, while production traffic continues using default sampling.

Question 7

A production LLM service on NVIDIA Triton Inference Server must handle bursty traffic while maintaining low latency. The team wants to cap the number of concurrent requests per model instance and queue excess requests. Which Triton configuration parameter should they tune?

Accepted Answer

Model instance concurrency limit via the backend's max concurrency setting.. Capping concurrent requests per model instance is achieved through the backend's concurrency limit, which queues excess requests rather than admitting them immediately. This protects GPU resources and keeps latency predictable under bursts. Parameters like max_batch_size, instance_group count, and queue delay affect batching and parallelism but do not enforce a per-instance concurrency ceiling.

Answer

instance_group count in the model configuration.

Answer

max_batch_size in the model configuration.

Answer

dynamic_batching max_queue_delay_microseconds.

Question 8

A financial services company is deploying a 13B-parameter LLM using NVIDIA TensorRT-LLM. Their security policy requires that model weights never leave the GPU memory unencrypted and that inference requests are processed with the lowest possible overhead. They have a single NVIDIA H100 GPU with 80 GB memory. Which TensorRT-LLM feature should they enable to meet these requirements while maximizing throughput?

Accepted Answer

TensorRT-LLM with NVIDIA confidential computing enabled. Confidential computing on NVIDIA H100 GPUs provides hardware-based encryption of data in use, ensuring model weights are protected while resident in GPU memory. This directly addresses the requirement that weights never leave GPU memory unencrypted. In-flight batching and quantization improve performance but do not provide encryption, and weight streaming moves weights to host memory, violating the policy.

Answer

Weight streaming from host memory

Answer

In-flight batching with paged KV cache

Answer

TensorRT-LLM's built-in weight-only quantization (INT8/INT4) with GPU-only weight residency

Question 9

A team is deploying a Llama-3-8B model with NVIDIA TensorRT-LLM on a single 24 GB A10G GPU. They configure the TensorRT-LLM build with a max batch size of 64 and the default paged KV cache. During a load test with 40 concurrent requests, the server begins returning CUDA out-of-memory errors after several minutes, even though the model weights alone fit comfortably in VRAM. Which configuration change most directly addresses the root cause?

Accepted Answer

Reduce the max batch size in the TensorRT-LLM build configuration. In TensorRT-LLM, the paged KV cache is provisioned according to the max batch size and maximum sequence length baked into the engine. With 40 concurrent sequences, the cache can grow beyond the VRAM left after weights, causing OOM. Reducing the max batch size directly caps that reservation and is the targeted fix.

Answer

Switch the deployment to a smaller context window by truncating prompts at the client

Answer

Enable FP8 quantization of the model weights in the TensorRT-LLM build

Answer

Increase the GPU memory utilization fraction in the TensorRT-LLM runtime

Question 10

A financial services company is deploying a Llama-2 13B model on NVIDIA Triton Inference Server. Compliance requires that the model must never be served with unvalidated weights. The MLOps team stores each retrained model in a new version directory under the model repository. They want Triton to automatically load the newest version as soon as it appears, but they also need the ability to immediately roll back to the previous version if a validation check fails. Which Triton model control mode should they configure?

Accepted Answer

Poll model control mode with a polling interval, so Triton detects new version directories and loads them automatically while keeping older versions available for rollback.. Poll mode is the Triton control mode that periodically scans the model repository and automatically loads newly discovered version directories. It preserves previously loaded versions, so an operator can route requests to an earlier version if validation fails. This matches the need for automatic adoption of a validated model plus a fast rollback path without writing a custom control plane.

Answer

None model control mode, because it disables all repository scanning and guarantees that only the first version loaded at startup is ever served.

Answer

Explicit model control mode with the model loaded via the repository API only after a validation webhook succeeds.

Answer

Ensemble model control mode, which schedules the new version behind the previous version and promotes it only after a health check passes.

Question 11

An e-commerce company deploys a customer support chatbot backed by a TensorRT-LLM model on NVIDIA Triton Inference Server. The model uses a KV cache and the team wants to support multiple concurrent conversations while keeping per-request latency predictable. They configure the Triton TensorRT-LLM backend with a fixed max_batch_size and a limited KV cache. During testing, they observe that some requests are rejected with out-of-memory errors when many long conversations occur simultaneously. Which configuration change best addresses the issue without reducing the maximum batch size?

Accepted Answer

Enable the backend's KV cache reuse and set a maximum sequence length per request, so long conversations cannot consume unbounded cache memory.. Enabling KV cache reuse and enforcing a maximum sequence length per request directly bound the memory consumed by long conversations and avoid duplicating cached tensors. This reduces out-of-memory rejections under concurrent long chats while keeping the configured maximum batch size intact, preserving throughput and latency characteristics.

Answer

Reduce the maximum batch size to one and rely on client-side queuing to serialize conversations.

Answer

Increase the max_num_tokens parameter so the backend can allocate a larger KV cache for longer sequences.

Answer

Switch from in-flight batching to static batching, because static batching reserves KV cache only for the exact batch size in use.

Question 12

A financial services company is deploying a 13B-parameter LLM on a single NVIDIA A100 80GB GPU using NVIDIA TensorRT-LLM. During peak traffic, they observe that throughput plateaus and GPU memory is nearly exhausted, but latency remains acceptable. They want to increase concurrent request handling without adding another GPU. Which configuration change should they make?

Accepted Answer

Enable in-flight batching (continuous batching) in the TensorRT-LLM runtime.. In-flight batching lets the runtime insert new requests into the batch as soon as individual sequences complete, keeping the GPU busy and increasing concurrent throughput without additional memory. It directly addresses the plateau caused by static batching, where the whole batch waits for the longest sequence. Other options either increase memory pressure or degrade model quality.

Answer

Increase the TensorRT-LLM max_batch_size to 256 and rebuild the engine.

Answer

Enable FP32 precision for all matrix multiplications to improve numerical stability.

Answer

Reduce the KV cache size by lowering the max sequence length to 128 tokens.

Question 13

An enterprise is deploying a large language model on NVIDIA Triton Inference Server. Which deployment strategy minimizes latency for requests that require high-throughput batching while maintaining consistent hardware utilization?

Accepted Answer

Enable Dynamic Batching in the Triton model configuration file.. Dynamic Batching is the optimal strategy for Triton Inference Server in production environments. It groups individual inference requests arriving within a short time window into a single batch, allowing the GPU to process them in parallel. This maximizes throughput by fully saturating CUDA cores, reducing the overhead of kernel launches, and ensuring that hardware utilization remains high even under variable traffic loads, effectively balancing latency and overall system capacity.

Answer

Implement static batching with a fixed size of 1.

Answer

Disable all batching features to process requests serially.

Answer

Offload all batching logic to the client-side application layer.

Question 14

Refer to the exhibit. An engineer observes that memory utilization spikes during inference, causing OOM errors. Given the configuration, what is the most likely cause of the failure?

Accepted Answer

The instance group count of 2 causes redundant loading of weights, exceeding VRAM.. The instance group configuration defines two concurrent instances on the same GPU. Each instance attempts to load a separate copy of the model weights into the GPU memory. If the model size is large, doubling the instances exceeds the available VRAM capacity. This configuration is a common mistake when deploying LLMs where model footprint is significant relative to total available device memory.

Answer

The count of 2 causes the GPU to oversubscribe its thermal limits.

Answer

Triton requires KIND_CPU for concurrent instance execution.

Answer

The gpus index [0] is invalid for multi-instance deployment.

Question 15

Which TWO factors should be prioritized when selecting a quantization strategy for deploying a large language model on constrained edge hardware? (Select TWO)

Accepted Answer

Reducing the memory footprint of the model weights.. Selecting a quantization strategy requires balancing precision loss against hardware performance gains. When deploying to edge devices, memory bandwidth and storage capacity are typically the primary bottlenecks. By reducing precision from FP16 or FP32 to INT8 or FP8, you directly decrease the model's memory footprint and increase the number of operations per clock cycle, which is essential for maintaining acceptable real-time inference speeds on resource-limited hardware.

Answer

Maximizing the model training loss to improve generalization.

Answer

Increasing the number of neural network layers in the architecture.

Answer

Replacing the Transformer architecture with a linear regression model.

Question 16

When deploying an LLM using NVIDIA TensorRT-LLM, what is the primary benefit of pre-compiling the model into a TensorRT engine?

Accepted Answer

It enables layer fusion and kernel selection for the target GPU.. Pre-compiling with TensorRT-LLM allows for layer fusion, kernel auto-tuning, and memory optimization tailored specifically to the target GPU architecture. By performing these heavy optimizations offline, the inference engine can execute at peak performance immediately upon loading. This eliminates the runtime overhead associated with graph compilation or dynamic graph execution, ensuring minimal latency and optimal resource utilization from the very first inference request.

Answer

It allows the model to run on any generic CPU architecture.

Answer

It eliminates the need for any GPU memory during inference.

Answer

It automatically scales the model across multiple distributed nodes.

Question 17

Which component in the NVIDIA AI Enterprise stack is primarily responsible for serving multiple models, managing model versions, and providing metrics for monitoring model health?

Accepted Answer

NVIDIA Triton Inference Server. NVIDIA Triton Inference Server is the standard tool for model serving in production. It provides a unified API for various frameworks, supports concurrent model execution, and performs health checks. It is designed to handle model versioning and provide detailed telemetry data, which is essential for maintaining reliable and scalable AI deployments in enterprise production pipelines.

Answer

CUDA Toolkit

Answer

NVIDIA NeMo

Answer

NVIDIA DALI

Question 18

What is the primary role of a Model Repository in the NVIDIA Triton Inference Server architecture?

Accepted Answer

To act as a centralized filesystem for serving multiple models.. The Model Repository acts as a centralized storage location for all deployed models, their configurations, and their versions. Triton monitors this directory to detect when new versions are added or updated, allowing for seamless model updates without restarting the server. This design supports robust MLOps practices by decoupling the model storage from the inference engine runtime, ensuring that deployments remain manageable and version-controlled.

Answer

To store training datasets for real-time model retraining.

Answer

To handle network traffic load balancing between server nodes.

Answer

To compile the model into an optimized executable format.

Question 19

In the context of LLM deployment, why is it recommended to use a dedicated inference server like Triton rather than a basic Flask or FastAPI wrapper?

Accepted Answer

Triton manages concurrent model execution and hardware utilization.. While basic web frameworks are easy to implement, they lack the sophisticated features required for production AI. Triton offers hardware-optimized concurrency, automatic dynamic batching, support for multiple backends (TensorRT, PyTorch, ONNX), and advanced metrics. These features are critical for handling the high computational demands and low-latency requirements of LLMs, providing better scalability and reliability compared to custom-built wrappers that do not natively manage GPU streams and queues.

Answer

Flask is incompatible with the NVIDIA CUDA drivers.

Answer

Custom wrappers require more memory than Triton servers.

Answer

Triton is the only way to support REST API endpoints.

Question 20

Refer to the exhibit. What is the most likely cause of the failure based on the log entries?

Accepted Answer

The KV cache size is exceeding the allocated GPU memory limit.. The logs indicate high processing latency followed by KV cache eviction and finally an OOM error. This sequence suggests that the system is running out of VRAM due to the growing KV cache during long-context generation. As the context length increases, the memory required for the KV cache exceeds the available capacity, forcing evictions, slowing down processing, and ultimately triggering an Out of Memory crash.

Answer

The model weights are corrupted in the repository.

Answer

Network latency is causing the client-side timeout.

Answer

The GPU driver version is incompatible with TensorRT.