NCP-GENL Model Deployment Practice Test 3 — 15 Questions

Question 1

A team is deploying a 13B-parameter LLM with NVIDIA TensorRT-LLM on a single A100 80GB GPU. They want to reduce GPU memory usage during inference without retraining the model, while keeping acceptable output quality. Which technique should they apply?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Enable INT8 weight-only quantization using TensorRT-LLM's quantization toolkit.. INT8 weight-only quantization is a post-training technique supported by TensorRT-LLM that reduces weight memory by roughly half while preserving output quality. It directly addresses the goal of lowering GPU memory usage for a large model on a single GPU without retraining. Other listed changes affect scheduling or activation memory rather than the core weight footprint.

Answer

Reduce the maximum sequence length to 512 tokens to lower activation memory.

Answer

Increase the KV cache block size to 128 tokens to reduce memory fragmentation.

Answer

Enable Tensor Parallelism across two GPUs to split the model.

Question 2

An engineer needs to expose an LLM served by NVIDIA Triton Inference Server to a web application over HTTP with token streaming. Which Triton feature should they enable?

Accepted Answer

Decoupled mode with streaming responses in the model backend.. Triton's decoupled mode lets a backend emit multiple responses for one request, which is the mechanism used for LLM token streaming. Configuring the model for decoupled transactions and using a streaming-capable client over HTTP or gRPC delivers tokens incrementally. Other Triton features like batching, ensembles, or instance groups improve throughput or composition but do not stream partial outputs.

Answer

Model ensembles combining preprocessing and postprocessing models.

Answer

Dynamic batching in the model configuration.

Answer

Instance groups with multiple GPU instances per model.

Question 3

A team is optimizing an NVIDIA TensorRT-LLM deployment of a 70B model on multiple GPUs. They want to reduce inter-GPU communication overhead and improve throughput. Which two techniques should they consider? (Choose two.)

Accepted Answer

Enable pipeline parallelism to split layers into stages.. Tensor parallelism benefits from NVLink's high bandwidth to reduce all-reduce overhead, while pipeline parallelism lowers communication frequency by staging layers across GPUs. Together they address the communication bottleneck for large multi-GPU models. KV cache block size, disabling in-flight batching, and FP32 precision do not reduce inter-GPU communication and can hurt throughput.

Answer

Disable in-flight batching to simplify scheduling.

Answer

Use FP32 precision for all weights to improve numerical stability.

Answer

Increase the KV cache block size to 256 tokens.

Question 4

A team is deploying a 70B-parameter LLM using NVIDIA Triton Inference Server with TensorRT-LLM backend on a node with four A100 80GB GPUs. They observe that during inference, only one GPU is utilized while the others remain idle. They have configured the model with tensor parallelism set to 1. What is the most likely cause of this underutilization?

Accepted Answer

Tensor parallelism is set to 1, which means the model is not partitioned across GPUs; it runs entirely on one GPU.. Tensor parallelism determines how a model is sharded across multiple GPUs. Setting it to 1 means no sharding, so the entire model runs on a single GPU, leaving others idle. To utilize all four GPUs for a single model instance, tensor parallelism should be set to 4. Other factors like instance groups or batch size do not override this fundamental partitioning setting.

Answer

The Triton model repository is not configured with the correct instance group for multi-GPU execution.

Answer

The model was compiled with a batch size that is too small to trigger multi-GPU execution.

Answer

The KV cache is not enabled, causing the model to fall back to single-GPU execution.

Question 5

An engineer is preparing an ensemble in NVIDIA Triton Inference Server that chains a Python preprocessing model with a TensorRT-LLM backend model. The preprocessing model must run on CPU while the LLM must run on GPU, and the ensemble must expose a single HTTP endpoint. Which configuration is required to make the ensemble execute correctly?

Accepted Answer

Define each step as a separate Triton model and reference them in a platform 'ensemble' model's ensemble_scheduling block. Triton ensembles use a model with platform 'ensemble' and an ensemble_scheduling block that lists each step with model name, version, and tensor mappings. This enables chaining a CPU Python preprocessing model with a GPU TensorRT-LLM model behind a single endpoint, which is exactly the requirement.

Answer

Write a custom backend that loads both models and orchestrates them in C++

Answer

Merge both models into a single TensorRT-LLM engine using a custom plugin

Answer

Use Triton's 'instance_group' to pin both models to the same GPU and rely on default routing

Question 6

An LLM service on NVIDIA Triton Inference Server experiences high time-to-first-token because the dynamic batcher waits for full batches. The team wants to reduce time-to-first-token while still benefiting from batching. Which adjustment is most appropriate?

Accepted Answer

Decrease max_queue_delay_microseconds to shorten batching wait time.. Time-to-first-token is driven by how long the dynamic batcher holds requests before dispatch. Lowering max_queue_delay_microseconds shortens that wait, so requests start sooner while still being batched with any available peers. Raising the delay, inflating preferred batch size, or disabling batching either increases latency or sacrifices throughput unnecessarily.

Answer

Set preferred_batch_size to a very large value.

Answer

Disable dynamic batching entirely for the model.

Answer

Increase max_queue_delay_microseconds to allow larger batches.

Question 7

An AI engineer is deploying a large language model using NVIDIA Triton Inference Server. They need to ensure that the server can handle multiple concurrent requests efficiently while maintaining low latency. Which Triton feature allows the server to dynamically batch incoming requests to maximize GPU utilization?

Accepted Answer

Dynamic batching. Dynamic batching is a Triton feature that aggregates concurrent inference requests into larger batches on the server side, improving GPU utilization and throughput while managing latency. It is the standard mechanism for handling high concurrency efficiently. Model versioning, instance groups, and ensemble scheduling serve different purposes and do not provide dynamic request batching.

Answer

Instance groups

Answer

Model versioning

Answer

Ensemble scheduling

Question 8

A platform team is deploying a 70B-parameter LLM with NVIDIA TensorRT-LLM across four 80 GB H100 GPUs and needs to serve long-context requests efficiently. They are deciding how to combine parallelism and memory techniques in the build and runtime configuration. (Choose two.)

Accepted Answer

Use tensor parallelism across the four GPUs so each GPU holds a shard of every layer's weights. Tensor parallelism shards each layer across the four H100s so a 70B model fits and serves with low latency over NVLink, while a properly sized paged KV cache prevents memory over-reservation for long contexts. Together they address both weight distribution and the dominant dynamic memory consumer in long-context LLM serving.

Answer

Reduce the number of GPUs to one and enable FP8 quantization to fit the model on a single device

Answer

Rely on CUDA Unified Memory to transparently page weights and KV cache between host and device

Answer

Enable pipeline parallelism with a single micro-batch to minimize inter-GPU communication

Question 9

A team is deploying a large language model on NVIDIA Triton Inference Server with TensorRT-LLM backend. They want to reduce GPU memory consumption to fit a larger model on a single GPU without significantly degrading output quality. Which two techniques should they use? (Choose two.)

Accepted Answer

Enable INT8 weight-only quantization. INT8 weight-only quantization reduces weight memory footprint with minimal quality loss, and paged KV cache optimizes memory usage during generation by reducing fragmentation. Together, they enable larger models to fit on a single GPU. FP32 increases memory, larger batch sizes demand more memory, and tensor parallelism requires multiple GPUs, so they do not meet the goal.

Answer

Increase the maximum batch size

Answer

Use FP32 precision for all computations

Answer

Use tensor parallelism across multiple GPUs

Question 10

An inference team runs a TensorRT-LLM model behind NVIDIA Triton Inference Server with the tensorrtllm_backend. They notice that when clients send requests with widely varying prompt lengths, throughput drops and some requests wait much longer than others. They want Triton to group compatible requests dynamically to improve GPU utilization without changing the model build. Which Triton feature should they configure?

Accepted Answer

Dynamic batching with a preferred batch size and max queue delay in the model's config.pbtxt. Triton's dynamic batching groups independent requests arriving within max_queue_delay into a single batch, using preferred_batch_size to target efficient batch shapes. This improves GPU utilization under varying prompt lengths without rebuilding the TensorRT-LLM engine, directly addressing the throughput and latency variance described.

Answer

Model warmup with sample inputs to pre-allocate memory at load time

Answer

Rate limiting with a max queue size to reject excess requests

Answer

Sequence batching with a sequence ID to maintain state across requests

Question 11

A media analytics company runs a TensorRT-LLM optimized GPT-J model on a single NVIDIA A100 80GB GPU using NVIDIA Triton Inference Server. During peak hours, request concurrency rises sharply and the team observes that the GPU is idle for long periods while waiting on host-side tokenization and detokenization. Profiling shows that CPU preprocessing and postprocessing dominate request latency. The team wants to reduce end-to-end latency without changing model weights or adding GPUs. Which Triton feature should they use?

Accepted Answer

The TensorRT-LLM backend's in-flight batching and built-in tokenizer, which perform tokenization and detokenization inside the backend and keep the GPU busy with continuous batching.. The TensorRT-LLM Triton backend provides an integrated tokenizer and in-flight batching, which eliminate the host-side tokenization and detokenization stalls observed in the profile. In-flight batching also continuously admits new requests into the running batch, improving GPU utilization and reducing end-to-end latency without changing model weights or adding GPUs.

Answer

Dynamic batching, because it groups requests on the server and reduces the number of model executions.

Answer

Sequence batching with a custom scheduler, because it keeps tokenization on the host and overlaps it with GPU compute across requests.

Answer

A Triton ensemble that places a Python model for tokenization before the TensorRT-LLM model and a Python model for detokenization after it.

Question 12

An engineer is deploying a LLM using NVIDIA Triton Inference Server with the TensorRT-LLM backend. They need to ensure that the model can handle a sudden surge in requests without increasing latency beyond a specified threshold. They have configured the model with a maximum batch size of 32 and dynamic batching with a preferred batch size of 16. However, during peak load, latency spikes are observed. Which Triton configuration parameter should they adjust to control the maximum time a request waits in the dynamic batching queue before being processed?

Accepted Answer

max_queue_delay_microseconds. The max_queue_delay_microseconds parameter in Triton's dynamic batching configuration specifies the maximum time a request can wait in the queue before being processed. Lowering it reduces latency spikes under surge conditions by preventing requests from waiting too long for a full batch. Other parameters like max_batch_size and preferred_batch_size affect batching but not the wait timeout.

Answer

max_batch_size

Answer

instance_group

Answer

preferred_batch_size

Question 13

A company needs to deploy a generative AI model that will serve prompts containing regulated customer data. Security policy requires that all inference stays on-premises, that the model be quantized to fit existing GPUs, and that no external network calls occur at runtime. Which deployment approach should the engineer choose?

Accepted Answer

Deploy the model with NVIDIA NIM microservices in an air-gapped environment using locally hosted containers from NVIDIA NGC. NVIDIA NIM microservices are distributed as containers that can be deployed on-premises or in air-gapped environments, support quantized model variants, and do not require external network calls at runtime. This satisfies the on-premises, quantized, and no-external-calls requirements simultaneously, unlike hosted or cloud-based options.

Answer

Run the model locally but allow it to fetch updated weights from a vendor endpoint at startup

Answer

Deploy the model in a public cloud region with a private VPC and customer-managed encryption keys

Answer

Use a public hosted API for the model and rely on TLS encryption for data in transit

Question 14

A healthcare startup is deploying a Mistral 7B model for internal clinical note summarization. They need to serve the model with NVIDIA Triton Inference Server and want to minimize GPU memory footprint during inference. The team plans to use TensorRT-LLM and is choosing a numerical precision for the engine. Which precision should they select to reduce memory usage while maintaining acceptable accuracy for summarization?

Accepted Answer

INT8 or FP8 quantization, because it reduces weight and activation precision further than FP16 while maintaining acceptable accuracy for text summarization.. INT8 or FP8 quantization reduces the bits per weight and activation below FP16, cutting GPU memory usage while keeping accuracy acceptable for summarization. TensorRT-LLM supports quantized engines with calibration or scaling factors, making it the appropriate precision choice when memory footprint is the primary constraint and the task tolerates small accuracy loss.

Answer

FP16, because it halves memory compared to FP32 and is widely supported on NVIDIA data center GPUs.

Answer

FP32, because it uses the least GPU memory and is the default for TensorRT-LLM engines.

Answer

TF32, because it is a Tensor Core mode that automatically compresses weights to one byte per parameter.

Question 15

A team is deploying a 70B parameter LLM with NVIDIA Triton Inference Server across four NVIDIA H100 GPUs. They are using TensorRT-LLM and need to fit the model within the combined GPU memory while maintaining high throughput. Which two techniques should they use? (Choose two.)

Accepted Answer

Pipeline parallelism, which assigns entire transformer layers to different GPUs and passes activations between stages.. Tensor parallelism and pipeline parallelism are the two model-parallel techniques that split a 70B model across multiple GPUs, allowing the weights to fit in aggregate memory on four H100s. Tensor parallelism shards layers and attention heads, while pipeline parallelism assigns layer groups to stages; together they enable large-model deployment with high throughput.

Answer

Increasing the maximum batch size to the largest value the backend accepts, so memory is fully utilized.

Answer

Data parallelism, which replicates the full model on every GPU and splits incoming requests across replicas.

Answer

Enabling FP32 precision for all weights and activations to maximize numerical stability across GPUs.