NCP-GENL Model Optimization Practice Test 4 — 15 Questions

Question 1

An engineer is using TensorRT-LLM to serve a chatbot model. They observe that the time to first token (TTFT) is high, but subsequent tokens are generated quickly. Which optimization should they prioritize to reduce TTFT?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Optimize the prefill phase with kernel fusion and faster attention.. Time to first token is primarily determined by the prefill phase, which processes the input prompt. Optimizing this phase with fused kernels, efficient attention mechanisms, and reduced memory latency directly reduces TTFT. TensorRT-LLM includes highly optimized prefill kernels and supports FlashAttention, which significantly speeds up the prefill computation. Other options target throughput or cache management, not TTFT.

Answer

Enable in-flight batching to increase GPU utilization.

Answer

Increase the KV cache size to avoid evictions.

Answer

Use a larger batch size for the prefill phase.

Question 2

An engineer is using NVIDIA TensorRT to optimize a Transformer model for inference on an NVIDIA A100 GPU. They want to maximize throughput while ensuring that the model runs correctly with varying input sequence lengths. Which TensorRT feature should they configure to allow the engine to handle different input shapes at runtime?

Accepted Answer

Dynamic shapes with optimization profiles. Dynamic shapes with optimization profiles enable a single TensorRT engine to handle inputs of different sizes by defining shape ranges. This allows the engine to optimize for the actual input shape at runtime, which is crucial for Transformer models with variable sequence lengths. Static shapes, multiple engines, or INT8 calibration do not provide this flexibility.

Answer

Static shapes with fixed dimensions

Answer

INT8 calibration with a representative dataset

Answer

Multiple engines for each possible input length

Question 3

A team is deploying a 13B-parameter chatbot on a single NVIDIA A10G GPU (24 GB VRAM). The model's weights are stored in FP16, and the runtime runs out of memory during KV cache allocation under concurrent user sessions. They must keep answer quality essentially unchanged while maximizing concurrent sessions. Which optimization should they apply first?

Accepted Answer

Apply INT8 weight-only quantization to the linear layers with NVIDIA TensorRT-LLM, keeping activations in FP16.. The bottleneck is persistent weight memory (FP16 weights dominate VRAM), which leaves too little room for paged KV cache blocks under concurrency. Weight-only INT8 quantization in TensorRT-LLM roughly halves the linear-layer weight footprint while keeping activations in FP16, preserving quality. That reclaimed VRAM goes directly to KV cache, raising the number of simultaneous sessions without retraining or a new GPU.

Answer

Convert the checkpoint to TF32 and rebuild the TensorRT-LLM engine with builder optimization level 5.

Answer

Increase the paged KV cache block size from 16 to 128 tokens per block and disable block reuse.

Answer

Enable CUDA graph capture for the decoder and set the maximum batch size to the peak observed concurrency.

Question 4

A team is deploying a large language model using NVIDIA TensorRT-LLM on a multi-GPU node with NVLink. They want to minimize inter-GPU communication overhead during inference. Which parallelism strategy should they use to achieve this?

Accepted Answer

Tensor parallelism (TP). Tensor parallelism splits layers across GPUs and uses all-reduce for activations. On a system with NVLink, the high bandwidth and low latency of NVLink make TP efficient, minimizing communication overhead. Pipeline parallelism introduces bubbles, data parallelism replicates the model, and expert parallelism is for MoE models and incurs all-to-all communication.

Answer

Pipeline parallelism (PP)

Answer

Expert parallelism (EP) in a Mixture-of-Experts model

Answer

Data parallelism (DP)

Question 5

An engineer is deploying a 13B-parameter LLM with TensorRT-LLM on a single NVIDIA A100 40GB GPU. The FP16 engine requires 26GB for weights, but during generation the KV cache grows beyond remaining memory, causing out-of-memory errors. The team wants to maximize concurrent requests without retraining. Which optimization should they apply first?

Accepted Answer

Reduce the KV cache memory footprint by quantizing the KV cache to INT8 while keeping model weights in FP16.. The KV cache grows linearly with sequence length and batch size, and it is the dominant memory consumer during LLM generation after weights are loaded. Quantizing only the KV cache to INT8 halves its footprint without retraining and without changing weight precision, directly relieving the OOM. This keeps the model on the existing A100 while allowing more concurrent requests, which matches the stated goal.

Answer

Increase the batch size and sequence length limits so the scheduler can pack more requests into the same memory.

Answer

Enable tensor parallelism across two A100 GPUs to split both weights and KV cache across devices.

Answer

Convert the model weights from FP16 to FP8 using post-training quantization to free memory for the KV cache.

Question 6

A team is building a TensorRT-LLM engine for a 7B model that must serve both single-turn short prompts and long multi-turn conversations with a shared system prompt. They want to maximize reuse of computation across requests without changing model weights. Which TWO techniques should they enable? (Choose two.)

Accepted Answer

Enable paged KV cache with block reuse so freed sequence blocks return to the pool for other requests.. Sharing a system prompt and multi-turn history means many requests share long identical prefixes, and mixed short and long prompts create variable cache pressure. KV cache reuse skips recomputing cached prefixes, while the paged KV cache with block reuse keeps memory defragmented and available to new sequences. Together they cut redundant prefill and raise concurrency without touching model weights.

Answer

Apply INT4 weight-only quantization to the attention projection matrices to shrink the cache footprint.

Answer

Set the builder to use strongly typed engines so the plugin graph is fully deterministic.

Answer

Increase the beam width so multiple candidate continuations are evaluated per request.

Question 7

A team is preparing a Llama-based chatbot for production and wants to reduce GPU memory and latency without retraining. They decide to apply post-training quantization. Which TensorRT-LLM workflow correctly produces an INT8 or FP8 quantized engine from an existing FP16 checkpoint?

Accepted Answer

Run the quantization toolkit to produce a quantized checkpoint, then build the TensorRT-LLM engine with the appropriate quantization flags.. Post-training quantization in TensorRT-LLM is a two-stage process: first produce a quantized checkpoint with calibration or scaling data, then build the engine with the matching quantization flags so the builder selects INT8 or FP8 kernels. This avoids retraining and yields memory and latency improvements. The other options either attempt unsupported post-build modification, rely on runtime precision switching that does not exist, or bypass the required engine compilation step.

Answer

Build the FP16 engine first, then apply a post-build conversion script that rewrites the engine plan to INT8 weights.

Answer

Convert the checkpoint to ONNX with INT8 operators and load it directly into the TensorRT-LLM runtime without an engine build.

Answer

Enable automatic mixed precision in the runtime and let the inference server choose INT8 kernels dynamically at request time.

Question 8

A developer wants to reduce the disk and memory footprint of a fine-tuned 70B model before serving it with TensorRT-LLM, and is willing to accept a small, measurable quality drop that they will validate with an evaluation harness. Which approach best matches that requirement?

Accepted Answer

Perform post-training quantization to INT4 or INT8 weights and validate with perplexity and task metrics.. The requirement is a smaller weight footprint with an accepted, validated quality trade-off and no retraining. Post-training quantization of the fine-tuned checkpoint to INT4 or INT8 delivers that reduction immediately and pairs naturally with an evaluation harness to confirm the quality delta stays within tolerance. Retraining-based methods exceed the stated scope.

Answer

Retrain the model from scratch using quantization-aware training with fake-quant nodes in the forward pass.

Answer

Convert the checkpoint to FP8 and rely on the runtime to upcast to FP16 for every matmul.

Answer

Increase the tensor parallelism degree so each GPU holds a smaller shard of the FP16 weights.

Question 9

An engineer is using NVIDIA TensorRT-LLM's in-flight batching to serve a mix of short and very long prompts. They observe that GPU utilization drops and latency for short requests spikes whenever a long prompt is admitted. Which mechanism should they tune to prevent long sequences from monopolizing the batch?

Accepted Answer

Adjust the maximum number of tokens processed per iteration and the KV cache block allocation policy to limit how much context a single request can consume.. In-flight batching improves throughput by mixing prefill and decode work, but a very long prompt can consume the entire per-iteration token budget, starving shorter sequences. Limiting tokens per iteration and controlling KV cache block allocation prevents any single request from monopolizing the batch. This restores GPU utilization and keeps short-request latency stable while still allowing long prompts to complete progressively.

Answer

Enable tensor parallelism so the long prompt's attention computation is split across multiple GPUs.

Answer

Switch from in-flight batching to a static batching scheme that groups requests by similar prompt length.

Answer

Increase the maximum batch size so more short requests can be admitted alongside the long prompt.

Question 10

An engineer is using TensorRT-LLM to serve a chatbot model. They observe that the time to first token (TTFT) is high, but subsequent tokens are generated quickly. Which optimization should they prioritize to reduce TTFT?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Set the maximum input length and maximum sequence length to accommodate 8K tokens and size the KV cache pool accordingly.. The failure is a capacity limit tied to the engine's configured maximum input and sequence lengths and the KV cache pool sized for them. Raising those limits to cover 8K tokens and sizing the pool to match lets long prompts reserve the blocks they need, while paged allocation means short requests still consume only a few blocks, so a single worst-case build shape is unnecessary.

Answer

Enable INT8 KV cache quantization so each token's cached keys and values consume fewer bytes.

Answer

Enable chunked context prefill so long prompts are processed in multiple smaller prefill passes.

Answer

Reduce the maximum batch size to 1 so all KV cache blocks are available to a single long request.

Question 11

An engineer is optimizing an LLM for inference with NVIDIA TensorRT-LLM and wants to reduce both memory footprint and latency without retraining the model. Which two techniques should they apply? (Choose two.)

Accepted Answer

Quantize model weights to INT8 or FP8 using post-training quantization with calibration.. Post-training quantization reduces weight memory and enables faster low-precision kernels, while paged KV cache manages generation-time memory efficiently and improves batching. Together they lower footprint and latency without retraining. The rejected options either require retraining, increase model size, or break the architecture, so they do not fit the no-retraining constraint or the optimization goals.

Answer

Disable attention and rely solely on feed-forward layers to speed up decoding.

Answer

Increase the number of transformer layers to improve parallelism across the GPU.

Answer

Retrain the model with quantization-aware training to learn optimal scaling factors.

Question 12

A team has built a TensorRT-LLM engine for a 70B model on four NVIDIA H100 GPUs using tensor parallelism. They now need to serve the same model on a single H100 for a development environment, accepting higher latency. What is the most appropriate approach?

Accepted Answer

Rebuild the engine with tensor parallelism set to one and apply weight quantization so the model fits in a single GPU's memory.. Tensor parallelism is a build-time property, so changing the number of GPUs requires rebuilding the engine with tensor parallel size one. Because a 70B model in FP16 exceeds a single H100's memory, weight quantization is also needed to fit. The other options either attempt to reuse an incompatible engine, misapply parallelism concepts, or increase precision, none of which yield a working single-GPU deployment.

Answer

Use pipeline parallelism with a single stage to split the model across the GPU's SMs automatically.

Answer

Load the existing four-GPU engine on a single H100 and set the runtime to use only one device.

Answer

Convert the engine to a TensorRT plan file with FP32 precision so it can use unified memory on the H100.

Question 13

A developer has a fine-tuned Llama-family model in Hugging Face format and wants to run it with NVIDIA TensorRT-LLM on an H100. Which artifact must be produced before the runtime can execute the model?

Accepted Answer

A TensorRT engine built from the model's network definition for the target GPU architecture.. TensorRT-LLM executes a serialized TensorRT engine that is compiled for a specific GPU architecture, precision, and parallelism configuration. After converting the Hugging Face checkpoint into the TensorRT-LLM checkpoint format, the developer must build the engine for the H100. Only that engine can be loaded by the runtime, so it is the required artifact before any inference can occur.

Answer

An ONNX graph exported with dynamic axes and a matching runtime configuration JSON.

Answer

A PyTorch TorchScript trace of the full forward pass saved as a .pt file.

Answer

A quantized GGUF file generated with a community conversion script.

Question 14

An engineer is tuning a TensorRT-LLM deployment of a 7B model for a latency-sensitive API. Profiling shows that time per output token is higher than expected and that many small kernels run back to back with gaps between them. Which TWO changes are most likely to reduce the per-token latency by cutting kernel launch overhead and redundant memory traffic? (Choose two.)

Accepted Answer

Enable CUDA Graphs so the decoding iteration is captured and replayed as a single graph launch.. The profile shows many short kernels with idle gaps, which is the classic signature of launch overhead and unfused elementwise work. CUDA Graphs collapse the decode iteration into a single replayable launch, and fused attention and GEMM plugins merge operations that would otherwise round-trip intermediates through HBM. Together they reduce both launch count and memory traffic, which are the two costs identified in the profile.

Answer

Enable a debug synchronization after every kernel so profiling timings are accurate.

Answer

Raise the maximum sequence length in the engine build to accommodate the longest possible prompt.

Answer

Switch the model weights to FP32 to avoid any dequantization kernels in the decode path.

Question 15

A team must serve a 70B model on a single 80 GB GPU for an internal assistant with modest concurrency. Full FP16 weights will not fit alongside the KV cache for the target context length. They want to keep accuracy loss minimal and are willing to spend additional build time. Which approach best fits these constraints?

Accepted Answer

Use INT4 weight-only quantization with a group size and calibration set chosen for the model.. The binding constraint is fitting a 70B model plus KV cache on one 80 GB GPU while preserving accuracy. Weight-only INT4 quantization cuts weight memory roughly fourfold while keeping activations and cache at higher precision, and group-wise scales with a representative calibration set limit degradation. The team's willingness to accept longer build time matches the calibration and engine-build cost this method requires.

Answer

Split the model with tensor parallelism across two GPUs even though only one is available.

Answer

Apply FP8 quantization to both weights and activations and rebuild the engine.

Answer

Keep FP16 weights and reduce the KV cache by capping the context length far below the requirement.