NCP-GENL Model Optimization — 20 Questions

Question 1

An engineer needs to deploy a large language model on resource-constrained edge hardware. Which optimization technique provides the best balance between memory footprint reduction and inference latency?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Post-training weight quantization. Weight quantization is the most effective approach for edge deployment as it reduces the memory bandwidth bottleneck by using lower-precision arithmetic. By converting FP16 weights to INT8 or INT4, the model size decreases significantly, allowing it to fit into tighter cache hierarchies. This optimization is critical for real-time edge applications where power consumption and memory overhead are primary constraints for maintaining throughput and responsiveness.

Answer

Model pruning

Answer

Knowledge distillation

Answer

Gradient checkpointing

Question 2

Which TWO of the following TensorRT optimization techniques are most effective for improving throughput in LLMs using the Transformer architecture?

Accepted Answer

Kernel fusion for Attention layers. In Transformer-based LLMs, the most significant overheads are related to memory bandwidth and kernel launch latency. Fusing operations like Multi-Head Attention allows the engine to keep intermediate tensors in fast on-chip memory. Similarly, KV Cache management is vital to avoid recomputing previous tokens, directly impacting throughput. These techniques ensure the GPU executes compute-heavy layers efficiently while minimizing expensive global memory access cycles during the autoregressive generation loop.

Answer

Increasing the batch size to the maximum

Answer

Applying data augmentation

Answer

Disabling GPU page faulting

Question 3

Which THREE of the following are benefits of using NVIDIA TensorRT over raw PyTorch for LLM inference in production?

Accepted Answer

Optimized kernel selection based on hardware. TensorRT provides superior production performance by optimizing the computation graph, utilizing specialized hardware primitives like Tensor Cores, and reducing kernel overhead. By compiling the model into a fixed-format engine, it eliminates the dynamic overhead found in PyTorch's eager execution. These optimizations lead to lower latency and higher throughput, which are essential for serving large-scale users where every millisecond of inference time impacts cost and user experience.

Answer

Automatic support for arbitrary dynamic graphs

Answer

Faster model training and fine-tuning

Question 4

Refer to the exhibit. The builder fails during engine construction. What is the most likely cause related to the TensorRT optimization pipeline?

Accepted Answer

Static shape configuration mismatch. TensorRT engines are often built with static shapes to enable aggressive kernel optimization. If the input shape specified in the build configuration does not match the actual inference data or the model's expected input tensor shapes, the engine will fail to execute or build. This error indicates a mismatch between the build-time shape definitions and the broadcast operations performed during the model's forward pass, leading to a dimension error.

Answer

The model is using dynamic batch sizes

Answer

The GPU lacks sufficient shared memory

Answer

The model weights are corrupted

Question 5

Which TWO of the following are true regarding the use of 'profiles' in TensorRT for LLMs?

Accepted Answer

Profiles allow optimization for dynamic input shapes. TensorRT optimization profiles allow developers to define specific ranges for dynamic dimensions like sequence length and batch size. By providing multiple profiles, the engine builder can optimize the kernels for the most frequent input sizes, balancing performance and flexibility. This is essential for LLMs, where input lengths vary widely during inference, as a single static shape would either be inefficient for long sequences or fail for short ones.

Answer

Profiles are required for static shape models

Answer

Profiles automatically determine the best hardware

Answer

Profiles reduce the total build time

Question 6

An engineer needs to deploy a large language model on resource-constrained edge hardware. Which optimization technique provides the best balance between memory footprint reduction and inference latency?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Post-Training Quantization. Post-training quantization reduces the bit-width of weights and activations, significantly lowering memory bandwidth requirements and computational complexity. By converting FP32 parameters to INT8 or FP8, the model footprint shrinks and hardware-level tensor operations can execute faster. This process is essential for deploying large models on NVIDIA Jetson or similar edge devices where memory and power budgets are strictly limited compared to data center GPUs.

Answer

Model Pruning

Answer

Knowledge Distillation

Answer

Weight Clustering

Question 7

Which TWO of the following NVIDIA TensorRT features are primarily responsible for optimizing Transformer-based model performance during the build phase?

Accepted Answer

Layer Fusion. TensorRT optimizes neural networks by applying layer and tensor fusion, which reduces kernel launch overhead and memory access patterns. Specifically, multi-head attention fusion and INT8 calibration are critical components. These optimizations ensure that the graph is restructured for maximum throughput on NVIDIA GPUs, minimizing the performance bottlenecks typically associated with high-latency memory operations in large Transformer architectures during production deployment.

Answer

Dynamic Shape Support

Answer

Model Parallelism

Answer

Weight Pruning

Question 8

Which optimization strategy is most effective for reducing the inference latency of an LLM when the model exceeds the memory capacity of a single NVIDIA GPU?

Accepted Answer

Tensor Parallelism. Tensor parallelism allows the model to be split across multiple GPUs, where each device processes a portion of the matrix multiplications. This allows the model to fit into the aggregate memory of the cluster and reduces latency by distributing the compute load. This is the standard architectural approach for deploying large-scale Generative AI models that cannot be serviced by a single card.

Answer

CPU Offloading

Answer

Batch Size Reduction

Answer

Operator Fusion

Question 9

Which THREE techniques are commonly used to optimize inference throughput in Generative AI applications deployed on NVIDIA Triton Inference Server?

Accepted Answer

Dynamic Batching. Throughput in inference servers is maximized by increasing the number of requests processed simultaneously. Dynamic batching collects individual requests into larger batches to saturate GPU cores. Concurrent model execution allows multiple instances of a model to run on the same GPU. Finally, TensorRT engine optimization ensures that the model executes with the highest possible efficiency on the target hardware architecture.

Answer

CPU-based Pre-processing

Answer

Gradient Checkpointing

Question 10

Which NVIDIA technology provides the most significant boost for LLM inference by utilizing specialized hardware to accelerate non-matrix operations like Softmax and LayerNorm?

Accepted Answer

Fused Transformer Kernels. NVIDIA TensorRT-LLM includes custom, highly optimized kernels specifically tuned for Transformer operations beyond standard matrix multiplication. These fused kernels handle operations like Softmax, LayerNorm, and Activation functions, which are often the bottleneck in LLM inference. By executing these operations in a single fused kernel, the framework avoids unnecessary data movement, significantly improving performance for the entire Transformer decoding process.

Answer

Tensor Cores

Answer

NVLink Interconnect

Answer

CUDA Streams

Question 11

What is the primary benefit of using FP8 precision for inference in modern Transformer models on NVIDIA H100 GPUs?

Accepted Answer

Reduced memory bandwidth usage. The NVIDIA Hopper architecture introduces native hardware support for FP8, which doubles the throughput of Tensor Cores compared to FP16 while maintaining near-identical accuracy for most LLMs. This precision offers a superior balance of speed and numerical stability, allowing for faster inference latency and lower memory consumption without the significant accuracy drops sometimes associated with aggressive INT8 quantization.

Answer

Increased model training speed

Answer

Improved model interpretability

Answer

Elimination of the need for calibration

Question 12

An engineer needs to deploy a large language model on resource-constrained edge hardware. Which optimization technique provides the best balance between memory footprint reduction and inference latency?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Apply FP8 quantization to the model weights and activations.. Weight-only quantization, specifically FP8 or INT8, significantly reduces the memory footprint of large model weights, which is critical when the model is memory-bandwidth bound. By reducing the data movement per parameter, the GPU can process more tokens per second. This approach is standard in high-throughput inference scenarios where the memory bus, rather than compute cycles, acts as the primary limiting factor for overall system performance.

Answer

Increase the batch size to the maximum memory capacity.

Answer

Implement model pruning to remove 50% of model layers.

Answer

Convert the model to a standard CPU-based execution framework.

Question 13

Which NVIDIA TensorRT-LLM feature is primarily used to optimize the parallel execution of the KV cache across multiple GPU instances in a multi-node setup?

Accepted Answer

Tensor Parallelism. Tensor Parallelism is the standard technique for splitting the model across multiple GPUs to handle massive memory requirements and increase compute throughput. By partitioning the weights and the intermediate KV cache across different GPUs, the system can parallelize the computation of tokens, which is essential for scaling inference for models that exceed the memory capacity of a single GPU device.

Answer

PagedAttention

Answer

Pipeline Parallelism

Answer

Continuous Batching

Question 14

When optimizing a Transformer-based model using NVIDIA TensorRT, which TWO steps are essential to enable the Fusion of layer normalization and activation kernels to improve performance?

Accepted Answer

Enable the TensorRT optimization profile for layer fusion.. Kernel fusion combines multiple sequential operations into a single GPU kernel, reducing the overhead of launching many small kernels and minimizing global memory access. By fusing layer normalization and activation layers, the system avoids writing intermediate results back to main memory. This optimization is a core component of the TensorRT graph optimization phase, which analyzes the model architecture to identify and collapse redundant operations for better hardware utilization.

Answer

Ensure the model is defined using static input shapes.

Answer

Manually split the model into smaller sub-graphs.

Answer

Increase the precision to FP64 for better stability.

Question 15

A team wants to reduce the cold-start latency for an LLM served with TensorRT-LLM. They've already built an engine with FP16 precision and paged KV cache. Which TensorRT-LLM runtime feature should they enable to overlap the compilation of GPU kernels with the loading of model weights?

Accepted Answer

Use the TensorRT-LLM engine with 'lean' runtime mode and precompiled plugins.. Reducing cold-start latency in TensorRT-LLM often involves minimizing kernel compilation and plugin loading at runtime. The 'lean' runtime mode loads a reduced set of kernels and precompiled plugins, allowing weight loading to proceed in parallel with execution environment setup. This is a supported TensorRT-LLM feature aimed at production serving, unlike generic CUDA Graphs or JIT compilation, which either don't address cold start or worsen it.

Answer

Enable the TensorRT-LLM JIT (just-in-time) engine compilation path.

Answer

Enable the TensorRT-LLM 'async' engine initialization option.

Answer

Enable CUDA Graph capture for the entire forward pass.

Question 16

An engineer is optimizing a 13B-parameter LLM for a single NVIDIA L40S GPU using TensorRT-LLM. The model must retain high accuracy on a domain-specific question-answering benchmark, and the team has a small calibration set of 512 representative prompts. Post-training INT8 quantization causes a noticeable drop in answer quality, particularly for numeric reasoning. Which approach best preserves accuracy while still reducing memory and latency?

Accepted Answer

Apply SmoothQuant-style activation smoothing during post-training quantization, using the calibration prompts to shift quantization difficulty from activations to weights.. SmoothQuant-style activation smoothing is a post-training technique that rescales activations and weights so that activation outliers no longer dominate the quantization range. Because it uses the domain calibration prompts to compute scaling factors, it adapts to the actual numeric-reasoning workload. This preserves accuracy far better than naive INT8 while still delivering the memory and latency benefits of 8-bit inference, and it avoids the cost of full retraining.

Answer

Keep the model in FP16 and instead reduce latency by lowering the maximum batch size and increasing the number of CUDA streams.

Answer

Increase the calibration set to 4096 generic web prompts so the quantizer observes a broader activation range and produces more robust scaling factors.

Answer

Convert the model to FP8 using Transformer Engine's per-tensor scaling and rely on the hardware's native FP8 support on the L40S.

Question 17

An engineer is optimizing a 70B parameter LLM for inference on a single NVIDIA H100 GPU with 80GB memory. They want to maximize throughput while keeping latency acceptable. Which combination of TensorRT-LLM settings should they use to fit the model and KV cache in memory?

Accepted Answer

INT4 weights with group-wise quantization and FP8 KV cache.. For a 70B model on a single 80GB H100, memory is the primary constraint. INT4 group-wise quantization reduces weight memory to about 35GB, and FP8 KV cache further compresses the cache, allowing sufficient headroom for activations and large batches. This combination is supported in TensorRT-LLM and balances throughput and accuracy. Other options either exceed memory, leave insufficient headroom, or violate the single-GPU requirement.

Answer

FP8 weights with tensor parallelism across 4 GPUs.

Answer

INT8 weights with per-channel quantization and paged KV cache.

Answer

FP16 weights with tensor parallelism across 2 GPUs.

Question 18

An engineer profiles a TensorRT-LLM Llama-2-70B engine on 4x H100 GPUs with tensor parallelism and observes that decode-step latency is dominated by inter-GPU communication rather than matmul time. The engine currently uses tensor parallelism degree 4 with no pipeline parallelism. Which change most directly reduces the communication share during decoding?

Accepted Answer

Switch to tensor parallelism degree 2 with pipeline parallelism degree 2 across the same 4 GPUs.. Decode latency in tensor-parallel inference is bounded by the per-layer all-reduce, whose cost scales with the tensor-parallel group size and repeats at every transformer block. Reducing tensor parallelism to 2 and using pipeline parallelism degree 2 for the remaining split keeps all 4 GPUs busy while shrinking each collective and cutting collective count per step. Pipeline stages pass activations only at stage boundaries, so decode-step communication drops substantially.

Answer

Enable FP8 quantization of the weights and activations to halve the bytes moved in each all-reduce.

Answer

Raise the builder optimization level to 5 so TensorRT picks lower-latency communication kernels for the all-reduce.

Answer

Increase the maximum batch size so each all-reduce carries more tokens per collective.

Question 19

A team is serving a 70B-parameter LLM with NVIDIA TensorRT-LLM on two NVIDIA H100 GPUs using tensor parallelism. Under production traffic with many concurrent users, GPU utilization drops to roughly 40% and throughput plateaus well below the target, even though per-token latency stays acceptable. Profiling shows the GPUs idle while waiting on memory transfers between the two GPUs. Which optimization should the team apply first to raise throughput?

Accepted Answer

Enable in-flight batching (continuous batching) in the TensorRT-LLM runtime.. The symptom of high concurrency, acceptable per-token latency, and GPUs idling between tensor-parallel collectives indicates the runtime is not keeping the pipeline full. In-flight batching (continuous batching) in TensorRT-LLM admits new requests at each decoding step and retires finished sequences immediately, so both GPUs stay busy across iterations. This directly targets the scheduling stall rather than the communication pattern or numeric precision.

Answer

Reduce tensor parallelism from two GPUs to one and run the model on a single H100.

Answer

Increase the maximum batch size in the TensorRT-LLM build configuration and rebuild the engine.

Answer

Switch the engine from FP16 to INT8 weights and rebuild with a calibration dataset.

Question 20

An engineer is building a TensorRT-LLM engine for a 13B chat model on a single A100 80 GB GPU. The workload mixes short prompts with long multi-turn conversations, and the team wants to minimize end-to-end latency for interactive use without retraining. After enabling FP8 or INT8 weight-only quantization, which additional optimization most directly attacks the memory-bandwidth bottleneck during autoregressive decoding?

Accepted Answer

Increase the batch size by admitting more concurrent requests so weight reads are amortized across tokens.. Autoregressive decoding is memory-bandwidth bound because each generated token requires reading the full weight tensors from HBM, and at low batch size those reads are not amortized. Admitting more concurrent requests increases arithmetic intensity: a single weight fetch now produces many tokens across sequences. That directly reduces the bandwidth cost per token, which is the dominant latency factor for interactive chat workloads after weight quantization.

Answer

Enable paged KV cache with a block size tuned to the observed sequence lengths.

Answer

Lower the KV cache precision to INT4 while keeping weights in FP16.

Answer

Apply kernel fusion and use the TensorRT-LLM fused multi-head attention plus GEMM plugins.