Model Optimization practice questions

Model pruning

Why it fails: Pruning removes redundant connections but often results in sparse weight matrices. Sparse operations frequently lack hardware acceleration support on standard edge devices, leading to negligible or even negative performance gains compared to the dense matrix multiplication optimizations achieved through effective quantization and packing techniques.

Knowledge distillation

Why it fails: Knowledge distillation involves training a smaller student model to mimic a larger teacher. While effective for reducing compute requirements, it requires significant retraining time and data resources, whereas quantization can be applied post-training to existing models to achieve immediate footprint reductions without extensive new training cycles.

Gradient checkpointing

Why it fails: Gradient checkpointing is an optimization technique designed to reduce memory consumption during the training phase by trading off computation for memory. It is not applicable to inference tasks where the model weights remain static and no backward passes or gradient calculations are performed during deployment.

Increasing the batch size to the maximum

Why it fails: While larger batch sizes can improve theoretical throughput, they cause linear increases in latency and memory consumption. If the batch size exceeds the available VRAM or causes excessive swapping, latency will spike, potentially leading to OOM errors. It is not a universally effective optimization technique for LLMs.

Applying data augmentation

Why it fails: Data augmentation is a training-time technique used to increase dataset size and prevent overfitting. It has no relevance to the optimization of the inference engine. TensorRT operates on static graphs and pre-trained weights; it cannot incorporate training-time data strategies into its runtime execution plan.

Disabling GPU page faulting

Why it fails: Disabling page faulting is a system-level configuration, not a model optimization technique. While it can prevent some operating system overhead, it does not address the core algorithmic bottlenecks like operation latency or memory bandwidth that define the performance characteristics of modern LLM inference engines on NVIDIA hardware.

Automatic support for arbitrary dynamic graphs

Why it fails: TensorRT excels with static graphs, which are common in production LLM deployments. PyTorch is superior for dynamic graphs where the architecture changes at runtime. TensorRT is not designed for arbitrary dynamic graphs; in fact, its reliance on static graph compilation is a key differentiator from PyTorch's eager mode.

Faster model training and fine-tuning

Why it fails: TensorRT is strictly an inference optimization tool, not a training library. While it can optimize the forward pass, it does not support backpropagation or gradient updates. Therefore, it provides no benefit for model training or fine-tuning, which remain the domain of frameworks like PyTorch or JAX.

The model is using dynamic batch sizes

Why it fails: While dynamic shapes can be configured in TensorRT, the error explicitly points to a dimension mismatch during a broadcast operation, not a batch size issue. If dynamic shapes were not defined correctly, the error would likely be a shape validation failure rather than an explicit broadcast dimension mismatch during graph construction.

The GPU lacks sufficient shared memory

Why it fails: Insufficient shared memory typically manifests as a kernel launch error or a compilation failure related to register usage. A broadcast dimension error is an algorithmic mismatch, not a hardware resource limitation. The problem is logical, not physical, and must be resolved by adjusting the shape definitions in the builder.

The model weights are corrupted

Why it fails: Corrupted weights would likely cause silent numerical failures or illegal memory access errors during execution. A dimension mismatch error during build time is almost always a configuration issue related to the definition of input/output shapes or the graph structure, not the content of the model's weight tensors.

Profiles are required for static shape models

Why it fails: Profiles are used specifically to handle dynamic shapes. If a model has strictly static shapes, the TensorRT builder does not require optimization profiles. Defining profiles for static models is unnecessary and does not provide any performance benefit compared to standard static build configurations.

Profiles automatically determine the best hardware

Why it fails: Optimization profiles define input shape ranges, not the target hardware. The TensorRT builder decides which hardware kernels to use based on the architecture, not the profile definitions. Profiles are purely about tensor dimensions, not about the selection of underlying GPU execution units or hardware-specific primitives.

Profiles reduce the total build time

Why it fails: Inclusion of multiple optimization profiles actually increases the build time, as the TensorRT builder must test and optimize the model for each profile's specified ranges. This leads to a more comprehensive search space, which requires additional compute time during the engine generation phase, not a reduction in time.

Model Pruning

Why it fails: Pruning involves removing redundant weights from a neural network to create sparse structures. While effective for reducing model size, it often requires extensive retraining or fine-tuning to recover accuracy lost during the removal process, making it less direct than quantization for immediate inference speedup on hardware.

Knowledge Distillation

Why it fails: Distillation involves training a smaller student model to mimic the behavior of a larger teacher model. This process requires significant compute resources and time for training, rather than being an immediate optimization step for an existing pre-trained model deployed on hardware with limited resources.

Weight Clustering

Why it fails: Weight clustering reduces the number of unique weights in a model by grouping them into clusters. While this helps with compression, it does not inherently speed up inference on standard hardware unless specialized sparse-aware compute kernels are utilized to skip calculations for the clustered weight values.

Dynamic Shape Support

Why it fails: Dynamic shape support allows TensorRT to handle variable input dimensions, which is necessary for flexible applications. However, it is a functional requirement for handling inputs rather than an optimization feature that intrinsically improves the raw computational throughput or reduces the latency of the model execution graph.

Model Parallelism

Why it fails: Model parallelism is a strategy for distributing a model across multiple GPUs to handle extremely large parameters. While it scales inference, it is a structural deployment strategy managed by libraries like Megatron-LM or NVIDIA Triton, not a feature of the TensorRT engine build process itself.

Weight Pruning

Why it fails: Pruning is a model compression technique typically applied before the model reaches the TensorRT build phase. TensorRT is an optimization engine that compiles existing model graphs into highly efficient execution plans; it does not perform the iterative weight removal process associated with network pruning.

CPU Offloading

Why it fails: CPU offloading moves model parts to system RAM when GPU memory is full. This causes a massive performance degradation due to the slow data transfer speed between the system and the GPU, making it unsuitable for high-performance generative inference, especially compared to multi-GPU parallelization techniques.

Batch Size Reduction

Why it fails: Reducing batch size can help the model fit into memory, but it does not address the fundamental limitation of the model parameters exceeding the VRAM capacity. Furthermore, it results in lower throughput per GPU, failing to maximize the hardware utilization of the expensive GPU resources available.

Operator Fusion

Why it fails: Operator fusion optimizes the execution graph by combining layers, which improves speed but does not increase the effective memory capacity of the system. If the model weights physically exceed the GPU VRAM, fusion will not prevent out-of-memory errors, as the weights must still reside in VRAM.

CPU-based Pre-processing

Why it fails: Offloading pre-processing to the CPU is typically done to free up GPU cycles, but it does not directly increase the throughput of the inference engine itself. In many high-performance pipelines, CPU overhead becomes a bottleneck, so this is rarely the primary method for increasing inference throughput.

Gradient Checkpointing

Why it fails: Gradient checkpointing is a technique used during the training phase to reduce memory consumption by storing only specific activations. It has no application in the inference phase and does not contribute to the optimization of throughput for deployed generative models running on the Triton Inference Server.

Tensor Cores

Why it fails: Tensor Cores are designed to accelerate matrix-multiply-accumulate operations. They provide massive throughput for dense linear algebra but are not directly responsible for accelerating element-wise operations like Softmax or LayerNorm, which require different hardware acceleration strategies within the GPU architecture.

NVLink Interconnect

Why it fails: NVLink is a high-speed communication interconnect for multi-GPU systems. While it improves data transfer rates between GPUs, it does not accelerate specific computational operations like Softmax. It is a hardware connectivity solution, not an algorithmic or kernel-level optimization for neural network layers.

CUDA Streams

Why it fails: CUDA Streams are a mechanism for managing the concurrent execution of kernels. While they are crucial for system-level orchestration, they are not a specific technology for accelerating the execution speed of individual Transformer-based operators like Softmax or LayerNorm; rather, they are a scheduling tool.

Increased model training speed

Why it fails: While FP8 can be used in training, the question specifically asks about inference optimization. The primary benefit in this context is the reduction of latency and memory footprint during the deployment phase, rather than the acceleration of the training loop for updating model weights.

Improved model interpretability

Why it fails: Precision levels like FP8, FP16, or INT8 do not impact the interpretability or the logic of the weights within the neural network. They are purely numerical representations used to optimize hardware performance and memory usage; they do not change how the model's decisions are derived or explained.

Elimination of the need for calibration

Why it fails: Even with FP8, calibration may still be beneficial or required to ensure that the dynamic range of activations is appropriately mapped to the lower-precision format. It does not eliminate the need for proper quantization-aware procedures to ensure the model maintains its predictive performance and robustness.

Increase the batch size to the maximum memory capacity.

Why it fails: Increasing batch size improves throughput but often degrades latency for individual requests. In high-concurrency environments, memory bus contention becomes a significant bottleneck, and simply increasing the batch size will eventually lead to out-of-memory errors or significant latency spikes, failing to address the fundamental memory bandwidth limitation.

Implement model pruning to remove 50% of model layers.

Why it fails: Pruning removes neurons or layers, which can impact the accuracy of the large language model significantly. While it reduces parameter count, modern NVIDIA Tensor Cores are highly optimized for dense matrix operations. Pruning often leads to sparse matrices that do not leverage the hardware acceleration as effectively as quantized dense matrices.

Convert the model to a standard CPU-based execution framework.

Why it fails: Moving the model to a CPU-based framework will drastically increase latency and decrease throughput compared to GPU acceleration. NVIDIA GPUs are specifically designed for the parallel processing required by LLMs. Relying on CPUs introduces massive computational bottlenecks that cannot be mitigated by standard optimization techniques like those available in TensorRT-LLM.

PagedAttention

Why it fails: PagedAttention is a memory management technique that organizes the KV cache into non-contiguous physical blocks to prevent fragmentation. While it is vital for maximizing memory utilization and enabling longer context windows, it does not manage the distribution of computation or memory state across multiple GPU devices for parallel execution.

Pipeline Parallelism

Why it fails: Pipeline Parallelism splits the model layers across different GPUs in a sequential order. While useful for very deep models, it creates a pipeline bubble where some GPUs remain idle while waiting for others to finish. It is less efficient for KV cache parallelization compared to Tensor Parallelism, which processes layers simultaneously.

Continuous Batching

Why it fails: Continuous batching is a scheduling mechanism that enables the system to process incoming requests as soon as others finish, maximizing GPU utilization. It does not handle the underlying distribution of the model's weights or the KV cache across hardware; it instead optimizes the request flow through the pre-allocated model instances.

Ensure the model is defined using static input shapes.

Why it fails: While static shapes can improve kernel performance by allowing for specialized code generation, they are not strictly required for kernel fusion. TensorRT can optimize dynamic shapes using profile-based planning, and layer normalization fusion can occur regardless of whether the input shapes are fixed or defined as dynamic ranges.

Manually split the model into smaller sub-graphs.

Why it fails: Manually splitting the model is counterproductive, as it forces the system to treat the graph as disjointed parts. The TensorRT optimizer operates best on the full graph, where it can identify contiguous chains of operations. Splitting the model actually prevents the optimizer from detecting valid fusion opportunities across the split point.

Increase the precision to FP64 for better stability.

Why it fails: Increasing precision to FP64 is unnecessary for LLM inference and consumes excessive memory bandwidth. Kernel fusion is most effective at FP16 or FP8, where the memory savings from fewer writes to global memory translate into major performance gains. FP64 does not support the same level of aggressive hardware-accelerated kernel fusion.

Enable the TensorRT-LLM JIT (just-in-time) engine compilation path.

Why it fails: JIT compilation in TensorRT-LLM builds kernels at runtime, which can reduce disk size but does not overlap compilation with weight loading. It typically increases cold-start latency because compilation happens when the request arrives rather than being precomputed, so it fails to meet the goal.

Enable the TensorRT-LLM 'async' engine initialization option.

Why it fails: TensorRT-LLM does not expose a runtime flag called 'async engine initialization' that overlaps kernel compilation with weight loading. While asynchronous APIs exist for some operations, they do not provide the described overlap for engine build artifacts, so this option is incorrect.

Enable CUDA Graph capture for the entire forward pass.

Why it fails: CUDA Graphs reduce kernel launch overhead during steady-state execution, but they capture a fixed graph after kernels are already compiled and weights are loaded. They do not overlap kernel compilation with weight loading, and they can actually increase cold-start time because capture must occur before serving begins.

Keep the model in FP16 and instead reduce latency by lowering the maximum batch size and increasing the number of CUDA streams.

Why it fails: This avoids quantization error entirely, but it does not reduce memory footprint or deliver the latency and throughput gains that quantization provides. Lowering batch size and adding streams can even reduce GPU utilization. The scenario explicitly asks for an approach that preserves accuracy while reducing memory and latency, and FP16 with scheduling tweaks fails to meet the reduction goals.

Increase the calibration set to 4096 generic web prompts so the quantizer observes a broader activation range and produces more robust scaling factors.

Why it fails: A larger but generic calibration set can actually widen the observed activation range and push more outliers into the quantization grid, degrading domain-specific accuracy. Calibration must match the deployment distribution; adding unrelated web prompts does not help numeric reasoning and may make the INT8 scaling less representative of the actual workload, offsetting any statistical benefit.

Convert the model to FP8 using Transformer Engine's per-tensor scaling and rely on the hardware's native FP8 support on the L40S.

Why it fails: FP8 can improve throughput, but per-tensor scaling applies a single scale to an entire tensor, which is too coarse when activations contain outliers, so accuracy loss can be similar to or worse than INT8. The L40S supports FP8, yet without finer-grained scaling such as per-channel or delayed scaling, numeric reasoning may still degrade, and the question asks for the best accuracy-preserving approach.

FP8 weights with tensor parallelism across 4 GPUs.

Why it fails: FP8 weights would require about 70GB, which fits on one H100, but tensor parallelism across 4 GPUs contradicts the single-GPU scenario. Moreover, FP8 KV cache is not mentioned, and the extra GPUs add communication overhead without necessity for this constraint.

INT8 weights with per-channel quantization and paged KV cache.

Why it fails: INT8 weights reduce the model to about 70GB, leaving little room for KV cache and activations on an 80GB GPU. For a 70B model, even with paged KV cache, the remaining memory is insufficient for reasonable batch sizes, leading to out-of-memory errors or severe throughput limits.

FP16 weights with tensor parallelism across 2 GPUs.

Why it fails: FP16 weights for a 70B model require about 140GB, which exceeds a single H100's 80GB. Tensor parallelism across 2 GPUs would split the weights, but the scenario specifies a single GPU. This option violates the hardware constraint and is therefore not viable.

Enable FP8 quantization of the weights and activations to halve the bytes moved in each all-reduce.

Why it fails: FP8 reduces weight and activation storage and can speed matmuls, but the tensor-parallel all-reduce still exchanges partial sums in the accumulation dtype, typically FP16 or BF16. The number and synchronization cost of collectives per layer is unchanged, so the communication-dominated decode latency remains governed by the tensor-parallel topology.

Raise the builder optimization level to 5 so TensorRT picks lower-latency communication kernels for the all-reduce.

Why it fails: Builder optimization level influences kernel tactic selection for the graph, and TensorRT-LLM does use optimized all-reduce plugins, but tactic choice cannot change the fundamental volume or count of tensor-parallel collectives. The topology that determines how many all-reduces occur per layer is the dominant factor the engineer must address.

Increase the maximum batch size so each all-reduce carries more tokens per collective.

Why it fails: Batching amortizes communication over more tokens and improves throughput, but it does not reduce the number of collectives per decode step or the latency of each one. Per-step decode latency is what the engineer measured, and larger batches can even increase it by raising the collective payload size without changing the fixed synchronization cost.

Reduce tensor parallelism from two GPUs to one and run the model on a single H100.

Why it fails: A 70B model in FP16 needs roughly 140 GB of weights, which does not fit in one 80 GB H100 without quantization or offloading. Removing the second GPU would force aggressive weight streaming over PCIe or host memory, which is far slower than the NVLink all-reduce traffic already observed, so it would worsen rather than fix the idle-GPU symptom.

Increase the maximum batch size in the TensorRT-LLM build configuration and rebuild the engine.

Why it fails: Raising max batch size enlarges internal buffers and can improve peak throughput only if requests actually queue up to that depth. The scenario shows GPUs idling between iterations while users are already waiting, which indicates the scheduler is not admitting work continuously; a larger static batch ceiling does not address that scheduling gap.

Switch the engine from FP16 to INT8 weights and rebuild with a calibration dataset.

Why it fails: INT8 weight-only quantization reduces memory footprint and can speed up matrix multiplies, but the profiling evidence points to communication stalls, not compute saturation. Precision reduction would not eliminate the inter-GPU transfer waits and adds accuracy risk, so it is a costly detour rather than the first fix for this throughput ceiling.

Enable paged KV cache with a block size tuned to the observed sequence lengths.

Why it fails: Paged KV cache reduces fragmentation and allows more concurrent sequences to share memory efficiently, but it does not reduce the bytes read per generated token for the weight matrices themselves. The decode-phase bottleneck in this scenario is repeatedly streaming model weights from HBM, which paged cache management does not address directly, so it is a capacity fix rather than a bandwidth fix.

Lower the KV cache precision to INT4 while keeping weights in FP16.

Why it fails: Shrinking the KV cache to INT4 frees memory and can allow longer contexts or more concurrent sequences, but KV cache reads are a small fraction of total bytes moved compared with the weight matrices. It also risks accuracy degradation on long conversations. It does not remove the per-token weight streaming that dominates decode latency here.

Apply kernel fusion and use the TensorRT-LLM fused multi-head attention plus GEMM plugins.

Why it fails: Fused attention and fused GEMM plugins cut kernel launch overhead and intermediate memory traffic, which helps latency, but they do not change the fundamental volume of weight bytes read per token during decode. They are valuable complements, yet the dominant cost when batch size is small remains reading the full weight set from HBM every step.