NCP-GENL Model Optimization Practice Test 2 — 15 Questions

Question 1

An engineer is optimizing a 13B-parameter LLM for a single NVIDIA L40S GPU using TensorRT-LLM. The model must retain high accuracy on a domain-specific question-answering benchmark, and the team has a small calibration set of 512 representative prompts. Post-training INT8 quantization causes a noticeable drop in answer quality, particularly for numeric reasoning. Which approach best preserves accuracy while still reducing memory and latency?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Apply SmoothQuant-style activation smoothing during post-training quantization, using the calibration prompts to shift quantization difficulty from activations to weights.. SmoothQuant-style activation smoothing is a post-training technique that rescales activations and weights so that activation outliers no longer dominate the quantization range. Because it uses the domain calibration prompts to compute scaling factors, it adapts to the actual numeric-reasoning workload. This preserves accuracy far better than naive INT8 while still delivering the memory and latency benefits of 8-bit inference, and it avoids the cost of full retraining.

Answer

Keep the model in FP16 and instead reduce latency by lowering the maximum batch size and increasing the number of CUDA streams.

Answer

Increase the calibration set to 4096 generic web prompts so the quantizer observes a broader activation range and produces more robust scaling factors.

Answer

Convert the model to FP8 using Transformer Engine's per-tensor scaling and rely on the hardware's native FP8 support on the L40S.

Question 2

An engineer is optimizing a 70B parameter LLM for inference on a single NVIDIA H100 GPU with 80GB memory. They want to maximize throughput while keeping latency acceptable. Which combination of TensorRT-LLM settings should they use to fit the model and KV cache in memory?

Accepted Answer

INT4 weights with group-wise quantization and FP8 KV cache.. For a 70B model on a single 80GB H100, memory is the primary constraint. INT4 group-wise quantization reduces weight memory to about 35GB, and FP8 KV cache further compresses the cache, allowing sufficient headroom for activations and large batches. This combination is supported in TensorRT-LLM and balances throughput and accuracy. Other options either exceed memory, leave insufficient headroom, or violate the single-GPU requirement.

Answer

FP8 weights with tensor parallelism across 4 GPUs.

Answer

INT8 weights with per-channel quantization and paged KV cache.

Answer

FP16 weights with tensor parallelism across 2 GPUs.

Question 3

An engineer profiles a TensorRT-LLM Llama-2-70B engine on 4x H100 GPUs with tensor parallelism and observes that decode-step latency is dominated by inter-GPU communication rather than matmul time. The engine currently uses tensor parallelism degree 4 with no pipeline parallelism. Which change most directly reduces the communication share during decoding?

Accepted Answer

Switch to tensor parallelism degree 2 with pipeline parallelism degree 2 across the same 4 GPUs.. Decode latency in tensor-parallel inference is bounded by the per-layer all-reduce, whose cost scales with the tensor-parallel group size and repeats at every transformer block. Reducing tensor parallelism to 2 and using pipeline parallelism degree 2 for the remaining split keeps all 4 GPUs busy while shrinking each collective and cutting collective count per step. Pipeline stages pass activations only at stage boundaries, so decode-step communication drops substantially.

Answer

Enable FP8 quantization of the weights and activations to halve the bytes moved in each all-reduce.

Answer

Raise the builder optimization level to 5 so TensorRT picks lower-latency communication kernels for the all-reduce.

Answer

Increase the maximum batch size so each all-reduce carries more tokens per collective.

Question 4

A team is serving a 70B-parameter LLM with NVIDIA TensorRT-LLM on two NVIDIA H100 GPUs using tensor parallelism. Under production traffic with many concurrent users, GPU utilization drops to roughly 40% and throughput plateaus well below the target, even though per-token latency stays acceptable. Profiling shows the GPUs idle while waiting on memory transfers between the two GPUs. Which optimization should the team apply first to raise throughput?

Accepted Answer

Enable in-flight batching (continuous batching) in the TensorRT-LLM runtime.. The symptom of high concurrency, acceptable per-token latency, and GPUs idling between tensor-parallel collectives indicates the runtime is not keeping the pipeline full. In-flight batching (continuous batching) in TensorRT-LLM admits new requests at each decoding step and retires finished sequences immediately, so both GPUs stay busy across iterations. This directly targets the scheduling stall rather than the communication pattern or numeric precision.

Answer

Reduce tensor parallelism from two GPUs to one and run the model on a single H100.

Answer

Increase the maximum batch size in the TensorRT-LLM build configuration and rebuild the engine.

Answer

Switch the engine from FP16 to INT8 weights and rebuild with a calibration dataset.

Question 5

An engineer is building a TensorRT-LLM engine for a 13B chat model on a single A100 80 GB GPU. The workload mixes short prompts with long multi-turn conversations, and the team wants to minimize end-to-end latency for interactive use without retraining. After enabling FP8 or INT8 weight-only quantization, which additional optimization most directly attacks the memory-bandwidth bottleneck during autoregressive decoding?

Accepted Answer

Increase the batch size by admitting more concurrent requests so weight reads are amortized across tokens.. Autoregressive decoding is memory-bandwidth bound because each generated token requires reading the full weight tensors from HBM, and at low batch size those reads are not amortized. Admitting more concurrent requests increases arithmetic intensity: a single weight fetch now produces many tokens across sequences. That directly reduces the bandwidth cost per token, which is the dominant latency factor for interactive chat workloads after weight quantization.

Answer

Enable paged KV cache with a block size tuned to the observed sequence lengths.

Answer

Lower the KV cache precision to INT4 while keeping weights in FP16.

Answer

Apply kernel fusion and use the TensorRT-LLM fused multi-head attention plus GEMM plugins.

Question 6

An enterprise deploying a large language model on an NVIDIA A100 GPU experiences high memory bandwidth bottlenecks during autoregressive token generation. Which optimization technique specifically addresses this memory-bound phase by merging element-wise operations and reducing global memory round-trips?

Accepted Answer

Enabling TensorRT-LLM custom kernel fusion for multi-head attention and activation blocks.. Kernel fusion combines multiple successive GPU operations, such as bias additions and activations, into a single CUDA kernel. This drastically reduces high-latency global memory read and write operations, directly mitigating the memory bandwidth bottleneck characteristic of autoregressive transformer decoding phases on NVIDIA hardware.

Answer

Applying static INT8 post-training quantization to all linear layers.

Answer

Increasing the global batch size to maximize arithmetic intensity.

Answer

Switching from FlashAttention to standard vanilla self-attention mechanisms.

Question 7

When optimizing a model using NVIDIA TensorRT, what is the primary benefit of enabling 'layer fusion' during the optimization process?

Accepted Answer

It reduces global memory access overhead. Layer fusion reduces the overhead of launching multiple small GPU kernels by combining them into a single, optimized kernel. This strategy minimizes the total number of reads and writes to global VRAM, which is often the bottleneck in modern deep learning. By keeping intermediate data in high-speed registers or shared memory, the model spends less time waiting for memory access and more time performing actual floating-point operations.

Answer

It increases the overall model weight precision

Answer

It automatically prunes redundant parameters

Answer

It replaces floating-point math with integer math

Question 8

Refer to the exhibit. The TensorRT build process fails with a memory limit error. Which configuration adjustment is most likely to resolve this build-time error?

Accepted Answer

Increasing the builder's workspace memory limit. The build error indicates that the workspace memory required for the optimization process exceeds the available VRAM or the limit specified by the build configuration. By explicitly increasing the workspace size in the builder configuration, the engineer provides the necessary scratchpad memory for TensorRT to evaluate and test various kernel implementations, allowing the builder to successfully complete the optimization process for all layers of the model.

Answer

Reducing the number of hidden layers

Answer

Switching to a lower precision inference

Answer

Disabling the TensorRT engine cache

Question 9

Which strategy is most effective for optimizing an LLM that is too large to fit into a single GPU's VRAM?

Accepted Answer

Implement tensor parallelism. Model parallelism, specifically tensor parallelism, is the standard approach for splitting a large model across multiple GPUs. By partitioning individual layers across different devices, the computation can be distributed, and the total memory requirement is spread proportionally. This allows for the deployment of models that exceed the capacity of a single GPU, enabling high-performance inference for massive parameter models that would otherwise be impossible to load.

Answer

Increase the inference batch size

Answer

Apply dynamic quantization to the weights

Answer

Convert the model to a CPU-only format

Question 10

What is the primary role of an inference 'calibrator' when converting a model to INT8 precision?

Accepted Answer

It calculates the optimal quantization scaling factors. The calibrator identifies the optimal range for mapping floating-point activations to 8-bit integers without significant loss of information. It analyzes the distribution of input data to determine the scaling factors. Because INT8 has a limited dynamic range compared to FP32, the calibrator is essential to prevent overflow or underflow and to ensure that the quantized model remains accurate for the target inference tasks.

Answer

It prunes zero-value weights

Answer

It re-trains the model for higher accuracy

Answer

It optimizes the GPU kernel execution path

Question 11

Which hardware architecture feature is specifically leveraged by TensorRT to accelerate FP16 and INT8 matrix multiplications?

Accepted Answer

NVIDIA Tensor Cores. NVIDIA Tensor Cores are specialized hardware units designed to perform mixed-precision matrix multiply-accumulate operations in a single cycle. TensorRT automatically detects the presence of these cores and maps compute-intensive layers to them. By utilizing Tensor Cores, the model achieves significantly higher throughput and reduced latency compared to using standard CUDA cores, which perform operations at a lower efficiency per clock cycle for matrix math.

Answer

CPU AVX-512 vector instructions

Answer

Shared memory buffers in L1 cache

Answer

Global memory coalescing hardware

Question 12

Refer to the exhibit. The engineer is attempting to deploy on an NVIDIA Orin platform but encounters a runtime error. What is the most likely cause of the failure?

Accepted Answer

Unsupported operators are being targeted for the DLA. The inclusion of '--dla 0' forces the engine to run on the Deep Learning Accelerator (DLA) core. Many complex LLM operations, such as specific activation functions or advanced attention mechanisms, are not supported by the DLA's fixed-function logic. If the model graph contains unsupported operators, the build will either fail or generate a non-functional plan, as the DLA has a more restricted operator set than the primary GPU cores.

Answer

The workspace memory is too large

Answer

The DLA hardware is not enabled on this device

Answer

FP16 precision is not supported on DLA

Question 13

In the context of model optimization, why is 'graph surgery' sometimes required before building a TensorRT engine?

Accepted Answer

To resolve unsupported operator compatibility. Graph surgery is the process of modifying the model's computational graph to replace unsupported or inefficient operations with more optimized, TensorRT-compliant versions. Some frameworks export nodes that TensorRT cannot parse or optimize effectively. By manually editing the graph to fuse operations or simplify the structure before building, engineers can ensure that the engine builder produces a high-performance execution plan that effectively utilizes the underlying hardware capabilities.

Answer

To increase the model's parameter count

Answer

To improve the quality of training data

Answer

To automatically quantize the model to INT8

Question 14

What is the primary advantage of using a 'Quantization Aware Training' (QAT) approach over post-training quantization for LLMs?

Accepted Answer

It results in higher accuracy for quantized models. QAT incorporates quantization errors into the training loop, allowing the model to adapt its weights to the loss of precision. Unlike post-training quantization, which can cause significant accuracy degradation for complex LLMs, QAT ensures that the model remains robust despite the restricted dynamic range of the INT8 or FP8 format. This results in superior final inference accuracy, making it the preferred choice for high-stakes generative applications where precision is critical.

Answer

It eliminates the need for a calibration dataset

Answer

It produces significantly smaller model files

Answer

It avoids the use of TensorRT builder

Question 15

Refer to the exhibit. An engineer receives these logs while converting a Transformer model to a TensorRT engine. What is the most appropriate action to resolve this build failure?

Accepted Answer

Increase the builder workspace size. The error indicates that the workspace memory allocated to the TensorRT builder is insufficient to explore the search space of kernels for specific operations. Increasing the workspace size allows the builder to allocate larger temporary memory buffers, which is necessary for complex Transformer operations that require high-memory intermediate calculations to optimize effectively on NVIDIA hardware.

Answer

Reduce the input batch size

Answer

Switch to a smaller base model

Answer

Disable INT8 calibration