NCP-GENL Model Optimization Practice Test 3 — 15 Questions

Question 1

When optimizing a Generative AI model using NVIDIA TensorRT-LLM, which component is primarily responsible for managing the KV cache to minimize memory fragmentation?

Accepted Answer

The Paged KV Cache Manager. TensorRT-LLM utilizes PagedAttention or similar paged memory management techniques to handle the Key-Value (KV) cache. By dividing the cache into fixed-size blocks, the framework avoids the fragmentation issues inherent in contiguous memory allocation. This ensures that the memory is used efficiently during autoregressive generation, allowing for larger context windows and higher concurrent user capacity on the GPU.

Answer

The TensorRT Builder

Answer

The Model Parallelism Engine

Answer

The Triton Inference Server

Question 2

Refer to the exhibit. An engineer observes that GPU memory utilization is high, but the GPU is frequently idling. How does the provided Triton configuration optimize the inference pipeline?

Accepted Answer

It enables concurrent execution on the GPU. By setting the instance count to 2 on a single GPU, the configuration enables concurrent model execution. This allows Triton to schedule multiple inference requests to be processed simultaneously on the same hardware. This overlap helps to hide memory latency and pipeline stalls, effectively increasing the utilization of the GPU and raising the overall throughput of the deployment.

Answer

It forces the model to use INT8 precision

Answer

It partitions the GPU for dedicated memory

Answer

It enables dynamic batching for the requests

Question 3

An engineer is tasked with optimizing a model that performs poorly due to excessive memory access latency. Which TensorRT optimization strategy specifically targets this issue?

Accepted Answer

Layer Fusion. Layer fusion is the primary strategy for reducing memory access latency. By combining multiple kernels into one, the engine keeps data in the high-speed cache of the GPU instead of constantly writing to and reading from slow global VRAM. This minimizes the time spent waiting for data movement, which is usually the dominant bottleneck for modern neural networks.

Answer

Precision Calibration

Answer

Kernel Auto-Tuning

Answer

Weight Pruning

Question 4

What is the primary function of the 'Triton Model Analyzer' in an optimization workflow?

Accepted Answer

It benchmarks configuration trade-offs. The Model Analyzer is a tool designed to explore the trade-offs between throughput, latency, and memory usage for different model configurations. It automatically runs benchmarks with varying batch sizes and instance counts, providing developers with empirical data to find the optimal deployment parameters that meet their specific service level agreements for generative AI applications.

Answer

It converts models to TensorRT

Answer

It automatically quantizes the model

Answer

It manages model version control

Question 5

Which THREE factors should be considered when choosing an optimal batch size for LLM inference on NVIDIA GPUs?

Accepted Answer

Available GPU VRAM. Selecting an optimal batch size involves a balance between hardware saturation, memory capacity, and latency requirements. Larger batches improve throughput by better utilizing GPU compute cores but increase memory demand and individual request latency. The goal is to reach the highest throughput possible while staying within the VRAM limit and meeting the required latency SLA for the specific application.

Answer

Model Training Loss

Answer

CPU Clock Speed

Question 6

Which optimization technique specifically helps to manage the memory bandwidth bottleneck during the autoregressive decoding phase of an LLM?

Accepted Answer

KV Cache Quantization. During decoding, the model must read all KV cache values for every generated token. This is memory-bandwidth bound. Techniques like Quantized KV Cache (reducing the precision of cached tokens) and PagedAttention significantly reduce the amount of data moved between the VRAM and the compute units, effectively alleviating the memory bottleneck and allowing for faster token generation rates.

Answer

Operator Fusion

Answer

Weight Pruning

Answer

Dynamic Batching

Question 7

Refer to the exhibit. An engineer notices that the TensorRT engine takes an excessively long time to build. What is the most likely cause, and how can it be mitigated?

Accepted Answer

Use an explicit tactic sources list. The long build time is likely due to the large workspace allocation combined with a large search space for the optimization tactics. TensorRT tests a variety of kernel implementations to find the fastest one. To reduce build time, the engineer can limit the 'tactic selection' or use a profile-based build where common shapes are pre-recorded, preventing the engine from exhaustively searching every possible configuration for all input shapes.

Answer

Decrease the workspace size

Answer

Use a calibration cache

Answer

Increase the batch size

Question 8

Which optimization method should be prioritized when the model's inference performance is bottlenecked by the CPU-to-GPU data transfer overhead?

Accepted Answer

GPU-based Pre-processing. When data transfer between the host and GPU is the bottleneck, the most effective strategy is to move the pre-processing logic onto the GPU itself. Using CUDA-accelerated kernels to perform operations like tokenization or normalization directly on the GPU avoids moving data across the PCIe bus, which is a slow operation that stalls the GPU during the inference pipeline.

Answer

Quantization

Answer

Model Pruning

Answer

Operator Fusion

Question 9

Which THREE of the following are primary benefits of using PagedAttention in NVIDIA TensorRT-LLM deployments?

Accepted Answer

Elimination of external memory fragmentation.. PagedAttention is a critical optimization for LLM inference, solving the problem of memory fragmentation. By managing memory in non-contiguous pages, it allows the system to allocate only what is needed, reducing memory waste significantly. This enables higher batch sizes, better GPU utilization, and the ability to serve longer sequences without running out of memory, which are essential for maintaining high performance in production-grade LLM serving environments.

Answer

Faster calculation of attention scores.

Answer

Direct hardware support in the GPU scheduler.

Question 10

Refer to the exhibit. The deployment is facing memory allocation errors during peak load. Based on the error log, what is the most effective configuration change to resolve the issue while keeping the model architecture constant?

Accepted Answer

Enable PagedAttention in the TensorRT-LLM runtime.. The error log indicates that the system is running out of memory because the KV cache is allocated statically, which often leads to fragmentation or over-allocation. Enabling PagedAttention is the standard NVIDIA-recommended solution for this scenario. It allows the system to manage KV cache memory dynamically, reclaiming space from finished requests and efficiently allocating blocks for new tokens, thereby preventing the out-of-memory (OOM) errors caused by static allocation.

Answer

Switch to a smaller model size.

Answer

Increase the GPU clock frequency.

Answer

Reduce the batch size to one.

Question 11

A team is serving a 70B-parameter LLM with TensorRT-LLM in a multi-tenant environment where requests arrive with widely varying prompt lengths and generation lengths. During load testing, they observe that throughput collapses when a long-context request is scheduled alongside many short requests, and GPU memory fragmentation causes intermittent out-of-memory errors even though total free memory appears sufficient. Which TensorRT-LLM runtime configuration change most directly addresses both the throughput collapse and the memory fragmentation?

Accepted Answer

Enable the paged KV cache with a tuned block size and configure the batch scheduler to use in-flight batching with a per-iteration token budget.. The paged KV cache solves external fragmentation by allocating KV memory in uniform blocks from a shared pool, which prevents the OOM that occurs when a long-context request needs a large contiguous region. In-flight batching with a token budget lets the scheduler continuously admit and retire requests each iteration, so short requests no longer wait behind a long generation. These two runtime settings together address both symptoms without sacrificing accuracy or throughput.

Answer

Increase the TensorRT-LLM build max_batch_size and max_seq_len to their maximum possible values so the engine supports every request shape.

Answer

Switch the engine from FP16 to INT8 precision using a calibration dataset that matches the production prompt distribution.

Answer

Reduce the number of concurrent client connections at the load balancer so that only one request is processed by the GPU at a time.

Question 12

A team is deploying a Llama 2 13B model with NVIDIA TensorRT-LLM on a single A100 40GB GPU. They need to serve 32 concurrent requests with a maximum sequence length of 4096 tokens. They observe that the GPU runs out of memory during inference. Which configuration parameter should they adjust to control the maximum GPU memory allocated for the KV cache?

Accepted Answer

kv_cache_free_gpu_memory_fraction. The KV cache in TensorRT-LLM is allocated from a memory pool whose size is controlled by kv_cache_free_gpu_memory_fraction. Lowering this fraction reduces the KV cache footprint, resolving out-of-memory errors when serving multiple long sequences. Other parameters like max_batch_size or max_num_tokens influence scheduling but do not directly bound the KV cache memory pool.

Answer

max_batch_size

Answer

max_num_tokens

Answer

max_input_len

Question 13

An engineer is optimizing a BERT-like model for inference using NVIDIA TensorRT. They want to reduce latency further by using lower precision without significant accuracy loss. Which TensorRT precision mode should they choose to enable INT8 inference while maintaining accuracy through calibration?

Accepted Answer

INT8. INT8 precision mode in TensorRT allows inference using 8-bit integers, reducing latency and memory footprint. To preserve accuracy, a calibrator computes scaling factors from a calibration dataset. The other precisions (FP32, FP16, TF32) do not provide INT8 inference and do not use calibration for quantization.

Answer

FP16

Answer

FP32

Answer

TF32

Question 14

An engineer is using NVIDIA TensorRT-LLM to optimize an LLM for inference. They want to reduce the memory footprint of the KV cache during long-context generation. Which TWO techniques are supported by TensorRT-LLM to achieve this? (Choose two.)

Accepted Answer

Enable paged KV cache with block sharing.. TensorRT-LLM provides native support for FP8 KV cache quantization and paged KV cache with block sharing. FP8 quantization halves cache memory, while paged cache with block sharing reduces duplication across sequences. Both are configuration-time features that do not require model changes. The other options involve model modifications or misapply TensorRT features, making them incorrect for this scenario.

Answer

Use TensorRT's INT8 calibration for the KV cache.

Answer

Apply structured pruning to remove attention heads.

Answer

Enable sliding window attention in the model architecture.

Question 15

A developer is using NVIDIA TensorRT-LLM to optimize a GPT-based model for inference. They want to reduce the model's memory footprint and improve throughput without retraining. Which two techniques can be applied during the TensorRT-LLM build process to achieve these goals? (Choose two.)

Accepted Answer

Enabling FP8 precision for both weights and activations on Hopper GPUs. Weight-only quantization and FP8 precision are both build-time optimizations in TensorRT-LLM that reduce memory footprint and improve throughput without retraining. Weight-only quantization lowers weight precision, while FP8 leverages Hopper GPU capabilities for both weights and activations. The other options either require retraining or increase memory usage.

Answer

Pruning the model by removing entire attention heads

Answer

Knowledge distillation from a larger teacher model

Answer

Using a larger batch size to amortize memory overhead