Scenario practice questions

Refer to the exhibit. Given this NeMo configuration, which prompt modification would best improve the reliability of technical support queries?

Exhibit

{"model_config": {"temperature": 0.2, "max_tokens": 512, "stop_sequences": ["Human:", "User:"], "system_prompt": "You are a helpful assistant. You must answer based on the provided technical manuals."}}

Trap 1: Append 'Be as creative as possible' to the system prompt.

Increasing creativity in a technical support context is dangerous, as it promotes hallucination. Technical manuals require high precision and factual adherence; prompting for creativity encourages the model to drift away from the source material, which is exactly the behavior you want to avoid in professional documentation support.

Trap 2: Change the stop sequences to be empty.

Stop sequences are essential for preventing the model from continuing to generate text after the answer is finished, such as simulating a user response. Removing these markers increases the likelihood of runaway generation and model drift, which reduces the overall professionalism and utility of the generated response.

Trap 3: Increase the temperature to 0.9 to ensure varied answers.

Raising the temperature to 0.9 would introduce significant randomness into the output, which is detrimental for technical tasks. In professional support scenarios, you want deterministic and consistent answers based on documentation. High temperatures make the model unreliable, leading to inconsistent guidance that frustrates users seeking accurate technical information.

A
Append 'Be as creative as possible' to the system prompt.
Why it fails: Increasing creativity in a technical support context is dangerous, as it promotes hallucination. Technical manuals require high precision and factual adherence; prompting for creativity encourages the model to drift away from the source material, which is exactly the behavior you want to avoid in professional documentation support.
B
Change the stop sequences to be empty.
Why it fails: Stop sequences are essential for preventing the model from continuing to generate text after the answer is finished, such as simulating a user response. Removing these markers increases the likelihood of runaway generation and model drift, which reduces the overall professionalism and utility of the generated response.
C
Require the model to state 'I cannot answer this' if the manual is silent.
This specific instruction provides a clear 'exit path' for the model when the provided context is inadequate. By explicitly defining the behavior for unsupported queries, you prevent the model from guessing or fabricating answers, which is crucial for maintaining the trust and reliability of your technical documentation bot.
D
Increase the temperature to 0.9 to ensure varied answers.
Why it fails: Raising the temperature to 0.9 would introduce significant randomness into the output, which is detrimental for technical tasks. In professional support scenarios, you want deterministic and consistent answers based on documentation. High temperatures make the model unreliable, leading to inconsistent guidance that frustrates users seeking accurate technical information.

Question 2easymultiple choice

Which technique is most effective for mitigating data leakage during the training of an LLM on time-series-related document data?

Trap 1: Random shuffling of all documents regardless of their timestamps.

Randomly shuffling documents in temporal datasets is the primary cause of data leakage. It allows the model to observe future information during the training phase, which is impossible in real-world scenarios. This invalidates performance evaluation metrics and leads to models that perform poorly once deployed in actual environments.

Trap 2: Oversampling the minority class in the training set to improve…

Oversampling can introduce duplicate entries that skew the distribution and lead to overfitting. While it addresses class imbalance, it does not mitigate the temporal data leakage caused by training on future information. It is a technique for balancing distributions, not for maintaining chronological integrity during dataset preparation.

Trap 3: Applying aggressive data normalization to all numerical values.

Normalization is a standard preprocessing step that ensures model stability by scaling inputs, but it does not address the temporal sequence leakage. It ensures that inputs are mathematically comparable, but it has no effect on whether the model is seeing future labels or temporal patterns during its training phase.

A
Random shuffling of all documents regardless of their timestamps.
Why it fails: Randomly shuffling documents in temporal datasets is the primary cause of data leakage. It allows the model to observe future information during the training phase, which is impossible in real-world scenarios. This invalidates performance evaluation metrics and leads to models that perform poorly once deployed in actual environments.
B
Chronological splitting of the dataset based on a cutoff date.
Chronological splitting ensures that the model is trained exclusively on data from the past, while validation and testing are performed on subsequent time periods. This mirrors the real-world deployment environment, where the model must predict future events based only on information that has already occurred in history.
C
Oversampling the minority class in the training set to improve balance.
Why it fails: Oversampling can introduce duplicate entries that skew the distribution and lead to overfitting. While it addresses class imbalance, it does not mitigate the temporal data leakage caused by training on future information. It is a technique for balancing distributions, not for maintaining chronological integrity during dataset preparation.
D
Applying aggressive data normalization to all numerical values.
Why it fails: Normalization is a standard preprocessing step that ensures model stability by scaling inputs, but it does not address the temporal sequence leakage. It ensures that inputs are mathematically comparable, but it has no effect on whether the model is seeing future labels or temporal patterns during its training phase.

Question 3mediummultiple choice

An enterprise fine-tunes a Llama-3-70B model using NVIDIA NeMo for automated technical support ticketing. The development team needs an automated evaluation pipeline that measures semantic similarity against human-curated reference answers without relying on costly human annotators. Which metric provides the most robust embedding-based semantic similarity assessment for this scenario?

Trap 1: ROUGE-1 measures unigram overlap but fails to capture semantic…

ROUGE-1 counts exact unigram matches, so paraphrases and synonyms in support answers score poorly despite identical meaning, making it unsuitable for embedding-based semantic assessment. It is tempting because it is a cheap, widely used reference-based metric, and it would be correct where surface lexical overlap with the reference genuinely reflects answer quality.

Trap 2: BLEU evaluates n-gram precision with a brevity penalty, heavily…

BLEU was originally designed for rigid machine translation tasks and focuses heavily on exact n-gram precision. It lacks semantic understanding and penalizes valid alternative phrasings, making it a poor choice for evaluating modern generative support assistants.

Trap 3: Perplexity measures how well a probability distribution predicts a…

Perplexity scores how fluently a model predicts token sequences, with no reference answer involved, so it cannot measure alignment with curated responses. It is tempting because it is a standard intrinsic language-model metric requiring no labelled data, and it would be correct for comparing base and fine-tuned model fluency, not semantic similarity evaluation.

A
ROUGE-1 measures unigram overlap but fails to capture semantic synonyms and contextual nuances common in enterprise technical support documentation.
Why it fails: ROUGE-1 counts exact unigram matches, so paraphrases and synonyms in support answers score poorly despite identical meaning, making it unsuitable for embedding-based semantic assessment. It is tempting because it is a cheap, widely used reference-based metric, and it would be correct where surface lexical overlap with the reference genuinely reflects answer quality.
B
BLEU evaluates n-gram precision with a brevity penalty, heavily penalizing valid creative paraphrasing typically found in conversational AI support responses.
Why it fails: BLEU was originally designed for rigid machine translation tasks and focuses heavily on exact n-gram precision. It lacks semantic understanding and penalizes valid alternative phrasings, making it a poor choice for evaluating modern generative support assistants.
C
BERTScore computes token similarity matrices using contextual embeddings to capture deep semantic meaning regardless of surface-level phrasing variations.
BERTScore aligns tokens between candidate and reference using contextual embeddings, computing precision, recall and F1 over cosine similarity. This satisfies the stem's requirement for embedding-based semantic assessment that captures meaning despite surface-level phrasing variation, without human annotators.
D
Perplexity measures how well a probability distribution predicts a sample, reflecting language fluency rather than semantic alignment with a specific reference answer.
Why it fails: Perplexity scores how fluently a model predicts token sequences, with no reference answer involved, so it cannot measure alignment with curated responses. It is tempting because it is a standard intrinsic language-model metric requiring no labelled data, and it would be correct for comparing base and fine-tuned model fluency, not semantic similarity evaluation.

Question 4hardmultiple choice

Refer to the exhibit. The deployment is facing memory allocation errors during peak load. Based on the error log, what is the most effective configuration change to resolve the issue while keeping the model architecture constant?

Exhibit

Error: [TRT-LLM] KV Cache block allocation failed. Current allocation: 80% capacity. Request rejected to prevent OOM. Optimization status: KV_CACHE_ENABLED=True, PAGED_ATTENTION=False.

Trap 1: Switch to a smaller model size.

Switching to a smaller model changes the architecture, which was ruled out by the requirement to keep the model constant. Furthermore, simply reducing the model size does not fix the fundamental issue of memory fragmentation in the KV cache, which is the root cause of the observed OOM errors.

Trap 2: Increase the GPU clock frequency.

Increasing the GPU clock frequency improves raw compute performance but does nothing to solve memory allocation failures. OOM errors are a result of memory capacity exhaustion or fragmentation, not computational speed. Adjusting clock speeds will likely increase power consumption and heat without addressing the memory management limitation shown.

Trap 3: Reduce the batch size to one.

Reducing the batch size to one will lower memory usage, but it effectively cripples the throughput of the inference service. This is a workaround, not an optimization. Enabling PagedAttention is a superior technical solution because it allows for high throughput while maintaining stable memory management, rather than sacrificing performance.

A
Switch to a smaller model size.
Why it fails: Switching to a smaller model changes the architecture, which was ruled out by the requirement to keep the model constant. Furthermore, simply reducing the model size does not fix the fundamental issue of memory fragmentation in the KV cache, which is the root cause of the observed OOM errors.
B
Increase the GPU clock frequency.
Why it fails: Increasing the GPU clock frequency improves raw compute performance but does nothing to solve memory allocation failures. OOM errors are a result of memory capacity exhaustion or fragmentation, not computational speed. Adjusting clock speeds will likely increase power consumption and heat without addressing the memory management limitation shown.
C
Enable PagedAttention in the TensorRT-LLM runtime.
PagedAttention is designed to solve exactly this type of KV cache allocation failure. By moving from static allocation to paged block allocation, the system can utilize memory more efficiently and pack more requests into the same GPU memory footprint, effectively eliminating the OOM errors caused by inflexible memory management.
D
Reduce the batch size to one.
Why it fails: Reducing the batch size to one will lower memory usage, but it effectively cripples the throughput of the inference service. This is a workaround, not an optimization. Enabling PagedAttention is a superior technical solution because it allows for high throughput while maintaining stable memory management, rather than sacrificing performance.

Question 5easymultiple choice

A technical support team is building a chatbot using an NVIDIA NIM microservice. The chatbot must answer questions about a specific product's warranty policy. The team wants to ensure the model's responses are grounded in the official warranty document, which is 50 pages long, and avoid inventing policy details. Which prompt engineering approach is most effective for this scenario?

Trap 1: Ask the model to 'think step by step' and reason about the warranty…

The model's pre-trained knowledge likely does not include the specific warranty policy of this product, so reasoning from memory will lead to invented details. Chain-of-thought does not inject factual grounding; it only structures reasoning. Without the actual document content, the model cannot provide accurate answers. This approach is inappropriate for proprietary or up-to-date information.

Trap 2: Fine-tune the model on the warranty document using NVIDIA NeMo,…

Fine-tuning can help the model learn the document's content, but it is costly and may not prevent hallucinations, especially for specific policy details that change. It also requires retraining when the document updates. For a 50-page document, fine-tuning is overkill and less flexible than RAG. Prompt engineering with context is more suitable for this use case.

Trap 3: Include the entire warranty document in the system prompt and…

While including the full document may seem thorough, it can exceed the model's context window or dilute attention, leading to missed details. Also, the model may still hallucinate if the document is not properly segmented or if instructions are not precise. This approach is inefficient and may not scale. A more targeted retrieval of relevant sections is generally more effective for grounding.

A
Ask the model to 'think step by step' and reason about the warranty policy from its pre-trained knowledge.
Why it fails: The model's pre-trained knowledge likely does not include the specific warranty policy of this product, so reasoning from memory will lead to invented details. Chain-of-thought does not inject factual grounding; it only structures reasoning. Without the actual document content, the model cannot provide accurate answers. This approach is inappropriate for proprietary or up-to-date information.
B
Use a retrieval-augmented generation (RAG) pipeline to fetch the most relevant warranty sections and include them in the prompt as context.
RAG retrieves only the pertinent sections of the warranty document and places them in the prompt, providing focused context that the model can use to generate accurate answers. This reduces hallucination because the model is conditioned on specific, relevant text. It also scales to large documents and keeps the prompt within token limits. This is the standard approach for grounding LLM responses in proprietary knowledge bases.
C
Fine-tune the model on the warranty document using NVIDIA NeMo, then use the fine-tuned model without any additional context in the prompt.
Why it fails: Fine-tuning can help the model learn the document's content, but it is costly and may not prevent hallucinations, especially for specific policy details that change. It also requires retraining when the document updates. For a 50-page document, fine-tuning is overkill and less flexible than RAG. Prompt engineering with context is more suitable for this use case.
D
Include the entire warranty document in the system prompt and instruct the model to answer based only on that document.
Why it fails: While including the full document may seem thorough, it can exceed the model's context window or dilute attention, leading to missed details. Also, the model may still hallucinate if the document is not properly segmented or if instructions are not precise. This approach is inefficient and may not scale. A more targeted retrieval of relevant sections is generally more effective for grounding.

Question 6mediummulti select

An engineer is using NVIDIA TensorRT-LLM to optimize an LLM for inference. They want to reduce the memory footprint of the KV cache during long-context generation. Which TWO techniques are supported by TensorRT-LLM to achieve this? (Choose two.)

Trap 1: Use TensorRT's INT8 calibration for the KV cache.

TensorRT's INT8 calibration is for model weights and activations, not for the KV cache. TensorRT-LLM handles KV cache quantization separately, and INT8 calibration does not apply to the cache. Thus, this is not a supported technique for reducing KV cache memory.

Trap 2: Apply structured pruning to remove attention heads.

Structured pruning of attention heads reduces model weights, not the KV cache memory. While it can lower overall memory, it does not target the KV cache specifically and requires retraining or fine-tuning. TensorRT-LLM does not provide built-in pruning for KV cache reduction during inference.

Trap 3: Enable sliding window attention in the model architecture.

Sliding window attention is a model architecture modification that limits the attention span, reducing KV cache size, but it requires changing the model and retraining. TensorRT-LLM does not automatically apply it; it must be part of the model definition. Therefore, it is not a TensorRT-LLM supported technique per se.

A
Use TensorRT's INT8 calibration for the KV cache.
Why it fails: TensorRT's INT8 calibration is for model weights and activations, not for the KV cache. TensorRT-LLM handles KV cache quantization separately, and INT8 calibration does not apply to the cache. Thus, this is not a supported technique for reducing KV cache memory.
B
Enable paged KV cache with block sharing.
TensorRT-LLM's paged KV cache divides the cache into blocks and allows sharing of identical blocks across sequences, reducing memory when multiple requests share prefixes. This is a core feature for efficient memory management in long-context scenarios and is enabled by default in recent versions.
C
Apply structured pruning to remove attention heads.
Why it fails: Structured pruning of attention heads reduces model weights, not the KV cache memory. While it can lower overall memory, it does not target the KV cache specifically and requires retraining or fine-tuning. TensorRT-LLM does not provide built-in pruning for KV cache reduction during inference.
D
Use FP8 KV cache quantization.
TensorRT-LLM supports quantizing the KV cache to FP8, which halves its memory footprint compared to FP16. This is done by setting the kv_cache_dtype to 'fp8' in the build configuration. It is a native feature and does not require model retraining, making it a direct way to reduce memory during long-context generation.
E
Enable sliding window attention in the model architecture.
Why it fails: Sliding window attention is a model architecture modification that limits the attention span, reducing KV cache size, but it requires changing the model and retraining. TensorRT-LLM does not automatically apply it; it must be part of the model definition. Therefore, it is not a TensorRT-LLM supported technique per se.

Question 7mediummultiple choice

A team is training a large language model on a single NVIDIA H100 GPU. They observe that training throughput is significantly lower than expected, and profiling with Nsight Systems shows long periods where the GPU is idle waiting for data. The data loading pipeline uses the default PyTorch DataLoader with num_workers=0 and no pinned memory. Which change is most likely to improve GPU utilization?

Trap 1: Use CUDA graphs to capture the training step and reduce kernel…

CUDA graphs reduce CPU launch overhead, but the symptom is GPU idle time waiting for data, not launch-bound gaps. If the GPU is starved by slow data loading, capturing the step in a graph will not feed it faster. The data pipeline must be optimized first.

Trap 2: Switch from FP32 to TF32 precision for matrix multiplications.

TF32 can accelerate compute-bound kernels, but the profiling data shows the GPU is idle due to data starvation, not compute limits. Changing precision would not reduce the idle periods; the bottleneck remains the input pipeline. It might even introduce accuracy trade-offs without addressing the observed stalls.

Trap 3: Increase the batch size to better saturate the GPU.

Increasing batch size may improve utilization only if the GPU is compute-bound, but here profiling shows idle time waiting for data. A larger batch could even worsen the problem by increasing memory pressure and requiring more data per step, potentially lengthening the stalls. The root cause is the data pipeline, not insufficient work per kernel launch.

A
Use CUDA graphs to capture the training step and reduce kernel launch overhead.
Why it fails: CUDA graphs reduce CPU launch overhead, but the symptom is GPU idle time waiting for data, not launch-bound gaps. If the GPU is starved by slow data loading, capturing the step in a graph will not feed it faster. The data pipeline must be optimized first.
B
Switch from FP32 to TF32 precision for matrix multiplications.
Why it fails: TF32 can accelerate compute-bound kernels, but the profiling data shows the GPU is idle due to data starvation, not compute limits. Changing precision would not reduce the idle periods; the bottleneck remains the input pipeline. It might even introduce accuracy trade-offs without addressing the observed stalls.
C
Increase the batch size to better saturate the GPU.
Why it fails: Increasing batch size may improve utilization only if the GPU is compute-bound, but here profiling shows idle time waiting for data. A larger batch could even worsen the problem by increasing memory pressure and requiring more data per step, potentially lengthening the stalls. The root cause is the data pipeline, not insufficient work per kernel launch.
D
Enable pinned memory and increase num_workers in the DataLoader.
Pinned memory allows asynchronous host-to-device copies, and multiple worker processes prefetch batches in parallel, overlapping data loading with GPU compute. This directly addresses the idle GPU time observed in profiling. With num_workers=0 and no pinning, data loading is serialized and synchronous, starving the GPU.

Question 8mediummultiple choice

You are responsible for the reliability of an LLM inference service running on NVIDIA Triton Inference Server across a fleet of A100 GPUs. The service is deployed with dynamic batching enabled, but during peak hours you observe that end-to-end latency for some requests exceeds the SLO while GPU utilization remains moderate. You suspect that the dynamic batching configuration is causing the issue. Which Triton configuration parameter should you adjust to directly limit the maximum time a request waits in the scheduler queue before being batched?

Trap 1: Increase the `instance_group` count to add more model instances.

Increasing `instance_group` count creates additional model instances that can process batches in parallel, potentially improving throughput and reducing queuing if the GPU has spare capacity. However, it does not directly control how long a request waits in the scheduler queue before being batched. The latency issue described is specifically due to batching delay, which is governed by `max_queue_delay_microseconds`. Adding instances may not solve the SLO violation if the delay is inherent to batching.

Trap 2: Set the `max_batch_size` parameter to a lower value.

Reducing `max_batch_size` limits the maximum number of requests that can be combined into a single batch, which may reduce per-request latency under some conditions but does not directly bound the time a request waits in the scheduler queue. The delay is controlled by the `max_queue_delay_microseconds` parameter, not by batch size. Additionally, lowering batch size can reduce throughput and may not address queue wait time.

Trap 3: Enable the `priority_levels` parameter to prioritize certain…

`priority_levels` allows you to assign different priorities to requests, which can be useful for differentiated service levels, but it does not directly limit the maximum queue wait time for all requests. It only affects the order in which requests are dequeued. Without adjusting `max_queue_delay_microseconds`, low-priority requests could still experience excessive latency. This parameter is not the primary control for bounding batching delay.

A
Increase the `instance_group` count to add more model instances.
Why it fails: Increasing `instance_group` count creates additional model instances that can process batches in parallel, potentially improving throughput and reducing queuing if the GPU has spare capacity. However, it does not directly control how long a request waits in the scheduler queue before being batched. The latency issue described is specifically due to batching delay, which is governed by `max_queue_delay_microseconds`. Adding instances may not solve the SLO violation if the delay is inherent to batching.
B
Set the `max_queue_delay_microseconds` parameter to a lower value.
The `max_queue_delay_microseconds` parameter in Triton's dynamic batching configuration specifies the maximum time a request can wait in the queue before the scheduler dispatches the batch. By lowering this value, you reduce the maximum latency contributed by batching, which directly addresses the observed SLO violations. This is the intended parameter for controlling batching-induced latency while still allowing some batching for efficiency.
C
Set the `max_batch_size` parameter to a lower value.
Why it fails: Reducing `max_batch_size` limits the maximum number of requests that can be combined into a single batch, which may reduce per-request latency under some conditions but does not directly bound the time a request waits in the scheduler queue. The delay is controlled by the `max_queue_delay_microseconds` parameter, not by batch size. Additionally, lowering batch size can reduce throughput and may not address queue wait time.
D
Enable the `priority_levels` parameter to prioritize certain requests.
Why it fails: `priority_levels` allows you to assign different priorities to requests, which can be useful for differentiated service levels, but it does not directly limit the maximum queue wait time for all requests. It only affects the order in which requests are dequeued. Without adjusting `max_queue_delay_microseconds`, low-priority requests could still experience excessive latency. This parameter is not the primary control for bounding batching delay.

Question 9mediummultiple choice

You are preparing a dataset of support tickets for a RAG system using NVIDIA NeMo. Many tickets are short and contain little context, which hurts retrieval quality. Which data preparation technique best improves retrieval by enriching each ticket with related information before embedding?

Trap 1: Increase the embedding dimension of the retrieval model.

A larger embedding dimension can capture more detail, but it cannot invent information that is absent from a short ticket. The fundamental problem is insufficient content, not insufficient representational capacity. Changing the embedding size does not enrich the source text and therefore does not resolve the retrieval quality issue described.

Trap 2: Lower the similarity threshold used during retrieval.

Lowering the similarity threshold returns more matches, including irrelevant ones, which can reduce precision without fixing the underlying sparse representations. The tickets still lack context, so their embeddings remain weak. This retrieval-time adjustment does not perform data preparation and fails to address the root cause of poor retrieval quality.

Trap 3: Apply deduplication to remove similar tickets from the corpus.

Deduplication reduces redundancy but does not add context to short tickets. After deduplication, the remaining tickets are still sparse and may embed poorly, so retrieval quality does not improve in the way the scenario requires. This technique addresses corpus size, not the lack of contextual information in individual records.

A
Increase the embedding dimension of the retrieval model.
Why it fails: A larger embedding dimension can capture more detail, but it cannot invent information that is absent from a short ticket. The fundamental problem is insufficient content, not insufficient representational capacity. Changing the embedding size does not enrich the source text and therefore does not resolve the retrieval quality issue described.
B
Lower the similarity threshold used during retrieval.
Why it fails: Lowering the similarity threshold returns more matches, including irrelevant ones, which can reduce precision without fixing the underlying sparse representations. The tickets still lack context, so their embeddings remain weak. This retrieval-time adjustment does not perform data preparation and fails to address the root cause of poor retrieval quality.
C
Apply deduplication to remove similar tickets from the corpus.
Why it fails: Deduplication reduces redundancy but does not add context to short tickets. After deduplication, the remaining tickets are still sparse and may embed poorly, so retrieval quality does not improve in the way the scenario requires. This technique addresses corpus size, not the lack of contextual information in individual records.
D
Use an LLM to generate contextual summaries or metadata for each ticket and prepend them to the text before embedding.
Generating a summary or metadata adds missing context to each short ticket, so the embedded representation captures more relevant semantics and matches user queries better. Prepending this enriched text before embedding directly addresses the sparsity problem and improves retrieval, making it the appropriate technique for the scenario.

Study the full Python automation breakdown →

Question 10mediummultiple choice

You are evaluating a fine-tuned LLM for a code generation task. The model was trained using NVIDIA NeMo on a dataset of Python functions. You want to measure the percentage of generated functions that pass a set of unit tests. Which evaluation metric is most appropriate?

Trap 1: ROUGE-L

ROUGE-L computes longest common subsequence overlap between generated and reference code, focusing on textual similarity. Like BLEU, it does not execute code or verify correctness. Two functions with similar text may behave differently, and correct functions may have low overlap. Thus, ROUGE-L cannot measure the percentage of functions that pass unit tests, making it inappropriate for this evaluation.

Trap 2: BLEU score

BLEU score measures n-gram overlap between generated and reference code, which does not reflect functional correctness. Two functionally equivalent functions can have low BLEU if they use different variable names or structures. Since the scenario requires measuring whether generated functions pass unit tests, BLEU is inadequate because it ignores execution behavior and only compares surface text.

Trap 3: Perplexity

Perplexity measures how well a language model predicts a sequence of tokens, reflecting confidence in the training distribution. It does not indicate whether generated code is functionally correct or passes tests. A low perplexity model can still produce code with syntax errors or logical flaws. Therefore, perplexity is not suitable for evaluating code generation success against unit tests.

A
ROUGE-L
Why it fails: ROUGE-L computes longest common subsequence overlap between generated and reference code, focusing on textual similarity. Like BLEU, it does not execute code or verify correctness. Two functions with similar text may behave differently, and correct functions may have low overlap. Thus, ROUGE-L cannot measure the percentage of functions that pass unit tests, making it inappropriate for this evaluation.
B
BLEU score
Why it fails: BLEU score measures n-gram overlap between generated and reference code, which does not reflect functional correctness. Two functionally equivalent functions can have low BLEU if they use different variable names or structures. Since the scenario requires measuring whether generated functions pass unit tests, BLEU is inadequate because it ignores execution behavior and only compares surface text.
C
Perplexity
Why it fails: Perplexity measures how well a language model predicts a sequence of tokens, reflecting confidence in the training distribution. It does not indicate whether generated code is functionally correct or passes tests. A low perplexity model can still produce code with syntax errors or logical flaws. Therefore, perplexity is not suitable for evaluating code generation success against unit tests.
D
pass@k
pass@k measures the percentage of problems for which at least one of k generated samples passes all unit tests. It directly evaluates functional correctness by executing the code against test cases. This metric is standard for code generation tasks and aligns with the scenario's goal of measuring how many generated functions pass unit tests. It accounts for the stochastic nature of LLM outputs by considering multiple samples.

Question 11hardmultiple choice

Refer to the exhibit. What is the most likely reason for the high P99 latency despite low GPU utilization?

Exhibit

{
  "model_name": "llama-3",
  "latency_percentile_99": 1200,
  "target_p99": 500,
  "queue_depth": 50,
  "gpu_util": "40%"
}

Trap 1: The GPU compute units are failing

If GPU units were failing, we would typically see an increase in error rates, kernel panics, or system crashes rather than high latency. Low utilization is generally a sign of waiting for work, not an inability to execute instructions.

Trap 2: The model is too large for memory

If the model were too large, the system would fail with an Out-of-Memory error. It would not manifest as high P99 latency with low utilization, as the model would not even be able to start the inference process.

Trap 3: Network bandwidth is saturated

While network saturation could cause latency, it would not explain the low GPU utilization unless the network was the only bottleneck. However, the presence of a high queue depth strongly implies the server is intentionally throttled, pointing to concurrency configuration.

A
The GPU compute units are failing
Why it fails: If GPU units were failing, we would typically see an increase in error rates, kernel panics, or system crashes rather than high latency. Low utilization is generally a sign of waiting for work, not an inability to execute instructions.
B
Inappropriate concurrency settings
Low GPU utilization combined with high queue depth confirms that the system is not processing enough concurrent requests to utilize the hardware fully. The bottleneck is the configuration limiting the number of parallel inferences, causing requests to queue unnecessarily.
C
The model is too large for memory
Why it fails: If the model were too large, the system would fail with an Out-of-Memory error. It would not manifest as high P99 latency with low utilization, as the model would not even be able to start the inference process.
D
Network bandwidth is saturated
Why it fails: While network saturation could cause latency, it would not explain the low GPU utilization unless the network was the only bottleneck. However, the presence of a high queue depth strongly implies the server is intentionally throttled, pointing to concurrency configuration.

Question 12easymultiple choice

You are evaluating a text generation model using NVIDIA NeMo Evaluation and want to measure how well the generated text matches a reference translation. Which metric is specifically designed for this purpose?

Trap 1: METEOR

METEOR is a translation evaluation metric that considers synonyms and stemming, aiming to improve correlation with human judgment. However, it is not the standard metric in NVIDIA NeMo Evaluation for translation; BLEU is more commonly used and supported. While METEOR is designed for translation, the question asks for the metric specifically designed for this purpose, which is BLEU. In practice, METEOR could be used, but BLEU is the canonical choice.

Trap 2: Perplexity

Perplexity measures the fluency of a language model by calculating the exponentiated average negative log-likelihood of a sequence. It does not compare against a reference translation and thus cannot assess translation accuracy. Therefore, perplexity is not designed for evaluating translation match, and it is unsuitable here.

Trap 3: ROUGE

ROUGE is primarily used for summarization evaluation, focusing on recall of n-grams or longest common subsequences against reference summaries. While it can be applied to translation, it is not specifically designed for that task and lacks the precision-oriented brevity penalty of BLEU. Thus, it is not the best choice for translation evaluation.

A
BLEU
BLEU (Bilingual Evaluation Understudy) is designed for machine translation evaluation. It computes n-gram precision between generated and reference translations, with a brevity penalty. It is the standard metric for translation quality and is directly applicable when a reference translation exists, making it the correct choice for this scenario.
B
METEOR
Why it fails: METEOR is a translation evaluation metric that considers synonyms and stemming, aiming to improve correlation with human judgment. However, it is not the standard metric in NVIDIA NeMo Evaluation for translation; BLEU is more commonly used and supported. While METEOR is designed for translation, the question asks for the metric specifically designed for this purpose, which is BLEU. In practice, METEOR could be used, but BLEU is the canonical choice.
C
Perplexity
Why it fails: Perplexity measures the fluency of a language model by calculating the exponentiated average negative log-likelihood of a sequence. It does not compare against a reference translation and thus cannot assess translation accuracy. Therefore, perplexity is not designed for evaluating translation match, and it is unsuitable here.
D
ROUGE
Why it fails: ROUGE is primarily used for summarization evaluation, focusing on recall of n-grams or longest common subsequences against reference summaries. While it can be applied to translation, it is not specifically designed for that task and lacks the precision-oriented brevity penalty of BLEU. Thus, it is not the best choice for translation evaluation.

Question 13hardmultiple choice

A team is evaluating a retrieval-augmented generation (RAG) pipeline using NVIDIA NeMo Evaluation. They notice that the generated answers are fluent but sometimes contradict the retrieved documents. Which evaluation approach best identifies this issue?

Trap 1: Compute BLEU score against reference answers.

BLEU compares n-gram overlap with reference answers, but references may not exist or may not reflect retrieved documents. Contradictions with retrieved context are not captured by BLEU, as it focuses on surface similarity to references, not factual consistency with source documents. Thus, it fails to detect the contradiction problem.

Trap 2: Calculate perplexity of the generated answers.

Perplexity measures language model confidence and fluency, not factual consistency with retrieved documents. A fluent but contradictory answer can have low perplexity. Therefore, perplexity cannot identify contradictions; it only indicates how predictable the text is under the model, which is unrelated to the retrieval grounding issue.

Trap 3: Measure the average length of retrieved documents.

Average document length is a retrieval statistic and does not assess the generated answer's fidelity to the documents. Contradictions can occur regardless of document length. Thus, this metric is irrelevant for detecting whether the generated answer contradicts the retrieved content, and it provides no insight into factual consistency.

A
Compute BLEU score against reference answers.
Why it fails: BLEU compares n-gram overlap with reference answers, but references may not exist or may not reflect retrieved documents. Contradictions with retrieved context are not captured by BLEU, as it focuses on surface similarity to references, not factual consistency with source documents. Thus, it fails to detect the contradiction problem.
B
Calculate perplexity of the generated answers.
Why it fails: Perplexity measures language model confidence and fluency, not factual consistency with retrieved documents. A fluent but contradictory answer can have low perplexity. Therefore, perplexity cannot identify contradictions; it only indicates how predictable the text is under the model, which is unrelated to the retrieval grounding issue.
C
Measure the average length of retrieved documents.
Why it fails: Average document length is a retrieval statistic and does not assess the generated answer's fidelity to the documents. Contradictions can occur regardless of document length. Thus, this metric is irrelevant for detecting whether the generated answer contradicts the retrieved content, and it provides no insight into factual consistency.
D
Use a natural language inference (NLI) model to check entailment between retrieved documents and generated answers.
NLI models determine whether a hypothesis (generated answer) is entailed by, neutral to, or contradicts a premise (retrieved document). By running an NLI model, you can flag contradictions directly. This approach is robust for RAG evaluation because it assesses factual consistency without requiring reference answers, aligning with the goal of detecting contradictions.

Question 14mediummultiple choice

A team is pretraining a decoder-only Transformer LLM on a large corpus of code and natural language. They observe that the model's training loss decreases smoothly, but during generation it sometimes produces degenerate repetition, and attention entropy on long sequences collapses. They suspect the issue is related to the positional encoding scheme. Which architectural change is most likely to mitigate the attention entropy collapse while preserving the model's ability to generalize to sequences longer than those seen during pretraining?

Trap 1: Increase the number of attention heads while keeping the total…

Adding more heads with a fixed hidden dimension reduces the per-head dimension, which can change representational capacity but does not alter how position is encoded. The root cause described is positional extrapolation and entropy collapse, not head count. This change is unlikely to fix repetition on long sequences and may even degrade per-head expressiveness.

Trap 2: Apply Layer Normalization before the self-attention sublayer…

Pre-LN versus post-LN affects gradient flow and training stability, but it does not change the positional encoding mechanism. The scenario attributes the repetition and attention entropy collapse to positional handling during long-sequence generation. Reordering normalization alone will not provide relative-position generalization or prevent entropy collapse when extrapolating beyond the pretraining length.

Trap 3: Add a sinusoidal positional encoding to the input embeddings and…

Sinusoidal encodings are fixed and added only at the input, so they do not provide the relative-position inductive bias inside each attention layer. Freezing input embeddings further limits the model's ability to adapt position representations to the code and language mixture. This does not address entropy collapse in deep attention layers or improve length generalization in the way the scenario requires.

A
Increase the number of attention heads while keeping the total hidden dimension constant, without changing the positional encoding.
Why it fails: Adding more heads with a fixed hidden dimension reduces the per-head dimension, which can change representational capacity but does not alter how position is encoded. The root cause described is positional extrapolation and entropy collapse, not head count. This change is unlikely to fix repetition on long sequences and may even degrade per-head expressiveness.
B
Apply Layer Normalization before the self-attention sublayer instead of after it, leaving the positional encoding unchanged.
Why it fails: Pre-LN versus post-LN affects gradient flow and training stability, but it does not change the positional encoding mechanism. The scenario attributes the repetition and attention entropy collapse to positional handling during long-sequence generation. Reordering normalization alone will not provide relative-position generalization or prevent entropy collapse when extrapolating beyond the pretraining length.
C
Replace learned absolute positional embeddings with Rotary Positional Embeddings (RoPE) applied to queries and keys in every self-attention layer.
RoPE encodes relative position by rotating query and key vectors by an angle proportional to their absolute position. Because attention scores depend only on the relative rotation between queries and keys, the model naturally generalizes to longer contexts and avoids the entropy collapse seen when learned absolute embeddings are extrapolated beyond their trained range. This matches the observed repetition and entropy issues.
D
Add a sinusoidal positional encoding to the input embeddings and freeze those embeddings during training.
Why it fails: Sinusoidal encodings are fixed and added only at the input, so they do not provide the relative-position inductive bias inside each attention layer. Freezing input embeddings further limits the model's ability to adapt position representations to the code and language mixture. This does not address entropy collapse in deep attention layers or improve length generalization in the way the scenario requires.

Question 15mediummultiple choice

Which of the following describes the purpose of 'gradient accumulation' in fine-tuning scenarios?

Trap 1: To increase the speed of the training process by bypassing the…

Gradient accumulation does not increase speed; it often slightly slows down the wall-clock time because it involves more forward passes before the weight update occurs. Its primary function is to manage memory effectively, not to provide a performance boost in terms of raw throughput or training speed.

Trap 2: To automatically optimize the learning rate based on the model's…

Gradient accumulation is independent of learning rate optimization strategies. While it influences how gradients are applied, it does not dynamically adjust the learning rate. Learning rate scheduling is typically handled by separate mechanisms like warm-up cycles or cosine annealing, which manage the step size during the training process.

Trap 3: To reduce the number of parameters being fine-tuned in the model.

Gradient accumulation does not change the number of trainable parameters. Techniques that affect parameter count, such as LoRA or pruning, are distinct from gradient accumulation. The primary objective of this technique is to control the training dynamics via batch size, not to compress the underlying model size.

A
To increase the speed of the training process by bypassing the GPU's bottleneck.
Why it fails: Gradient accumulation does not increase speed; it often slightly slows down the wall-clock time because it involves more forward passes before the weight update occurs. Its primary function is to manage memory effectively, not to provide a performance boost in terms of raw throughput or training speed.
B
To enable training with larger effective batch sizes on memory-constrained hardware.
This technique allows engineers to maintain effective batch sizes that would otherwise exceed available VRAM. By accumulating gradients over multiple steps and updating weights only after reaching a target batch size, the training process achieves the stability of large-batch learning while keeping the peak memory usage manageable.
C
To automatically optimize the learning rate based on the model's gradient magnitude.
Why it fails: Gradient accumulation is independent of learning rate optimization strategies. While it influences how gradients are applied, it does not dynamically adjust the learning rate. Learning rate scheduling is typically handled by separate mechanisms like warm-up cycles or cosine annealing, which manage the step size during the training process.
D
To reduce the number of parameters being fine-tuned in the model.
Why it fails: Gradient accumulation does not change the number of trainable parameters. Techniques that affect parameter count, such as LoRA or pruning, are distinct from gradient accumulation. The primary objective of this technique is to control the training dynamics via batch size, not to compress the underlying model size.