NCA-GENL Practice Exam 3 — 20 Questions

Question 1

An AI engineer is conducting an experiment to compare two different fine-tuning approaches for a large language model using NVIDIA NeMo: full fine-tuning versus parameter-efficient fine-tuning (PEFT) with LoRA. The engineer wants to determine which approach yields better performance on a downstream question-answering task while minimizing computational cost. Which metric should the engineer prioritize to evaluate the trade-off between performance and cost?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Validation accuracy per GPU-hour. Validation accuracy per GPU-hour effectively combines the two objectives: it measures the model's performance on the question-answering task (validation accuracy) and normalizes it by the computational cost (GPU-hours). This allows the engineer to compare full fine-tuning and LoRA on a level playing field, identifying which method provides the best accuracy for the resources invested. It directly supports the goal of minimizing cost while maximizing performance.

Answer

Training loss convergence rate

Answer

Number of trainable parameters

Answer

Inference latency on a CPU

Question 2

Which of the following describes the 'Stop-Loss' technique in the context of LLM experimentation?

Accepted Answer

Terminating training if loss does not improve. Stop-loss is an essential technique for managing compute costs. By monitoring training metrics in real-time, the system automatically halts experiments that show no sign of convergence or diverge prematurely. This prevents the waste of expensive GPU resources on doomed runs, allowing researchers to reallocate capacity to more promising experiments and improving the overall efficiency of the research team's pipeline.

Answer

Stopping training after a fixed number of days

Answer

Increasing the learning rate when loss is high

Answer

Saving a checkpoint every 100 iterations

Question 3

A team is evaluating an LLM for a customer-support summarization task. They want to compare three prompt templates. Which experimental design most directly isolates the effect of the prompt template?

Accepted Answer

Use the same model, decoding parameters, and evaluation dataset for all three templates, changing only the template text.. A controlled comparison requires that only the variable of interest changes between conditions. By fixing the model, decoding parameters, and evaluation data and varying only the prompt template text, the team ensures that observed differences in summary quality are attributable to the template. This makes the experiment reproducible and the conclusions defensible.

Answer

Use a different model for each template to see which combination performs best overall.

Answer

Vary temperature and top-p across templates so each template is tested under its own best decoding settings.

Answer

Evaluate each template on a different dataset to cover more customer scenarios.

Question 4

Which THREE factors influence the reproducibility of an LLM experiment?

Accepted Answer

Random seed initialization. Reproducibility requires strict control over the experimental environment. Because LLM training involves non-deterministic factors, controlling the random seeds, the exact data splits, and the library versions is paramount. Professional experiment tracking requires the ability to recreate identical results, which is only possible when every component of the pipeline is version-controlled and explicitly defined in the experiment's configuration manifest.

Answer

The ambient temperature of the datacenter

Answer

The number of hours the researcher works

Question 5

A team trains a transformer language model on a large corpus but the model achieves very low training loss while performing poorly on held-out text. Which action most directly addresses this outcome?

Accepted Answer

Apply dropout and weight decay to regularize the model. Low training loss combined with weak held-out performance indicates overfitting, meaning the model memorizes training text instead of learning generalizable patterns. Regularization techniques such as dropout and weight decay constrain model capacity and penalize reliance on specific training examples, which directly reduces the train-validation gap and improves performance on unseen text.

Answer

Lower the learning rate for the final training steps

Answer

Increase the number of training epochs on the same corpus

Answer

Reduce the size of the training corpus

Question 6

Which memory management strategy in TensorRT-LLM is specifically designed to minimize fragmentation and allow for efficient KV cache allocation in multi-user environments?

Accepted Answer

PagedAttention.. PagedAttention is the core innovation here. It manages the Key-Value (KV) cache in fixed-size blocks, similar to how an operating system manages virtual memory. By treating the KV cache as non-contiguous memory, TensorRT-LLM can efficiently allocate and reclaim space on the fly. This prevents the memory fragmentation that usually occurs with static allocation, allowing for higher concurrency and supporting more simultaneous users without running out of GPU memory.

Answer

Static contiguous memory allocation.

Answer

Global unified memory pooling.

Answer

Dynamic weight quantization.

Question 7

During a fine-tuning experiment in NVIDIA NeMo, validation loss begins to rise after epoch 4 while training loss continues to fall. The team wants to determine the earliest epoch at which the model still generalizes well. Which experimental action is most appropriate?

Accepted Answer

Enable checkpoint saving at every epoch and select the checkpoint with the lowest validation loss for downstream evaluation.. The divergence between falling training loss and rising validation loss indicates the model is memorizing training data. The practical remedy in an experimentation context is to checkpoint each epoch and choose the model with the lowest validation loss, which corresponds to the point of best generalization. This yields the earliest useful epoch without altering the training dynamics.

Answer

Train for more epochs to let validation loss eventually decrease again.

Answer

Increase the learning rate so the model escapes the overfitting region faster.

Answer

Reduce the size of the validation set so the measured validation loss becomes less noisy.

Question 8

An ML engineer at a healthcare analytics company is starting a fine-tuning experiment on a Llama 2 7B model using NVIDIA NeMo Framework. Before launching the training job, the engineer wants a single immutable record that captures the exact model checkpoint, dataset version, hyperparameters, and evaluation scores so that any later run can be traced back to it. Which component of the NVIDIA NeMo experimentation workflow should the engineer use to store that record?

Accepted Answer

NeMo Experiment Manager. The requirement is a traceable record combining model checkpoint, dataset version, hyperparameters, and evaluation metrics. NeMo Experiment Manager is built to log and organize exactly that metadata during fine-tuning, enabling reproducibility and comparison across runs. Inference servers, inference compilers, and profilers serve different purposes and do not persist the training experiment lineage needed here.

Answer

NVIDIA Triton Inference Server model repository

Answer

NVIDIA TensorRT-LLM build configuration

Answer

NVIDIA Nsight Systems profiling report

Question 9

You are analyzing token probability distributions from an LLM inference service to detect hallucination risk. You need a single visualization that shows, for one generated response, how the model's confidence evolved token-by-token and where it suddenly dropped. Which visualization is most appropriate?

Accepted Answer

A line chart of the top-1 token probability across generation steps, with a marked threshold line.. Tracking the top-1 token probability across generation steps preserves sequence order and directly encodes model confidence at each step. Adding a threshold line turns the chart into an alerting view where abrupt dips flag possible hallucination regions. Aggregated or embedding-based views lose the temporal resolution needed for this per-response diagnosis.

Answer

A confusion matrix of predicted versus reference tokens for the entire batch.

Answer

A t-SNE scatter plot of the final hidden state embeddings for the batch.

Answer

A histogram of maximum token probabilities across all requests in the last hour.

Question 10

When conducting A/B testing on an LLM-based application, which metric is the most reliable indicator of user satisfaction regarding response quality?

Accepted Answer

LLM-as-a-Judge score. Human-in-the-loop evaluation or automated model-based grading (LLM-as-a-Judge) are the gold standards for response quality. Unlike latency or throughput, which measure system performance, quality metrics quantify the utility and accuracy of the generated text. In AI experimentation, balancing technical performance with qualitative user feedback is essential to ensure that system optimizations do not inadvertently degrade the helpfulness of the model output.

Answer

Inference latency

Answer

Tokens per second

Answer

GPU memory utilization

Question 11

Refer to the exhibit. A developer implements this NVIDIA NeMo Guardrails configuration. A user submits a query about financial advice. What is the expected behavior of the LLM?

Accepted Answer

The guardrail blocks the request because finance is not in the whitelist.. The configured guardrail includes topic whitelisting, which restricts the model to only discussing 'tech' and 'science'. When the user submits a financial query, the system identifies the topic mismatch. Since the policy does not explicitly allow financial topics, the guardrail intercepts the input, preventing the model from generating a response. This proactive filtering is a key component of trustworthy AI, ensuring models operate within predefined domain boundaries.

Answer

The model generates a generic response about finance using its internal knowledge.

Answer

The system processes the request because the toxicity score is below 0.85.

Answer

The model prompts the user to clarify if the finance query is related to technology.

Question 12

When designing an experiment to evaluate the performance of an LLM on a downstream classification task, which THREE factors should be controlled to ensure the results are comparable across different model sizes?

Accepted Answer

Decoding strategy (temperature, top-p). To conduct valid cross-model comparisons, one must neutralize external variables. Inference-time settings like decoding parameters, the specific prompt template, and the evaluation dataset must remain constant. If these vary, the results reflect differences in the evaluation environment rather than the intrinsic capabilities of the models being tested, rendering the experimentation data inconclusive for determining which model size is truly optimal for the specific classification application.

Answer

The learning rate used during fine-tuning

Answer

The hardware GPU generation (e.g., A100 vs H100)

Question 13

A developer is writing a Python service that calls an NVIDIA-hosted NIM endpoint for a Llama model. The service must recover gracefully when the endpoint returns HTTP 429 responses during peak traffic, without dropping user requests. Which implementation approach best satisfies this requirement?

Accepted Answer

Implement exponential backoff with jitter and a bounded retry count, then surface a fallback error to the caller.. Rate-limit responses require the client to slow down and retry deliberately. Exponential backoff with jitter spreads retry attempts so multiple clients do not collide, and a bounded retry count keeps latency predictable. Because some requests may still fail after all retries, returning a controlled fallback error preserves a defined service contract for callers rather than hanging or crashing.

Answer

Retry the request in a tight loop immediately after each 429 response until it succeeds.

Answer

Increase the client's request timeout value so the 429 responses stop occurring.

Answer

Switch the endpoint URL to a different NIM model and continue sending the same request payload unchanged.

Question 14

A team is comparing two LLM fine-tuning runs on the same dataset. Run A used a cosine learning-rate schedule, and Run B used a constant learning rate. They plot validation loss versus training step for both runs on the same axes. Run A's curve is smooth, while Run B's curve shows a sharp upward spike around step 800 and then recovers. The team wants to determine whether the spike in Run B indicates a data-order artifact or a genuine optimization instability. Which additional visualization is most useful for that diagnosis?

Accepted Answer

A scatter plot of per-batch training loss for Run B colored by batch index modulo epoch length. Plotting per-batch training loss with color encoding the batch's position modulo the epoch length exposes whether the spike recurs at a fixed data position. A recurring pattern implies a data-order artifact, such as a hard shard or mislabeled examples, while a random pattern implies optimization instability. Summary charts and hardware metrics do not preserve the temporal and data-order information needed for this diagnosis.

Answer

A bar chart of final validation loss for Run A and Run B

Answer

A heatmap of attention weights for one validation example

Answer

A line chart of GPU utilization for Run A and Run B

Question 15

An AI engineer at a financial services company is running an LLM experimentation pipeline using NVIDIA NeMo. The primary objective is to evaluate how different tokenizers affect the accuracy of a named entity recognition (NER) task on financial documents. The engineer has already fixed the model architecture, the training dataset, and the hyperparameters. To ensure the experiment isolates the effect of the tokenizer, which action should the engineer take next?

Accepted Answer

Replace the tokenizer with a different one while keeping all other configurations unchanged, then compare NER accuracy.. To isolate the effect of the tokenizer, the engineer must change only the tokenizer while keeping all other factors constant. This controlled approach ensures that any observed difference in NER accuracy is due to tokenization rather than confounding variables like model architecture or dataset size. Such isolation is fundamental to valid experimentation and enables clear conclusions about tokenizer impact.

Answer

Increase the training dataset size to improve NER accuracy before changing the tokenizer.

Answer

Fine-tune the model on a general-domain corpus before evaluating on financial documents.

Answer

Retrain the model with the same tokenizer but different random seeds to measure variance.

Question 16

Which TWO factors should be considered when evaluating the cost-benefit of an LLM experimentation strategy?

Accepted Answer

Total GPU hours per experiment. Evaluating the cost-benefit of LLM experimentation requires balancing the financial cost of compute with the performance gains observed. In professional environments, experiments must be scoped to maximize meaningful insights while minimizing wasted GPU cycles. This involves prioritizing experiments that are likely to yield the highest impact on model quality relative to the resources consumed by the training runs.

Answer

The color scheme of the monitoring dashboard

Answer

The popularity of the LLM framework used

Answer

The number of research papers published by the team

Question 17

Which of the following describes the 'Warm-up' phase in the context of training deep neural networks?

Accepted Answer

Gradually increasing the learning rate at the start of training.. Warm-up is a technique where the learning rate starts at a very low value and is gradually increased over the first few hundred or thousand steps. This prevents the model from diverging early in the training process when weights are randomly initialized and gradients can be unstable. Proper warm-up is essential for successfully scaling the training of modern LLMs on large compute clusters, ensuring stable convergence from the very first iterations.

Answer

Pre-loading the model into the GPU cache.

Answer

A method to reduce training time by caching gradients.

Answer

Reducing the batch size to fit in memory.

Question 18

An application requires streaming responses from a deployed LLM. Which communication protocol is most suitable for minimizing latency and ensuring efficient data delivery in a real-time generative AI application?

Accepted Answer

gRPC with server-side streaming.. Streaming generative responses requires a protocol that supports persistent connections and low-overhead message framing. gRPC with server-side streaming is the industry standard for high-performance AI inference, as it facilitates efficient binary serialization and maintains low latency across the network. Using this protocol is critical for user-facing applications where perceived latency is directly tied to the speed at which text tokens appear on the screen.

Answer

HTTP/1.1 REST without streaming support.

Answer

Standard SMTP email protocols.

Answer

Synchronous SQL query polling.

Question 19

An ML engineer is deploying a Transformer-based inference service on an NVIDIA TensorRT-LLM runtime. To maximize inference throughput and reduce latency under heavy concurrent user traffic, the engineer needs to select the optimal decoding batching strategy. Which technique allows multiple incoming dynamic sequence requests to be batched together at the token level rather than waiting for entire sequences to complete?

Accepted Answer

Inflight batching (continuous batching) that schedules individual generation iterations at the token level across requests.. Inflight batching (also known as continuous batching) dynamically groups individual generation steps from different requests at each token generation iteration. This eliminates the idle padding overhead and waiting times associated with traditional static batching, maximizing GPU compute utilization in LLM serving.

Answer

Static request batching that groups fixed numbers of incoming inference prompts and pads them to equal sequence lengths.

Answer

Data-parallel replica sharding that duplicates the model across separate GPU devices to process independent streams.

Answer

Model parallelism that splits individual transformer weight matrices across multiple interconnected GPU devices.

Question 20

A developer is using TensorRT-LLM to build a chatbot and wants to reduce the memory footprint of the KV cache during inference. Which technique should they use?

Accepted Answer

Enable paged KV cache. Paged KV cache is a memory management technique in TensorRT-LLM that divides the KV cache into fixed-size blocks, allowing non-contiguous storage and dynamic allocation. This reduces fragmentation and enables more sequences to fit in memory. It is the recommended approach to minimize KV cache memory footprint during inference.

Answer

Disable quantization

Answer

Increase the beam width

Answer

Use a larger batch size