NCA-GENL Practice Test 2 — 30 Questions

Question 1

A team has fine-tuned a small LLM with NeMo and now wants to quantify how much the fine-tuning improved performance on a domain question-answering task relative to the base model. They have a curated set of 500 question-answer pairs that were never used during training. What is the most appropriate next step?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Evaluate both the base model and the fine-tuned model on the held-out 500-question set using the same metric and decoding settings.. Quantifying the benefit of fine-tuning requires an unbiased comparison between the adapted model and its base counterpart on data neither has seen in training. Using the same held-out questions, prompts, decoding settings, and metric for both models yields a clean estimate of the improvement. Training loss, training-set accuracy, or a second training run do not measure generalization gain.

Answer

Run the fine-tuned model on the original training questions and compare its answers to the stored labels.

Answer

Fine-tune the base model a second time with a different seed and compare the two training curves.

Answer

Report the final training loss of the fine-tuned model as the measure of improvement.

Question 2

In the context of NVIDIA's Tensor Core architecture, what is the primary purpose of 'Sparsity' support?

Accepted Answer

To double the computational throughput during GEMM operations.. NVIDIA's Fine-Grained Structured Sparsity is a hardware feature that allows the GPU to skip computations for zero-valued weights in neural networks. By enforcing a 2:4 sparsity pattern (where at least 2 out of every 4 consecutive weights are zero), the hardware can double the throughput of matrix multiplication operations. This feature is vital for accelerating LLMs without sacrificing model accuracy, as it effectively doubles compute density on supported modern NVIDIA architectures.

Answer

To reduce the amount of VRAM consumed by model weights.

Answer

To automatically prune the model during the training process.

Answer

To allow models to run without any normalization layers.

Question 3

A retail company uses an LLM to generate product descriptions. A reviewer notices that descriptions for kitchen knives are consistently written in a more aggressive tone than descriptions for other product categories, and that the model refuses to describe certain cultural cookware items at all. The team wants to understand which trustworthiness property is most directly implicated by these observations.

Accepted Answer

Model fairness, because the model produces systematically different treatment across product categories and cultural items.. The observations describe systematically different treatment across product categories and cultural items, which is the defining symptom of a fairness problem. Fairness focuses on whether a model applies consistent, equitable standards across groups. Other properties such as latency, provenance, and interpretability may be relevant to a broader investigation, but they do not directly name the unequal-treatment behavior the reviewer observed.

Answer

Data provenance, because the company cannot trace which supplier provided the product catalog.

Answer

Model interpretability, because the team cannot read the model's internal attention weights.

Answer

Model latency, because refusal behavior indicates the inference server is timing out on certain prompts.

Question 4

A team is pre-training a 7-billion-parameter LLM on a large text corpus. They observe that the training loss decreases steadily but the validation loss begins to increase after a certain number of steps. The training and validation data come from the same distribution, and the model has not yet reached the compute budget. Which action is most appropriate to address this behavior?

Accepted Answer

Apply regularization techniques such as dropout or weight decay, and consider early stopping.. A widening gap between decreasing training loss and increasing validation loss signals overfitting. The model is memorizing training-specific patterns rather than generalizing. Applying regularization like dropout or weight decay reduces this tendency, and early stopping captures the best validation checkpoint. These steps are standard and directly target the observed behavior without unnecessary architectural changes.

Answer

Reduce the model size to match the dataset size.

Answer

Increase the learning rate to escape the local minimum.

Answer

Collect more training data from the same distribution.

Question 5

A team is running an LLM fine-tuning experiment using NVIDIA NeMo and wants to track how the validation loss changes over training. They need a reliable way to detect overfitting early. Which metric should they monitor most directly during the experiment?

Accepted Answer

Validation loss computed on a held-out dataset at regular intervals during training.. Overfitting is characterized by a growing gap between training and validation performance. Validation loss on a held-out set is the most direct indicator: when it stops improving and begins to rise while training loss keeps falling, the model is memorizing rather than generalizing. Monitoring it during training lets the team intervene early with regularization or early stopping.

Answer

Training loss computed on the same data batches used for gradient updates.

Answer

GPU utilization percentage reported by the NeMo training logs.

Answer

The number of tokens processed per second during training.

Question 6

What is the primary function of the 'Softmax' layer at the output of a multi-class classification model?

Accepted Answer

To convert logits into a probability distribution.. The Softmax function converts the raw output scores (logits) of the final layer into a probability distribution. This involves exponentiating each score and normalizing by the sum of all exponentiated scores, ensuring all outputs are in the range (0, 1) and sum to exactly 1.0. This makes the model's output directly interpretable as probabilities for each class, which is vital for decision-making tasks where confidence scores are needed.

Answer

To squash output values to a range of -1 to 1.

Answer

To linearize the output for regression tasks.

Answer

To enforce sparsity on the output vector.

Question 7

When evaluating an LLM for a domain-specific task, why is 'Few-Shot Prompting' often superior to 'Zero-Shot Prompting'?

Accepted Answer

It provides in-context learning examples to guide the model.. Few-shot prompting provides the model with concrete examples of the desired input-output mapping, which acts as a guide for structure and reasoning. This reduces ambiguity and aligns the model's output with application-specific requirements. It is a fundamental technique for improving model precision in specialized domains where the model needs to understand unique formatting or logical patterns that are not explicitly defined in its base training data.

Answer

It decreases the number of tokens processed per request.

Answer

It forces the model to ignore its internal pre-trained knowledge.

Answer

It is the only method to ensure the model produces non-hallucinated results.

Question 8

A developer is using the NVIDIA API Catalog to experiment with a hosted LLM. They want to send a prompt and receive a completion. Which endpoint should they use?

Accepted Answer

/v1/chat/completions. The /v1/chat/completions endpoint is designed for conversational LLMs and accepts a structured messages array. It is the primary interface for interacting with hosted models on NVIDIA API Catalog. Other endpoints serve different purposes like listing models or generating embeddings, and the legacy completions endpoint is less suitable for chat-based models.

Answer

/v1/models

Answer

/v1/completions

Answer

/v1/embeddings

Question 9

You are monitoring a production LLM inference service on an NVIDIA GPU. The service's request latency distribution is heavily right-skewed, and a small fraction of requests take far longer than the rest. You need a visualization that shows the full distribution shape, including the median and the extreme tail, to decide whether the GPU is under-provisioned. Which visualization should you use?

Accepted Answer

A box plot of request latency with whiskers extending to the 1.5 IQR and outliers plotted individually.. Because the latency distribution is right-skewed, summary statistics like the mean hide the tail. A box plot exposes the median, quartiles, and individual outliers, so the extreme slow requests remain visible. This lets you decide whether the long tail is severe enough to justify additional GPU capacity.

Answer

A single line chart of mean request latency sampled every minute.

Answer

A pie chart showing the proportion of requests in each latency bucket.

Answer

A scatter plot of GPU utilization against time for the last 24 hours.

Question 10

Which of the following best defines 'Generalization' in machine learning?

Accepted Answer

The model's performance on previously unseen data.. Generalization is the model's ability to perform accurately on new, unseen data that was not part of the training set. A model that performs perfectly on training data but fails on new data has failed to generalize. Achieving high generalization is the ultimate goal of all machine learning, as it ensures that the model provides value in real-world scenarios rather than just memorizing input-output patterns from a specific training batch.

Answer

The ability of a model to memorize training data perfectly.

Answer

The process of reducing the model's parameter count.

Answer

The speed at which a model converges during training.

Question 11

An AI engineer is conducting an experiment to compare two different fine-tuning approaches for a large language model using NVIDIA NeMo: full fine-tuning versus parameter-efficient fine-tuning (PEFT) with LoRA. The engineer wants to determine which approach yields better performance on a downstream question-answering task while minimizing computational cost. Which metric should the engineer prioritize to evaluate the trade-off between performance and cost?

Accepted Answer

Validation accuracy per GPU-hour. Validation accuracy per GPU-hour effectively combines the two objectives: it measures the model's performance on the question-answering task (validation accuracy) and normalizes it by the computational cost (GPU-hours). This allows the engineer to compare full fine-tuning and LoRA on a level playing field, identifying which method provides the best accuracy for the resources invested. It directly supports the goal of minimizing cost while maximizing performance.

Answer

Training loss convergence rate

Answer

Number of trainable parameters

Answer

Inference latency on a CPU

Question 12

Which of the following describes the 'Stop-Loss' technique in the context of LLM experimentation?

Accepted Answer

Terminating training if loss does not improve. Stop-loss is an essential technique for managing compute costs. By monitoring training metrics in real-time, the system automatically halts experiments that show no sign of convergence or diverge prematurely. This prevents the waste of expensive GPU resources on doomed runs, allowing researchers to reallocate capacity to more promising experiments and improving the overall efficiency of the research team's pipeline.

Answer

Stopping training after a fixed number of days

Answer

Increasing the learning rate when loss is high

Answer

Saving a checkpoint every 100 iterations

Question 13

A team is evaluating an LLM for a customer-support summarization task. They want to compare three prompt templates. Which experimental design most directly isolates the effect of the prompt template?

Accepted Answer

Use the same model, decoding parameters, and evaluation dataset for all three templates, changing only the template text.. A controlled comparison requires that only the variable of interest changes between conditions. By fixing the model, decoding parameters, and evaluation data and varying only the prompt template text, the team ensures that observed differences in summary quality are attributable to the template. This makes the experiment reproducible and the conclusions defensible.

Answer

Use a different model for each template to see which combination performs best overall.

Answer

Vary temperature and top-p across templates so each template is tested under its own best decoding settings.

Answer

Evaluate each template on a different dataset to cover more customer scenarios.

Question 14

Which THREE factors influence the reproducibility of an LLM experiment?

Accepted Answer

Random seed initialization. Reproducibility requires strict control over the experimental environment. Because LLM training involves non-deterministic factors, controlling the random seeds, the exact data splits, and the library versions is paramount. Professional experiment tracking requires the ability to recreate identical results, which is only possible when every component of the pipeline is version-controlled and explicitly defined in the experiment's configuration manifest.

Answer

The ambient temperature of the datacenter

Answer

The number of hours the researcher works

Question 15

A team trains a transformer language model on a large corpus but the model achieves very low training loss while performing poorly on held-out text. Which action most directly addresses this outcome?

Accepted Answer

Apply dropout and weight decay to regularize the model. Low training loss combined with weak held-out performance indicates overfitting, meaning the model memorizes training text instead of learning generalizable patterns. Regularization techniques such as dropout and weight decay constrain model capacity and penalize reliance on specific training examples, which directly reduces the train-validation gap and improves performance on unseen text.

Answer

Lower the learning rate for the final training steps

Answer

Increase the number of training epochs on the same corpus

Answer

Reduce the size of the training corpus

Question 16

Which memory management strategy in TensorRT-LLM is specifically designed to minimize fragmentation and allow for efficient KV cache allocation in multi-user environments?

Accepted Answer

PagedAttention.. PagedAttention is the core innovation here. It manages the Key-Value (KV) cache in fixed-size blocks, similar to how an operating system manages virtual memory. By treating the KV cache as non-contiguous memory, TensorRT-LLM can efficiently allocate and reclaim space on the fly. This prevents the memory fragmentation that usually occurs with static allocation, allowing for higher concurrency and supporting more simultaneous users without running out of GPU memory.

Answer

Static contiguous memory allocation.

Answer

Global unified memory pooling.

Answer

Dynamic weight quantization.

Question 17

During a fine-tuning experiment in NVIDIA NeMo, validation loss begins to rise after epoch 4 while training loss continues to fall. The team wants to determine the earliest epoch at which the model still generalizes well. Which experimental action is most appropriate?

Accepted Answer

Enable checkpoint saving at every epoch and select the checkpoint with the lowest validation loss for downstream evaluation.. The divergence between falling training loss and rising validation loss indicates the model is memorizing training data. The practical remedy in an experimentation context is to checkpoint each epoch and choose the model with the lowest validation loss, which corresponds to the point of best generalization. This yields the earliest useful epoch without altering the training dynamics.

Answer

Train for more epochs to let validation loss eventually decrease again.

Answer

Increase the learning rate so the model escapes the overfitting region faster.

Answer

Reduce the size of the validation set so the measured validation loss becomes less noisy.

Question 18

An ML engineer at a healthcare analytics company is starting a fine-tuning experiment on a Llama 2 7B model using NVIDIA NeMo Framework. Before launching the training job, the engineer wants a single immutable record that captures the exact model checkpoint, dataset version, hyperparameters, and evaluation scores so that any later run can be traced back to it. Which component of the NVIDIA NeMo experimentation workflow should the engineer use to store that record?

Accepted Answer

NeMo Experiment Manager. The requirement is a traceable record combining model checkpoint, dataset version, hyperparameters, and evaluation metrics. NeMo Experiment Manager is built to log and organize exactly that metadata during fine-tuning, enabling reproducibility and comparison across runs. Inference servers, inference compilers, and profilers serve different purposes and do not persist the training experiment lineage needed here.

Answer

NVIDIA Triton Inference Server model repository

Answer

NVIDIA TensorRT-LLM build configuration

Answer

NVIDIA Nsight Systems profiling report

Question 19

You are analyzing token probability distributions from an LLM inference service to detect hallucination risk. You need a single visualization that shows, for one generated response, how the model's confidence evolved token-by-token and where it suddenly dropped. Which visualization is most appropriate?

Accepted Answer

A line chart of the top-1 token probability across generation steps, with a marked threshold line.. Tracking the top-1 token probability across generation steps preserves sequence order and directly encodes model confidence at each step. Adding a threshold line turns the chart into an alerting view where abrupt dips flag possible hallucination regions. Aggregated or embedding-based views lose the temporal resolution needed for this per-response diagnosis.

Answer

A confusion matrix of predicted versus reference tokens for the entire batch.

Answer

A t-SNE scatter plot of the final hidden state embeddings for the batch.

Answer

A histogram of maximum token probabilities across all requests in the last hour.

Question 20

When conducting A/B testing on an LLM-based application, which metric is the most reliable indicator of user satisfaction regarding response quality?

Accepted Answer

LLM-as-a-Judge score. Human-in-the-loop evaluation or automated model-based grading (LLM-as-a-Judge) are the gold standards for response quality. Unlike latency or throughput, which measure system performance, quality metrics quantify the utility and accuracy of the generated text. In AI experimentation, balancing technical performance with qualitative user feedback is essential to ensure that system optimizations do not inadvertently degrade the helpfulness of the model output.

Answer

Inference latency

Answer

Tokens per second

Answer

GPU memory utilization