NCA-GENL Practice Test 13 — 15 Questions

Question 1

A data scientist is preparing an LLM fine-tuning experiment on NVIDIA NeMo and wants every run to be reproducible weeks later. The team's experiment tracker currently logs only the final validation loss. Which additional item is most important to record so a run can be reproduced exactly?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

The random seed, framework version, and full training configuration. Exact reproduction of an LLM experiment depends on controlling every stochastic and environmental input. The seed governs initialization and data ordering, the framework and CUDA versions govern kernel behavior and numerics, and the full configuration governs optimization. Logging only a final metric captures an outcome but discards the recipe, so the run cannot be recreated reliably.

Answer

The wall-clock duration of the final epoch only

Answer

A screenshot of the loss curve rendered in the dashboard

Answer

The names of the engineers who launched each training job

Question 2

An engineer must decide how to split a labeled dataset of 50,000 customer support conversations before fine-tuning a NeMo LLM for intent classification. The goal is an honest estimate of how the tuned model will behave on never-before-seen tickets once deployed. Which splitting approach best supports that goal?

Accepted Answer

Randomly assign 70 percent to training, 15 percent to validation, and 15 percent to a test set that is touched only once at the end.. An untouched holdout test set is the only split that yields an unbiased generalization estimate, because it never influences training or hyperparameter choices. Pairing it with a validation split lets the team tune decisions without contaminating the final measurement, and random assignment keeps class balance comparable across splits.

Answer

Use all 50,000 conversations for training and rely on the training loss curve to judge generalization.

Answer

Tune hyperparameters repeatedly against the test split until accuracy is maximized.

Answer

Split the data by conversation length, putting the longest 15 percent in the test set.

Question 3

A team is comparing two fine-tuned variants of the same base LLM for a customer-support summarization task. Variant A was trained on 10,000 examples and variant B on 2,000 examples drawn from the same distribution. To make a fair comparison of their generalization, which evaluation practice should the team adopt?

Accepted Answer

Evaluate both variants on the same held-out test set that neither saw during training. Fair comparison requires controlling the evaluation data. When both variants are scored on the same held-out set that neither encountered during training, differences in their scores reflect the training choices rather than variation in test difficulty. Training loss and parameter counts are process metrics that do not measure generalization to unseen customer-support summaries.

Answer

Report the training loss of each variant at the final step

Answer

Compare the number of parameters each variant updated during fine-tuning

Answer

Let each variant be scored on its own randomly split test set

Question 4

A team is designing a controlled experiment to measure whether increasing LoRA rank improves instruction-following accuracy on a held-out benchmark. They want the comparison to be scientifically valid. Which experimental design choice best supports a valid conclusion?

Accepted Answer

Hold the base model, dataset, prompt template, and evaluation harness fixed while varying only LoRA rank.. A scientifically valid comparison changes one independent variable while holding everything else constant. Here, LoRA rank is the factor under test, so the base model, training data, prompt template, and evaluation harness must remain identical across runs. Using a shared held-out benchmark and reporting aggregate accuracy across seeds ensures observed differences reflect rank rather than confounding factors or evaluation noise.

Answer

Change LoRA rank and the base model simultaneously so the experiment covers more configurations in one run.

Answer

Report only the best accuracy observed across several random seeds for each rank.

Answer

Evaluate each rank on a freshly sampled held-out set to reduce any dataset bias.

Question 5

A research team is running a hyperparameter sweep over learning rate and warmup steps for a NeMo fine-tuning job. They notice that runs with identical hyperparameters produce different final validation losses across repeated executions. Which two changes would most directly improve the reproducibility of these experiments? (Choose two.)

Accepted Answer

Enable deterministic kernels and disable nondeterministic cuDNN algorithms in the framework settings.. Run-to-run variance in identical configurations usually comes from uncontrolled randomness in data order, initialization, dropout, and nondeterministic GPU kernels. Fixing the random seed and enabling deterministic kernels address both the algorithmic and hardware-level sources. Adding GPUs, logging, or lowering the learning rate change the experiment or its observability but do not make repeated runs converge.

Answer

Log the loss curve to TensorBoard so the team can visually compare runs.

Answer

Increase the number of GPUs per run so each experiment finishes faster.

Answer

Reduce the learning rate by a factor of ten across all sweep configurations.

Question 6

An ML engineer is comparing two fine-tuned variants of the same base LLM on an internal question-answering benchmark. Variant 1 scores higher on exact-match accuracy, but Variant 2 produces answers that human reviewers judge more helpful and better grounded. The engineer must decide which variant to promote. Which evaluation approach is most appropriate?

Accepted Answer

Combine the automated metric with a structured human or LLM-judge rubric that scores helpfulness and grounding, then decide using both.. When automated and human signals disagree, the right move is usually to measure both dimensions rather than discard one. A rubric-based human or LLM-judge evaluation for helpfulness and grounding complements exact-match accuracy, which is reproducible but blind to paraphrase and partial credit. Deciding with both signals yields a promotion choice that reflects real user value while retaining an objective regression check.

Answer

Promote Variant 1 because exact-match accuracy is an objective, reproducible metric that removes human subjectivity.

Answer

Promote Variant 2 without further measurement because human reviewers are the ultimate authority on answer quality.

Answer

Retrain both variants on the benchmark questions until exact-match accuracy converges, then promote the higher scorer.

Question 7

A data scientist is experimenting with an LLM for a text generation task using NVIDIA NeMo. They want to measure how the model's output diversity changes when adjusting the temperature parameter. They plan to generate 100 samples for each temperature setting and compute the distinct-n metric. Which experimental design principle are they applying?

Accepted Answer

Controlled experiment. The scientist is manipulating a single independent variable (temperature) while holding other factors constant, and measuring its effect on a dependent variable (output diversity via distinct-n). This systematic approach is a controlled experiment, which allows for causal inference about the relationship between temperature and diversity. It is a fundamental design principle in LLM experimentation.

Answer

Ablation study

Answer

Cross-validation

Answer

Hyperparameter optimization

Question 8

A researcher is running a hyperparameter sweep over learning rate and batch size for an LLM fine-tune. To keep the experiment tractable, they want to prune unpromising trials early. Which approach best supports early stopping of poorly performing trials while preserving statistical validity?

Accepted Answer

Use a successive halving or Hyperband-style scheduler that allocates small budgets to many trials and promotes only the best performers to larger budgets.. Successive halving and Hyperband explicitly trade exploration for exploitation by giving many configurations a small budget and progressively promoting the best ones. This preserves the chance to discover strong hyperparameters while avoiding full-length runs for clearly weak trials. Compared with fixed patience or first-epoch thresholds, the promotion structure is more robust to differences in loss-curve shape across hyperparameters.

Answer

Stop any trial whose training loss has not decreased in the last 100 steps and discard its results.

Answer

Run all trials to full length but evaluate them on a smaller validation set to save time.

Answer

Eliminate trials whose first-epoch loss is above the median of all trials, without further evaluation.

Question 9

A team has 1,024-dimensional document embeddings from a retrieval corpus and needs an interactive visualization to explore semantic neighborhoods for debugging retrieval failures. They want to preserve both global structure and local neighborhoods as faithfully as possible while keeping the tool responsive during pan and zoom. Which approach best fits?

Accepted Answer

Reduce to two dimensions with UMAP and render points in an interactive plotting library with hover labels.. Exploring semantic neighborhoods in high-dimensional embeddings calls for a nonlinear reduction that preserves local structure while retaining some global layout, plus an interactive renderer for inspection. UMAP with hover labels meets both needs. Linear projection loses neighborhood detail, raw dimensions are semantically meaningless, and pairwise distance heatmaps do not scale or support spatial exploration.

Answer

Apply PCA to two components and display a static scatter plot image.

Answer

Compute a full pairwise distance matrix and render it as a static heatmap.

Answer

Plot the first two raw embedding dimensions without any dimensionality reduction.

Question 10

During an ablation study on a retrieval-augmented LLM in NeMo, an engineer removes the reranking stage and observes that answer accuracy drops by 12 points, but latency improves by 40 percent. A stakeholder asks whether reranking should be kept. Which experimental next step best supports a defensible recommendation?

Accepted Answer

Measure the accuracy-latency trade-off across several reranker sizes and retrieval depths, then compare against the application's latency budget and quality target.. A single ablation establishes that reranking trades latency for accuracy but does not show whether a middle ground exists. Sweeping reranker sizes and retrieval depths produces a trade-off curve, and evaluating those points against the application's latency budget and quality target converts measurements into a recommendation. Asserting a priority, removing retrieval, or compensating with more documents does not answer the stakeholder's question.

Answer

Keep reranking because accuracy is always more important than latency in production systems.

Answer

Increase the number of retrieved documents while keeping the reranker disabled to recover the lost accuracy.

Answer

Remove the retrieval stage entirely to see whether the model's parametric knowledge is sufficient.

Question 11

You are preparing a quarterly report on an LLM's inference latency for stakeholders. The raw data contains 50,000 individual request latencies in milliseconds. You need a single visualization that shows the full distribution shape, including any long tail of slow requests, without losing information to binning. Which visualization should you use?

Accepted Answer

An empirical cumulative distribution function (ECDF) plot of the latency values. An ECDF plot is ideal when the goal is to preserve all information about a distribution, including its tail, without binning or smoothing. It plots the proportion of requests at or below each latency value, so stakeholders can directly read percentiles and see the exact shape of the slow-request tail in an LLM serving workload.

Answer

A histogram with 20 bins of the latency values

Answer

A box plot of the latency values

Answer

A violin plot of the latency values

Question 12

A team is fine-tuning a NeMo Megatron GPT model on an internal corpus and observes that validation loss begins rising after epoch three while training loss continues to fall. They want to detect this condition automatically during future experiments without manually watching the curves. Which NeMo callback or mechanism should they configure to stop training when validation loss stops improving?

Accepted Answer

NeMo EarlyStopping callback monitoring validation loss. The EarlyStopping callback in NeMo Framework is designed to monitor a validation metric and stop training after a configurable patience window without improvement. That matches the scenario of validation loss rising while training loss falls, letting the team end runs automatically. Profilers, quantization passes, and checkpoint retention settings do not influence when training stops, so they cannot address the overfitting signal.

Answer

TensorRT-LLM quantization calibration pass

Answer

NeMo ModelCheckpoint with save_top_k set to a negative value

Answer

NVIDIA Nsight Compute kernel replay

Question 13

An engineer is evaluating a RAG-based assistant and wants to isolate whether retrieval quality or the generator is responsible for wrong answers. They build a small labeled set of questions with known correct passages and known correct answers. Which experimental design most cleanly separates the contribution of the retriever from that of the generator?

Accepted Answer

Measure retriever recall against the known correct passages, then feed the known correct passages to the generator and measure answer accuracy separately.. Isolating retriever versus generator contributions requires scoring each stage against ground truth. Retriever recall measured against known correct passages reveals retrieval quality, and feeding those oracle passages to the generator reveals generation quality independent of retrieval. End-to-end comparisons and top-k tuning mix the two stages and cannot attribute failures.

Answer

Compare the end-to-end accuracy of the RAG system against the same generator without retrieval.

Answer

Increase the retriever top-k until end-to-end accuracy stops improving, then freeze the retriever and tune the generator.

Answer

Run the full pipeline and measure end-to-end answer accuracy, then retrain the generator on the failures.

Question 14

A data scientist is analyzing a large corpus of LLM training documents and wants to visualize which topics appear together across documents. After computing TF-IDF vectors, they apply non-negative matrix factorization (NMF) to reduce dimensionality. Which visualization best shows the relationships between the discovered topics and the documents?

Accepted Answer

A heatmap of the document-topic matrix with documents on one axis and topics on the other. The document-topic matrix produced by NMF is a two-dimensional array of weights, and a heatmap is the natural visualization for such data. It lets the analyst see which topics are active in each document and which topics tend to co-occur, directly answering the question about topic relationships across the corpus.

Answer

A pie chart of the top 10 most frequent words in the corpus

Answer

A line chart of the reconstruction error across NMF iterations

Answer

A scatter plot of documents positioned by their first two TF-IDF principal components

Question 15

An engineer is setting up an automated experiment sweep over temperature and top-p for a NeMo-served LLM, and wants the results to be comparable and reproducible. Which two practices should be applied? (Choose two.)

Accepted Answer

Use the same prompt set and maximum token count across all temperature and top-p combinations.. Comparable decoding sweeps require controlling both stochasticity and inputs. Fixing the sampler seed makes each configuration repeatable, while holding prompts and maximum token count constant ensures the only varying factors are temperature and top-p. Changing the model, the prompt template, or removing parameter logs introduces confounds or destroys traceability, so those practices undermine the experiment.

Answer

Allow each trial to choose its own prompt template so the model can show its best behavior.

Answer

Increase the model's parameter count for trials with higher temperature to compensate for randomness.

Answer

Disable logging of per-trial parameters to reduce storage overhead during the sweep.