NCA-GENL Experimentation Practice Test 6 — 15 Questions

Question 1

A team is running an LLM fine-tuning experiment using NVIDIA NeMo and wants to track how the validation loss changes over training. They need a reliable way to detect overfitting early. Which metric should they monitor most directly during the experiment?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Validation loss computed on a held-out dataset at regular intervals during training.. Overfitting is characterized by a growing gap between training and validation performance. Validation loss on a held-out set is the most direct indicator: when it stops improving and begins to rise while training loss keeps falling, the model is memorizing rather than generalizing. Monitoring it during training lets the team intervene early with regularization or early stopping.

Answer

Training loss computed on the same data batches used for gradient updates.

Answer

GPU utilization percentage reported by the NeMo training logs.

Answer

The number of tokens processed per second during training.

Question 2

A research team is comparing three fine-tuning recipes for a NeMo LLM and wants the comparison to be defensible in a later review. Which two practices most improve the credibility of the reported comparison? (Choose two.)

Accepted Answer

Evaluate every recipe on the same held-out evaluation set with identical decoding settings.. Credible comparisons rest on two pillars: full provenance of each run so results are reproducible, and a shared, uncontaminated evaluation protocol so scores are measured identically. Provenance lets reviewers attribute differences to the recipes, while a fixed held-out set with consistent decoding settings ensures the numbers being compared actually measure the same thing.

Answer

Tune each recipe's learning rate separately until its score is maximized.

Answer

Reuse the evaluation set during training as an additional validation signal.

Answer

Report only the single best run for each recipe to keep the summary concise.

Question 3

A team is running an A/B experiment comparing two prompt templates for a customer-facing LLM assistant. After one week, template A shows a 2% higher task-completion rate with a p-value of 0.04. The team lead wants to declare A the winner immediately. Which consideration is most important before making that decision?

Accepted Answer

Whether the experiment was stopped early or the sample size was fixed in advance, because repeatedly checking and stopping inflates false-positive rates.. A p-value of 0.04 is fragile if the team monitored results continuously and stopped at the first significant reading, because optional stopping inflates the false-positive rate. The critical check is whether the sample size and stopping rule were fixed in advance or whether a sequential correction applies. Test direction, author identity, and sampling temperature are secondary concerns.

Answer

Whether the two prompt templates were written by different engineers, since authorship affects output quality.

Answer

Whether the assistant uses a temperature greater than zero, since sampling randomness invalidates A/B tests.

Answer

Whether the p-value was computed with a one-tailed or two-tailed test, since one-tailed tests are always invalid.

Question 4

A data scientist wants to determine how sensitive an LLM's summarization quality is to the temperature sampling parameter. They plan a sweep across several temperature values. Which experimental approach gives the clearest signal about temperature's effect?

Accepted Answer

Vary temperature while keeping the model, prompts, and evaluation metric constant.. To measure sensitivity to a single parameter, that parameter must be the only thing changing. Fixing the model, prompts, and evaluation metric ensures any quality difference across the sweep comes from temperature. This controlled setup produces a curve that shows whether summarization quality is robust or fragile with respect to the sampling temperature setting.

Answer

Use a different evaluation metric for each temperature value to capture more aspects of quality.

Answer

Change temperature and the prompt template together to explore the joint space faster.

Answer

Test only the lowest and highest temperature values to save compute.

Question 5

An ML engineer is running a hyperparameter sweep with NVIDIA NeMo and notices that runs with identical configurations sometimes produce slightly different final loss values. Which cause should the engineer investigate first?

Accepted Answer

Nondeterministic GPU operations and unseeded randomness in the data pipeline. Run-to-run variation with identical configuration points to uncontrolled randomness and nondeterministic computation. Unseeded data shuffling changes which samples appear in which batch, and nondeterministic GPU kernels can accumulate floating-point differences. Together these produce slightly different loss trajectories. Validation set size and checkpoint intervals affect measurement and I/O, not the training computation itself.

Answer

The checkpoint saving interval is too frequent

Answer

The number of GPUs is too high for the batch size being used

Answer

The validation set is too small to measure loss accurately

Question 6

An engineer at a customer-support automation company is experimenting with top-p sampling values for a NeMo-served LLM. They want to quantify how output diversity changes across settings without relying on human judgment alone. Which evaluation approach best supports this experiment?

Accepted Answer

Compute distinct-n and self-BLEU across a fixed prompt set for each top-p value.. To quantify output diversity across top-p settings, the engineer needs metrics that capture lexical variety and redundancy in generated text. Distinct-n and self-BLEU are computed automatically over a fixed prompt set and directly reflect diversity changes. Latency, GPU profiling, and training loss describe performance or training behavior, not the diversity of inference-time outputs, so they cannot answer the experiment's question.

Answer

Measure end-to-end inference latency with NVIDIA Triton Inference Server metrics.

Answer

Compare training loss curves from the original fine-tuning job.

Answer

Track GPU utilization and memory bandwidth during generation with Nsight Systems.

Question 7

An AI researcher is designing an experiment to compare two prompt templates for a customer-support LLM using NVIDIA NeMo. To ensure the comparison is fair and reproducible, which two practices should they follow? (Choose two.)

Accepted Answer

Use the same underlying LLM checkpoint and decoding parameters for both prompt templates.. A fair prompt-template comparison requires isolating the template as the only independent variable. Keeping the model checkpoint, decoding parameters, evaluation dataset, and scoring criteria identical across both conditions ensures that observed differences are caused by the prompt itself. This controlled design is the foundation of reproducible LLM experimentation and supports confident deployment decisions.

Answer

Vary the temperature for each template to explore a wider range of outputs.

Answer

Fine-tune the LLM separately for each prompt template before comparing them.

Answer

Use a different evaluation metric for each template to capture their unique strengths.

Question 8

During an ablation study, a team removes the instruction-tuning stage from their NeMo pipeline and observes that the model still answers factual questions but frequently ignores the requested output format. They want to attribute this change in behavior to the removed stage rather than to noise. Which experimental design element is most important for supporting that attribution?

Accepted Answer

Run both the full pipeline and the ablated pipeline with all other variables held constant, then compare on the same evaluation set.. An ablation is only interpretable when the manipulated component is the sole difference between conditions. Keeping data, hyperparameters, training budget, and evaluation protocol identical, and scoring both variants on the same held-out set, lets the team attribute the observed format-compliance change to removing instruction tuning instead of to unrelated variation.

Answer

Increase the ablated model's training steps so it receives more optimization than the full pipeline.

Answer

Retrain the ablated model several times with different random seeds and report the highest format-compliance score.

Answer

Evaluate the ablated model with a different, more format-focused prompt template than the full pipeline.

Question 9

A research team is comparing two LoRA fine-tuning runs of the same Llama-based model in NeMo. Run 1 uses rank 8 and alpha 16; Run 2 uses rank 64 and alpha 16, with all other hyperparameters identical. Run 2 achieves lower training loss but worse accuracy on a held-out evaluation set. Which conclusion is most defensible from this experiment?

Accepted Answer

The rank-64 run overfits the training data, so the lower training loss does not translate to better generalization.. The pattern of lower training loss with worse held-out accuracy indicates that the higher-rank adapter used its additional capacity to fit training-specific noise. LoRA rank sets the dimensionality of the update matrices, so rank 64 has more trainable parameters than rank 8. The defensible conclusion is that the larger adapter overfit, and the more constrained adapter generalized better on this evaluation set.

Answer

The two runs cannot be compared because LoRA rank changes the base model architecture.

Answer

Higher LoRA rank always improves downstream accuracy, so the evaluation set must be mislabeled.

Answer

The alpha value should be doubled for Run 2 to restore the intended scaling ratio.

Question 10

During an ablation study on a retrieval-augmented LLM pipeline, the team removes the reranking stage and observes a large drop in answer accuracy on their benchmark. Before concluding that reranking is essential, which additional experiment is most important to run?

Accepted Answer

Run a control condition that keeps the pipeline identical except for a neutral change unrelated to reranking.. An ablation shows that a change in the pipeline correlates with an accuracy drop, but correlation is not causation unless other differences are ruled out. A control condition with a neutral modification tests whether the pipeline is sensitive to arbitrary changes. If the control shows little effect while removing reranking causes a large drop, the causal claim about reranking is substantially strengthened.

Answer

Increase the number of retrieved documents to compensate for the missing reranker.

Answer

Replace the benchmark with a larger one to increase statistical power.

Answer

Re-run the ablation with a different random seed to see if the accuracy drop persists.

Question 11

A research team is designing an experiment to measure how prompt phrasing affects the factuality of an LLM in a retrieval-augmented question-answering pipeline. Which two design choices are necessary to attribute observed factuality differences to the prompt rather than to other pipeline components? (Choose two.)

Accepted Answer

Hold the retrieved context, model checkpoint, and decoding parameters constant across prompt variants. Attributing factuality differences to prompt phrasing requires isolating the prompt as the only changed variable. Holding retrieval, model, and decoding settings constant removes competing explanations, while a shared evaluation set and rubric make scores comparable. Altering the retrieval index, temperature, or datasets per variant introduces confounds that make any observed improvement impossible to credit to the prompt itself.

Answer

Evaluate each prompt variant on a different dataset tailored to its strengths

Answer

Increase the temperature for variants that produce shorter answers to equalize response length

Answer

Allow the retrieval index to be rebuilt with different embedding models for each prompt variant

Question 12

An ML team is running an ablation study with NVIDIA NeMo to determine which components of their LLM pipeline contribute most to answer quality. They remove one component at a time and re-evaluate. After several runs, they notice that removing the retrieval component causes a large drop in quality, but removing the reranker causes almost no change. What is the most reasonable interpretation of this result?

Accepted Answer

Retrieval is a critical contributor to quality in this pipeline, while the reranker adds little measurable value under the current evaluation setup.. Ablation studies estimate each component's marginal contribution by removing it and measuring the performance change. A large drop when retrieval is removed indicates it is essential; a negligible drop when the reranker is removed indicates it adds little value under the current evaluation. This supports decisions such as simplifying the pipeline or re-evaluating the reranker with a harder test set.

Answer

Retrieval and the reranker are equally important, but the reranker's effect is masked by the retriever.

Answer

The reranker is broken and must be replaced with a different model before any conclusion can be drawn.

Answer

The evaluation metric is too noisy to detect the reranker's effect, so the experiment should be discarded.

Question 13

An engineer is setting up an automated experiment sweep over temperature and top-p for a NeMo-served LLM, and wants the results to be comparable and reproducible. Which two practices should be applied? (Choose two.)

Accepted Answer

Use the same prompt set and maximum token count across all temperature and top-p combinations.. Comparable decoding sweeps require controlling both stochasticity and inputs. Fixing the sampler seed makes each configuration repeatable, while holding prompts and maximum token count constant ensures the only varying factors are temperature and top-p. Changing the model, the prompt template, or removing parameter logs introduces confounds or destroys traceability, so those practices undermine the experiment.

Answer

Allow each trial to choose its own prompt template so the model can show its best behavior.

Answer

Increase the model's parameter count for trials with higher temperature to compensate for randomness.

Answer

Disable logging of per-trial parameters to reduce storage overhead during the sweep.

Question 14

A team's NeMo fine-tuning experiment runs on a fixed compute budget and they must choose how to allocate it between searching hyperparameters and training the final model. Their hyperparameter search space is large and each trial is expensive. Which allocation strategy best balances finding a strong configuration against producing a well-trained final model?

Accepted Answer

Use a budget-aware search such as successive halving or Bayesian optimization, then spend the remaining budget training the best configuration to completion.. With a fixed budget and expensive trials, the efficient path is to let early results prune weak candidates and concentrate resources on promising ones, then commit remaining budget to fully training the winner. This avoids both the waste of exhaustive grid search and the risk of delivering an under-trained final model.

Answer

Spend the entire budget on a dense grid search over every hyperparameter combination.

Answer

Run many short trials with identical settings to reduce measurement noise, then pick any configuration.

Answer

Train one configuration to completion and skip hyperparameter search entirely.

Question 15

A team runs a NeMo fine-tuning experiment and observes that validation loss decreases for several epochs and then steadily rises while training loss keeps falling. They want to confirm whether the checkpoint from the best validation epoch is genuinely better than the final checkpoint. Which action provides the strongest evidence?

Accepted Answer

Evaluate both checkpoints on a separate held-out test set and compare task metrics. When a checkpoint is selected using validation loss, that metric becomes optimistically biased for the chosen model. The strongest evidence comes from an untouched test set that was not involved in any selection decision. Evaluating both checkpoints there with task-relevant metrics yields an unbiased comparison, whereas training loss and repeated validation checks cannot settle the question.

Answer

Compare the training loss values of the two checkpoints

Answer

Continue training the final checkpoint for more epochs and recheck validation loss

Answer

Pick whichever checkpoint has the lower validation loss