NCA-GENL Experimentation Practice Test 5 — 15 Questions

Question 1

A research team is running a hyperparameter sweep over learning rate and warmup steps for a NeMo fine-tuning job. They notice that runs with identical hyperparameters produce different final validation losses across repeated executions. Which two changes would most directly improve the reproducibility of these experiments? (Choose two.)

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Enable deterministic kernels and disable nondeterministic cuDNN algorithms in the framework settings.. Run-to-run variance in identical configurations usually comes from uncontrolled randomness in data order, initialization, dropout, and nondeterministic GPU kernels. Fixing the random seed and enabling deterministic kernels address both the algorithmic and hardware-level sources. Adding GPUs, logging, or lowering the learning rate change the experiment or its observability but do not make repeated runs converge.

Answer

Log the loss curve to TensorBoard so the team can visually compare runs.

Answer

Increase the number of GPUs per run so each experiment finishes faster.

Answer

Reduce the learning rate by a factor of ten across all sweep configurations.

Question 2

A researcher is running a hyperparameter sweep over learning rate and batch size for an LLM fine-tune. To keep the experiment tractable, they want to prune unpromising trials early. Which approach best supports early stopping of poorly performing trials while preserving statistical validity?

Accepted Answer

Use a successive halving or Hyperband-style scheduler that allocates small budgets to many trials and promotes only the best performers to larger budgets.. Successive halving and Hyperband explicitly trade exploration for exploitation by giving many configurations a small budget and progressively promoting the best ones. This preserves the chance to discover strong hyperparameters while avoiding full-length runs for clearly weak trials. Compared with fixed patience or first-epoch thresholds, the promotion structure is more robust to differences in loss-curve shape across hyperparameters.

Answer

Stop any trial whose training loss has not decreased in the last 100 steps and discard its results.

Answer

Run all trials to full length but evaluate them on a smaller validation set to save time.

Answer

Eliminate trials whose first-epoch loss is above the median of all trials, without further evaluation.

Question 3

A team fine-tuning a NeMo large language model runs the same training configuration three times and obtains validation loss values of 2.14, 2.31, and 2.09 at the end of the same number of steps. They need subsequent runs to produce tightly clustered, comparable numbers so hyperparameter comparisons are meaningful. Which change most directly addresses this problem?

Accepted Answer

Set a fixed random seed and enable deterministic behavior for the training run.. Run-to-run variance in loss comes from uncontrolled randomness: weight initialization, dropout sampling, data shuffling, and nondeterministic GPU kernels. Pinning a seed and enabling deterministic execution makes those sources identical across repeated runs, so differences in outcomes can be attributed to the hyperparameters under test rather than to chance.

Answer

Add early stopping based on validation loss plateau detection.

Answer

Increase the global batch size while holding the learning rate constant.

Answer

Switch the optimizer from AdamW to SGD with momentum.

Question 4

A data scientist is preparing an LLM fine-tuning experiment on NVIDIA NeMo and wants every run to be reproducible weeks later. The team's experiment tracker currently logs only the final validation loss. Which additional item is most important to record so a run can be reproduced exactly?

Accepted Answer

The random seed, framework version, and full training configuration. Exact reproduction of an LLM experiment depends on controlling every stochastic and environmental input. The seed governs initialization and data ordering, the framework and CUDA versions govern kernel behavior and numerics, and the full configuration governs optimization. Logging only a final metric captures an outcome but discards the recipe, so the run cannot be recreated reliably.

Answer

The wall-clock duration of the final epoch only

Answer

A screenshot of the loss curve rendered in the dashboard

Answer

The names of the engineers who launched each training job

Question 5

A team is fine-tuning a NeMo Megatron GPT model on an internal corpus and observes that validation loss begins rising after epoch three while training loss continues to fall. They want to detect this condition automatically during future experiments without manually watching the curves. Which NeMo callback or mechanism should they configure to stop training when validation loss stops improving?

Accepted Answer

NeMo EarlyStopping callback monitoring validation loss. The EarlyStopping callback in NeMo Framework is designed to monitor a validation metric and stop training after a configurable patience window without improvement. That matches the scenario of validation loss rising while training loss falls, letting the team end runs automatically. Profilers, quantization passes, and checkpoint retention settings do not influence when training stops, so they cannot address the overfitting signal.

Answer

TensorRT-LLM quantization calibration pass

Answer

NeMo ModelCheckpoint with save_top_k set to a negative value

Answer

NVIDIA Nsight Compute kernel replay

Question 6

A research team is running a hyperparameter sweep over learning rate and warmup steps for a NeMo fine-tuning job. They notice that runs with identical hyperparameters produce different final validation losses across repeated executions. Which two changes would most directly improve the reproducibility of these experiments? (Choose two.)

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Run each configuration with multiple fixed random seeds and compare mean evaluation scores with their confidence intervals.. When run-to-run variance exceeds the differences between configurations, single-run comparisons cannot support a conclusion. Repeating each configuration with several fixed seeds and reporting mean scores with confidence intervals turns noise into a measurable uncertainty, so the team can tell whether a hyperparameter effect is real. This is the standard remedy for high-variance sweeps and prevents selecting configurations on the basis of lucky seeds.

Answer

Report only the single best score achieved by each configuration, since the best result shows the model's potential.

Answer

Increase the learning rate for all runs so the model converges faster and the noise is averaged out.

Answer

Reduce the number of sweep configurations so fewer comparisons are affected by variance.

Question 7

A data scientist is experimenting with an LLM for a text generation task using NVIDIA NeMo. They want to measure how the model's output diversity changes when adjusting the temperature parameter. They plan to generate 100 samples for each temperature setting and compute the distinct-n metric. Which experimental design principle are they applying?

Accepted Answer

Controlled experiment. The scientist is manipulating a single independent variable (temperature) while holding other factors constant, and measuring its effect on a dependent variable (output diversity via distinct-n). This systematic approach is a controlled experiment, which allows for causal inference about the relationship between temperature and diversity. It is a fundamental design principle in LLM experimentation.

Answer

Ablation study

Answer

Cross-validation

Answer

Hyperparameter optimization

Question 8

An ML engineer is setting up an experiment log for a fine-tuning run and wants to record the metadata necessary to reproduce the resulting model later. Which two items are most essential to capture for reproducibility? (Choose two.)

Accepted Answer

The random seed used for data shuffling and initialization.. Reproducibility requires capturing the inputs that determine the trained weights: the randomness source and the exact training data. The seed governs shuffling and initialization, and the dataset version pins the examples used. Together they let another engineer rerun the same configuration and obtain a comparable model, which is the practical definition of a reproducible experiment.

Answer

The names of the engineers who reviewed the training logs.

Answer

The wall-clock duration of each training epoch.

Answer

The GPU model and driver version used during training.

Question 9

An ML engineer is using NVIDIA NeMo to evaluate a retrieval-augmented generation pipeline. They want to measure whether adding a reranker improves answer faithfulness, but they must ensure the experiment is reproducible and comparable across runs. Which practice best supports a valid comparison between the pipeline with and without the reranker?

Accepted Answer

Use the same evaluation dataset, same base LLM checkpoint, and same decoding parameters for both pipeline variants, changing only the reranker component.. To attribute a change in faithfulness to the reranker, every other element of the RAG pipeline must be held constant. Using the same evaluation set, base checkpoint, and decoding parameters for both variants ensures the only difference is the presence or absence of the reranker. This controlled design yields a clean, reproducible comparison and avoids confounding variables that would invalidate the result.

Answer

Increase the temperature of the LLM in the reranker variant to encourage more diverse answers and better faithfulness.

Answer

Evaluate the reranker variant on a larger and more diverse dataset than the baseline so the results are more statistically significant.

Answer

Run the baseline and reranker variants on different GPU models to test robustness across hardware configurations.

Question 10

An engineer must decide how to split a labeled dataset of 50,000 customer support conversations before fine-tuning a NeMo LLM for intent classification. The goal is an honest estimate of how the tuned model will behave on never-before-seen tickets once deployed. Which splitting approach best supports that goal?

Accepted Answer

Randomly assign 70 percent to training, 15 percent to validation, and 15 percent to a test set that is touched only once at the end.. An untouched holdout test set is the only split that yields an unbiased generalization estimate, because it never influences training or hyperparameter choices. Pairing it with a validation split lets the team tune decisions without contaminating the final measurement, and random assignment keeps class balance comparable across splits.

Answer

Use all 50,000 conversations for training and rely on the training loss curve to judge generalization.

Answer

Tune hyperparameters repeatedly against the test split until accuracy is maximized.

Answer

Split the data by conversation length, putting the longest 15 percent in the test set.

Question 11

An engineer is evaluating a RAG-based assistant and wants to isolate whether retrieval quality or the generator is responsible for wrong answers. They build a small labeled set of questions with known correct passages and known correct answers. Which experimental design most cleanly separates the contribution of the retriever from that of the generator?

Accepted Answer

Measure retriever recall against the known correct passages, then feed the known correct passages to the generator and measure answer accuracy separately.. Isolating retriever versus generator contributions requires scoring each stage against ground truth. Retriever recall measured against known correct passages reveals retrieval quality, and feeding those oracle passages to the generator reveals generation quality independent of retrieval. End-to-end comparisons and top-k tuning mix the two stages and cannot attribute failures.

Answer

Compare the end-to-end accuracy of the RAG system against the same generator without retrieval.

Answer

Increase the retriever top-k until end-to-end accuracy stops improving, then freeze the retriever and tune the generator.

Answer

Run the full pipeline and measure end-to-end answer accuracy, then retrain the generator on the failures.

Question 12

A team is comparing two fine-tuned variants of the same base LLM for a customer-support summarization task. Variant A was trained on 10,000 examples and variant B on 2,000 examples drawn from the same distribution. To make a fair comparison of their generalization, which evaluation practice should the team adopt?

Accepted Answer

Evaluate both variants on the same held-out test set that neither saw during training. Fair comparison requires controlling the evaluation data. When both variants are scored on the same held-out set that neither encountered during training, differences in their scores reflect the training choices rather than variation in test difficulty. Training loss and parameter counts are process metrics that do not measure generalization to unseen customer-support summaries.

Answer

Report the training loss of each variant at the final step

Answer

Compare the number of parameters each variant updated during fine-tuning

Answer

Let each variant be scored on its own randomly split test set

Question 13

An engineer is designing an experiment to measure how prompt phrasing affects the output quality of a deployed LLM. They will test several prompt templates against a fixed evaluation set and want the comparison to be valid. Which two practices are required? (Choose two.)

Accepted Answer

Hold the model, decoding parameters, and evaluation dataset constant across all prompt templates.. A valid prompt experiment isolates the prompt template as the only variable. Keeping the model, decoding parameters, and evaluation dataset fixed means any quality difference can be attributed to phrasing, and scoring all outputs with the same rubric ensures the measurement itself does not shift. Changing datasets, retraining, or varying temperature introduces confounds that make the comparison meaningless.

Answer

Retrain the model after each prompt template so it adapts to the new phrasing.

Answer

Use a different evaluation dataset for each prompt template so the results are not correlated.

Answer

Increase the temperature for each successive template to explore more diverse outputs.

Question 14

A researcher is running an ablation study in which they vary the number of attention heads in a NeMo Megatron GPT model while holding parameter count, dataset, and learning rate fixed. After the first run, they change tensor parallel size and pipeline parallel size to fit larger variants on the available GPUs. A colleague argues this invalidates the comparison. Which statement best explains the scientific concern?

Accepted Answer

Different parallelism configurations change numerical reduction order and kernel selection, introducing variance unrelated to the attention-head variable.. An ablation study aims to attribute observed differences to one variable. When tensor and pipeline parallel sizes change between runs, the reduction order, kernel selection, and communication patterns change too, introducing numerical and convergence variance that is unrelated to attention head count. That makes the comparison confounded, even though optimizer sharding and micro-batching can be handled without altering the effective update or global batch size.

Answer

Attention head count is not a valid ablation variable because it is determined by hidden size and cannot be varied independently.

Answer

Changing parallelism alters the optimizer state sharding, so the effective learning rate changes per parameter.

Answer

Pipeline parallelism forces different micro-batch sizes, which always changes the global batch size and therefore the loss landscape.

Question 15

An AI engineer is conducting an experiment to compare two different fine-tuning approaches for a large language model using NVIDIA NeMo: full fine-tuning versus parameter-efficient fine-tuning (PEFT) with LoRA. The engineer wants to determine which approach yields better performance on a downstream question-answering task while minimizing computational cost. Which metric should the engineer prioritize to evaluate the trade-off between performance and cost?

Accepted Answer

Validation accuracy per GPU-hour. Validation accuracy per GPU-hour effectively combines the two objectives: it measures the model's performance on the question-answering task (validation accuracy) and normalizes it by the computational cost (GPU-hours). This allows the engineer to compare full fine-tuning and LoRA on a level playing field, identifying which method provides the best accuracy for the resources invested. It directly supports the goal of minimizing cost while maximizing performance.

Answer

Training loss convergence rate

Answer

Number of trainable parameters

Answer

Inference latency on a CPU