NCA-GENL Experimentation Practice Test 4 — 15 Questions

Question 1

An enterprise AI researcher is conducting ablation studies on a large language model using NVIDIA NeMo. To ensure the experimentation results are scientifically valid and statistically sound, which THREE practices must be enforced during the study?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Evaluate across multiple random initialization seeds to account for stochastic variance in training.. Rigorous ablation studies require isolating individual components while holding all other experimental variables constant. Enforcing multiple random seeds accounts for initialization variance, controlling confounding hyperparameters prevents attribution errors, and applying rigorous statistical testing confirms whether observed performance deltas are statistically significant.

Answer

Vary multiple architectural hyperparameters simultaneously in every single experimental iteration.

Answer

Discard all experimental runs that fail to meet performance expectations without logging the failure.

Question 2

An AI engineer at a financial services company is running an LLM experimentation pipeline using NVIDIA NeMo. The primary objective is to evaluate how different tokenizers affect the accuracy of a named entity recognition (NER) task on financial documents. The engineer has already fixed the model architecture, the training dataset, and the hyperparameters. To ensure the experiment isolates the effect of the tokenizer, which action should the engineer take next?

Accepted Answer

Replace the tokenizer with a different one while keeping all other configurations unchanged, then compare NER accuracy.. To isolate the effect of the tokenizer, the engineer must change only the tokenizer while keeping all other factors constant. This controlled approach ensures that any observed difference in NER accuracy is due to tokenization rather than confounding variables like model architecture or dataset size. Such isolation is fundamental to valid experimentation and enables clear conclusions about tokenizer impact.

Answer

Increase the training dataset size to improve NER accuracy before changing the tokenizer.

Answer

Fine-tune the model on a general-domain corpus before evaluating on financial documents.

Answer

Retrain the model with the same tokenizer but different random seeds to measure variance.

Question 3

A team is evaluating an LLM for a customer-support summarization task. They want to compare three prompt templates. Which experimental design most directly isolates the effect of the prompt template?

Accepted Answer

Use the same model, decoding parameters, and evaluation dataset for all three templates, changing only the template text.. A controlled comparison requires that only the variable of interest changes between conditions. By fixing the model, decoding parameters, and evaluation data and varying only the prompt template text, the team ensures that observed differences in summary quality are attributable to the template. This makes the experiment reproducible and the conclusions defensible.

Answer

Use a different model for each template to see which combination performs best overall.

Answer

Vary temperature and top-p across templates so each template is tested under its own best decoding settings.

Answer

Evaluate each template on a different dataset to cover more customer scenarios.

Question 4

A research team is using NVIDIA NeMo to experiment with a large language model for a summarization task. They observe that the model's ROUGE scores vary significantly across different runs even when using the same hyperparameters and dataset. They suspect that non-deterministic operations in the training pipeline are causing this variance. Which step should they take to improve reproducibility of their experiment results?

Accepted Answer

Enable deterministic algorithms and set a fixed random seed in the NeMo training configuration.. To achieve reproducibility, the team must eliminate non-deterministic sources. Setting a fixed random seed and enabling deterministic algorithms ensures that random number generation, data shuffling, and GPU operations are consistent across runs. This directly reduces variance in ROUGE scores and allows other researchers to replicate the experiment exactly, which is a cornerstone of rigorous LLM experimentation.

Answer

Run the experiment multiple times and average the ROUGE scores to report a mean value.

Answer

Use a different optimizer with adaptive learning rates to stabilize training.

Answer

Increase the batch size to reduce the number of gradient updates per epoch.

Question 5

During a fine-tuning experiment in NVIDIA NeMo, validation loss begins to rise after epoch 4 while training loss continues to fall. The team wants to determine the earliest epoch at which the model still generalizes well. Which experimental action is most appropriate?

Accepted Answer

Enable checkpoint saving at every epoch and select the checkpoint with the lowest validation loss for downstream evaluation.. The divergence between falling training loss and rising validation loss indicates the model is memorizing training data. The practical remedy in an experimentation context is to checkpoint each epoch and choose the model with the lowest validation loss, which corresponds to the point of best generalization. This yields the earliest useful epoch without altering the training dynamics.

Answer

Train for more epochs to let validation loss eventually decrease again.

Answer

Increase the learning rate so the model escapes the overfitting region faster.

Answer

Reduce the size of the validation set so the measured validation loss becomes less noisy.

Question 6

A data science team is fine-tuning a Llama 3 8B model on a proprietary customer-support corpus using NVIDIA NeMo. They need to run dozens of experiments with different learning rates and batch sizes. Because the dataset contains personally identifiable information, they cannot send any telemetry to an external tracking server, but they still need to compare runs later and reproduce the best configuration. Which approach best satisfies both the reproducibility and data-privacy requirements?

Accepted Answer

Use the built-in NeMo experiment manager with a local file store, logging configs, metrics, and checkpoints to a shared on-premises directory.. The team needs reproducibility without external telemetry. NeMo's experiment manager with a local file store writes configurations, metrics, and checkpoints to a path the team controls, so nothing leaves the secure environment. External SaaS trackers violate the privacy constraint, checkpoints alone lack the metrics needed for comparison, and manual logging is unreliable and incomplete.

Answer

Enable Weights & Biases integration inside the NeMo experiment manager and configure the project to log only aggregate metrics.

Answer

Disable all logging and rely on the saved .nemo checkpoint files, since the checkpoint embeds the full training configuration.

Answer

Run each experiment in a separate container and manually copy the console output into a spreadsheet after each run.

Question 7

An ML engineer at a healthcare analytics company is starting a fine-tuning experiment on a Llama 2 7B model using NVIDIA NeMo Framework. Before launching the training job, the engineer wants a single immutable record that captures the exact model checkpoint, dataset version, hyperparameters, and evaluation scores so that any later run can be traced back to it. Which component of the NVIDIA NeMo experimentation workflow should the engineer use to store that record?

Accepted Answer

NeMo Experiment Manager. The requirement is a traceable record combining model checkpoint, dataset version, hyperparameters, and evaluation metrics. NeMo Experiment Manager is built to log and organize exactly that metadata during fine-tuning, enabling reproducibility and comparison across runs. Inference servers, inference compilers, and profilers serve different purposes and do not persist the training experiment lineage needed here.

Answer

NVIDIA Triton Inference Server model repository

Answer

NVIDIA TensorRT-LLM build configuration

Answer

NVIDIA Nsight Systems profiling report

Question 8

A data scientist is running a fine-tuning experiment with NVIDIA NeMo on a single A100 GPU. They want to establish a repeatable baseline before sweeping any hyperparameters, so that a later run can be compared fairly. Which practice best supports this goal?

Accepted Answer

Fix a random seed in the NeMo training configuration and record the exact dataset version, model checkpoint, and NeMo container tag used.. A trustworthy baseline requires controlling every input that affects the result, then recording those inputs so the run can be repeated. Fixing the random seed, freezing the dataset version, naming the starting checkpoint, and noting the NeMo container tag together make the run reproducible and comparable. Performance tweaks and best-of-N reporting change or obscure the result rather than establishing a stable reference point.

Answer

Increase the batch size to the maximum the GPU memory allows so the baseline finishes quickly.

Answer

Enable automatic mixed precision and Tensor Core acceleration to reduce training time.

Answer

Run the experiment three times and report the best validation loss observed.

Question 9

You are running a NeMo fine-tuning experiment where validation loss decreases for the first three epochs, then rises steadily while training loss keeps falling. You want to confirm overfitting and select the most appropriate intervention. Which experiment action should you take first?

Accepted Answer

Add regularization such as dropout or weight decay, then re-run the experiment and compare validation curves.. Rising validation loss alongside falling training loss indicates the model is fitting training-specific patterns that do not generalize. Adding regularization such as dropout or weight decay directly limits effective capacity and is a controlled, reversible change. Re-running with the same split and seed isolates the intervention, so the resulting validation curve tells you whether overfitting was indeed the dominant cause.

Answer

Switch the optimizer from Adam to SGD without changing any other hyperparameter.

Answer

Reduce the size of the validation set to lower evaluation noise.

Answer

Increase the number of training epochs and re-run to see if validation loss recovers.

Question 10

You are designing an experiment to measure how quantization (FP16 versus INT8) affects inference latency and answer quality for an LLM deployed with NVIDIA TensorRT-LLM. Which two practices are required for the comparison to be valid? (Choose two.)

Accepted Answer

Benchmark both configurations on the same hardware and under the same batch size and concurrency conditions.. A valid quantization comparison must control everything except precision. Using identical prompts and generation parameters ensures the workload is the same, and benchmarking on identical hardware under identical batch and concurrency conditions ensures the environment is the same. Together these practices let the team attribute measured differences in latency and quality to FP16 versus INT8 rather than to confounds.

Answer

Retrain the model after quantization so the weights adapt to the lower precision.

Answer

Measure latency only on the first generated token, since subsequent tokens are less affected by precision.

Answer

Apply the same random seed to both configurations so token sampling is identical.

Question 11

An ML engineer is comparing two fine-tuned variants of the same base LLM on an internal question-answering benchmark. Variant A scores higher on exact-match accuracy, while Variant B produces answers that human reviewers rate as more helpful and better formatted. The team must choose one variant to ship to production. Which evaluation approach is most appropriate for making this decision?

Accepted Answer

Combine automated metrics with a structured human-preference evaluation, and choose based on the product's primary success criterion.. For generative LLM tasks, automated metrics and human preference capture different quality dimensions. The right decision combines both and weights them by the product's primary success criterion. Shipping on accuracy alone ignores user-perceived helpfulness, shipping on preference alone ignores objective regressions, and averaging without justified weights hides the real trade-off.

Answer

Average the exact-match score and a normalized human-preference score into a single number and pick the higher value.

Answer

Ship Variant A because exact-match accuracy is an objective metric and human ratings are subjective.

Answer

Ship Variant B because human preference is the only metric that matters for generative models.

Question 12

A research team is running a series of controlled LLM fine-tuning experiments on NVIDIA DGX systems using NeMo Framework to compare two learning-rate schedules. They want the comparison to be scientifically valid and repeatable by another engineer next quarter. Which two practices are required to make the experiments reproducible? (Choose two.)

Accepted Answer

Record the exact dataset version, tokenizer configuration, and NeMo container image tag used.. Reproducibility in a controlled fine-tuning comparison requires controlling randomness and pinning the environment. Seeding all stochastic operations keeps data order, dropout, and initialization identical, while recording dataset version, tokenizer configuration, and container image tag lets another engineer rebuild the same setup. Performance-oriented choices such as GPU count, mixed precision, and inference benchmarking do not establish repeatability of the training comparison.

Answer

Deploy the resulting checkpoints to NVIDIA Triton Inference Server for latency benchmarking.

Answer

Increase the number of GPUs in every run so that throughput is maximized.

Answer

Enable mixed precision training to reduce memory usage on each DGX node.

Question 13

An ML engineer is comparing two fine-tuned variants of the same base LLM on an internal question-answering benchmark. Variant 1 scores higher on exact-match accuracy, but Variant 2 produces answers that human reviewers judge more helpful and better grounded. The engineer must decide which variant to promote. Which evaluation approach is most appropriate?

Accepted Answer

Combine the automated metric with a structured human or LLM-judge rubric that scores helpfulness and grounding, then decide using both.. When automated and human signals disagree, the right move is usually to measure both dimensions rather than discard one. A rubric-based human or LLM-judge evaluation for helpfulness and grounding complements exact-match accuracy, which is reproducible but blind to paraphrase and partial credit. Deciding with both signals yields a promotion choice that reflects real user value while retaining an objective regression check.

Answer

Promote Variant 1 because exact-match accuracy is an objective, reproducible metric that removes human subjectivity.

Answer

Promote Variant 2 without further measurement because human reviewers are the ultimate authority on answer quality.

Answer

Retrain both variants on the benchmark questions until exact-match accuracy converges, then promote the higher scorer.

Question 14

A data science team is running a controlled experiment with NVIDIA NeMo to compare two fine-tuning recipes for a 7B-parameter LLM: one with a constant learning rate and one with a cosine decay schedule. They notice the evaluation loss curves diverge significantly after step 500, but they cannot tell whether the difference is caused by the learning-rate schedule or by random seed variance. Which experimental change should they make to isolate the effect of the schedule?

Accepted Answer

Run both recipes with a fixed random seed and identical data ordering, then compare the loss curves.. A controlled experiment requires isolating the independent variable—here the learning-rate schedule—while holding all other factors constant. Random seed and data ordering are common sources of run-to-run variance in LLM fine-tuning. Fixing them across both recipes ensures that observed differences in evaluation loss are caused by the schedule rather than initialization or batch order, making the comparison valid.

Answer

Switch both recipes to a warmup-stable-decay schedule and compare the final evaluation loss instead of the curves.

Answer

Increase the batch size for both recipes until the loss curves become smoother and easier to compare visually.

Answer

Run each recipe on a different GPU type to see whether hardware differences explain the divergence.

Question 15

A team is designing a controlled experiment to measure whether increasing LoRA rank improves instruction-following accuracy on a held-out benchmark. They want the comparison to be scientifically valid. Which experimental design choice best supports a valid conclusion?

Accepted Answer

Hold the base model, dataset, prompt template, and evaluation harness fixed while varying only LoRA rank.. A scientifically valid comparison changes one independent variable while holding everything else constant. Here, LoRA rank is the factor under test, so the base model, training data, prompt template, and evaluation harness must remain identical across runs. Using a shared held-out benchmark and reporting aggregate accuracy across seeds ensures observed differences reflect rank rather than confounding factors or evaluation noise.

Answer

Change LoRA rank and the base model simultaneously so the experiment covers more configurations in one run.

Answer

Report only the best accuracy observed across several random seeds for each rank.

Answer

Evaluate each rank on a freshly sampled held-out set to reduce any dataset bias.