NCA-GENL Practice Test 14 — 15 Questions

Question 1

You are analyzing token frequency distribution across a 50 GB pretraining corpus before fine-tuning an NVIDIA NIM-deployed Llama model. The raw frequency histogram is heavily right-skewed, making it impossible to compare low-frequency tokens. Which transformation should you apply to the x-axis to make the distribution easier to compare across the full vocabulary?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Plot token rank on a logarithmic x-axis against frequency on a logarithmic y-axis.. Token frequencies in natural-language corpora follow a Zipfian power law, so both rank and frequency span several orders of magnitude. Plotting rank and frequency on logarithmic axes linearizes that relationship and makes the entire vocabulary comparable in a single chart. Linear or weakly transformed axes cannot compress the dynamic range enough to reveal the tail behavior that matters for vocabulary and sampling decisions.

Answer

Normalize each token count by the total corpus token count and keep a linear x-axis.

Answer

Bin tokens into deciles and plot only the top ten most frequent tokens.

Answer

Apply a square-root transform to token counts and keep the vocabulary index on the x-axis.

Question 2

A team is running an A/B experiment comparing two prompt templates for a customer-facing LLM assistant. After one week, template A shows a 2% higher task-completion rate with a p-value of 0.04. The team lead wants to declare A the winner immediately. Which consideration is most important before making that decision?

Accepted Answer

Whether the experiment was stopped early or the sample size was fixed in advance, because repeatedly checking and stopping inflates false-positive rates.. A p-value of 0.04 is fragile if the team monitored results continuously and stopped at the first significant reading, because optional stopping inflates the false-positive rate. The critical check is whether the sample size and stopping rule were fixed in advance or whether a sequential correction applies. Test direction, author identity, and sampling temperature are secondary concerns.

Answer

Whether the two prompt templates were written by different engineers, since authorship affects output quality.

Answer

Whether the assistant uses a temperature greater than zero, since sampling randomness invalidates A/B tests.

Answer

Whether the p-value was computed with a one-tailed or two-tailed test, since one-tailed tests are always invalid.

Question 3

A researcher is running an ablation study in which they vary the number of attention heads in a NeMo Megatron GPT model while holding parameter count, dataset, and learning rate fixed. After the first run, they change tensor parallel size and pipeline parallel size to fit larger variants on the available GPUs. A colleague argues this invalidates the comparison. Which statement best explains the scientific concern?

Accepted Answer

Different parallelism configurations change numerical reduction order and kernel selection, introducing variance unrelated to the attention-head variable.. An ablation study aims to attribute observed differences to one variable. When tensor and pipeline parallel sizes change between runs, the reduction order, kernel selection, and communication patterns change too, introducing numerical and convergence variance that is unrelated to attention head count. That makes the comparison confounded, even though optimizer sharding and micro-batching can be handled without altering the effective update or global batch size.

Answer

Attention head count is not a valid ablation variable because it is determined by hidden size and cannot be varied independently.

Answer

Changing parallelism alters the optimizer state sharding, so the effective learning rate changes per parameter.

Answer

Pipeline parallelism forces different micro-batch sizes, which always changes the global batch size and therefore the loss landscape.

Question 4

A research team is comparing three fine-tuning recipes for a NeMo LLM and wants the comparison to be defensible in a later review. Which two practices most improve the credibility of the reported comparison? (Choose two.)

Accepted Answer

Evaluate every recipe on the same held-out evaluation set with identical decoding settings.. Credible comparisons rest on two pillars: full provenance of each run so results are reproducible, and a shared, uncontaminated evaluation protocol so scores are measured identically. Provenance lets reviewers attribute differences to the recipes, while a fixed held-out set with consistent decoding settings ensures the numbers being compared actually measure the same thing.

Answer

Tune each recipe's learning rate separately until its score is maximized.

Answer

Reuse the evaluation set during training as an additional validation signal.

Answer

Report only the single best run for each recipe to keep the summary concise.

Question 5

An AI researcher is designing an experiment to compare two prompt templates for a customer-support LLM using NVIDIA NeMo. To ensure the comparison is fair and reproducible, which two practices should they follow? (Choose two.)

Accepted Answer

Use the same underlying LLM checkpoint and decoding parameters for both prompt templates.. A fair prompt-template comparison requires isolating the template as the only independent variable. Keeping the model checkpoint, decoding parameters, evaluation dataset, and scoring criteria identical across both conditions ensures that observed differences are caused by the prompt itself. This controlled design is the foundation of reproducible LLM experimentation and supports confident deployment decisions.

Answer

Vary the temperature for each template to explore a wider range of outputs.

Answer

Fine-tune the LLM separately for each prompt template before comparing them.

Answer

Use a different evaluation metric for each template to capture their unique strengths.

Question 6

An ML engineer is running a hyperparameter sweep with NVIDIA NeMo and notices that runs with identical configurations sometimes produce slightly different final loss values. Which cause should the engineer investigate first?

Accepted Answer

Nondeterministic GPU operations and unseeded randomness in the data pipeline. Run-to-run variation with identical configuration points to uncontrolled randomness and nondeterministic computation. Unseeded data shuffling changes which samples appear in which batch, and nondeterministic GPU kernels can accumulate floating-point differences. Together these produce slightly different loss trajectories. Validation set size and checkpoint intervals affect measurement and I/O, not the training computation itself.

Answer

The checkpoint saving interval is too frequent

Answer

The number of GPUs is too high for the batch size being used

Answer

The validation set is too small to measure loss accurately

Question 7

A research team is designing an experiment to measure how prompt phrasing affects the factuality of an LLM in a retrieval-augmented question-answering pipeline. Which two design choices are necessary to attribute observed factuality differences to the prompt rather than to other pipeline components? (Choose two.)

Accepted Answer

Hold the retrieved context, model checkpoint, and decoding parameters constant across prompt variants. Attributing factuality differences to prompt phrasing requires isolating the prompt as the only changed variable. Holding retrieval, model, and decoding settings constant removes competing explanations, while a shared evaluation set and rubric make scores comparable. Altering the retrieval index, temperature, or datasets per variant introduces confounds that make any observed improvement impossible to credit to the prompt itself.

Answer

Evaluate each prompt variant on a different dataset tailored to its strengths

Answer

Increase the temperature for variants that produce shorter answers to equalize response length

Answer

Allow the retrieval index to be rebuilt with different embedding models for each prompt variant

Question 8

A data scientist is fine-tuning a pretrained large language model on a small domain-specific dataset. The model achieves high accuracy on the training set but poor performance on a held-out validation set. Which technique is most likely to improve the model's generalization?

Accepted Answer

Apply regularization techniques such as dropout or weight decay.. The described symptoms—high training accuracy but poor validation accuracy—are classic signs of overfitting. Regularization techniques such as dropout, weight decay, or early stopping are designed to reduce overfitting by penalizing complexity or adding noise during training. Among the options, applying regularization directly targets the cause, whereas the others either exacerbate overfitting or do not address generalization.

Answer

Increase the number of training epochs to further reduce training loss.

Answer

Reduce the size of the validation set to decrease evaluation variance.

Answer

Increase the learning rate to speed up convergence on the training set.

Question 9

A research team is comparing two LoRA fine-tuning runs of the same Llama-based model in NeMo. Run 1 uses rank 8 and alpha 16; Run 2 uses rank 64 and alpha 16, with all other hyperparameters identical. Run 2 achieves lower training loss but worse accuracy on a held-out evaluation set. Which conclusion is most defensible from this experiment?

Accepted Answer

The rank-64 run overfits the training data, so the lower training loss does not translate to better generalization.. The pattern of lower training loss with worse held-out accuracy indicates that the higher-rank adapter used its additional capacity to fit training-specific noise. LoRA rank sets the dimensionality of the update matrices, so rank 64 has more trainable parameters than rank 8. The defensible conclusion is that the larger adapter overfit, and the more constrained adapter generalized better on this evaluation set.

Answer

The two runs cannot be compared because LoRA rank changes the base model architecture.

Answer

Higher LoRA rank always improves downstream accuracy, so the evaluation set must be mislabeled.

Answer

The alpha value should be doubled for Run 2 to restore the intended scaling ratio.

Question 10

A developer is building a customer-support assistant on NVIDIA NIM microservices. After a model update, responses that previously arrived in under 300 ms now take over two seconds, and the streaming client shows a long pause before the first token appears. GPU utilization is low and the prompt template was not changed. Which action should the developer take first to diagnose the regression?

Accepted Answer

Compare the time to first token (TTFT) and inter-token latency measured before and after the update to isolate whether the delay is in prefill or decode.. The reported symptom is a latency regression with low GPU utilization and unchanged prompts, so the developer must localize the delay before changing deployment configuration. Instrumenting time to first token and inter-token latency cleanly separates prompt prefill from token decode and reveals which phase regressed. Batching, parallelism, and disabling streaming either mask the signal or alter behavior, so measurement comes first.

Answer

Increase the tensor-parallel degree of the NIM container so the model weights are sharded across more GPUs.

Answer

Switch the client from streaming to a single non-streaming request so the full response is timed end to end.

Answer

Enable dynamic batching in the NIM microservice so concurrent requests are grouped into larger inference batches.

Question 11

A data scientist wants to determine how sensitive an LLM's summarization quality is to the temperature sampling parameter. They plan a sweep across several temperature values. Which experimental approach gives the clearest signal about temperature's effect?

Accepted Answer

Vary temperature while keeping the model, prompts, and evaluation metric constant.. To measure sensitivity to a single parameter, that parameter must be the only thing changing. Fixing the model, prompts, and evaluation metric ensures any quality difference across the sweep comes from temperature. This controlled setup produces a curve that shows whether summarization quality is robust or fragile with respect to the sampling temperature setting.

Answer

Use a different evaluation metric for each temperature value to capture more aspects of quality.

Answer

Change temperature and the prompt template together to explore the joint space faster.

Answer

Test only the lowest and highest temperature values to save compute.

Question 12

A data scientist is analyzing token-level loss values produced by an LLM evaluation run on a summarization dataset. Losses are stored as a list of floats, and most values cluster around 2.1, but a few exceed 9.0. The team wants a visualization that shows the shape of the loss distribution, including those extreme values, without hiding them through bin aggregation. Which visualization is most appropriate?

Accepted Answer

A strip plot (jittered scatter) of token-level loss. A strip plot plots every token-level loss as an individual mark, so extreme values remain visible instead of being merged into a bin or smoothed away. It reveals the distribution shape and the location of outliers simultaneously. Histograms, box plots, and violin plots each aggregate or summarize the data, which would obscure the few very high losses the team needs to inspect.

Answer

A histogram with 50 equal-width bins

Answer

A violin plot of token-level loss

Answer

A box plot of token-level loss

Question 13

An engineer at a customer-support automation company is experimenting with top-p sampling values for a NeMo-served LLM. They want to quantify how output diversity changes across settings without relying on human judgment alone. Which evaluation approach best supports this experiment?

Accepted Answer

Compute distinct-n and self-BLEU across a fixed prompt set for each top-p value.. To quantify output diversity across top-p settings, the engineer needs metrics that capture lexical variety and redundancy in generated text. Distinct-n and self-BLEU are computed automatically over a fixed prompt set and directly reflect diversity changes. Latency, GPU profiling, and training loss describe performance or training behavior, not the diversity of inference-time outputs, so they cannot answer the experiment's question.

Answer

Measure end-to-end inference latency with NVIDIA Triton Inference Server metrics.

Answer

Compare training loss curves from the original fine-tuning job.

Answer

Track GPU utilization and memory bandwidth during generation with Nsight Systems.

Question 14

During an ablation study on a retrieval-augmented LLM pipeline, the team removes the reranking stage and observes a large drop in answer accuracy on their benchmark. Before concluding that reranking is essential, which additional experiment is most important to run?

Accepted Answer

Run a control condition that keeps the pipeline identical except for a neutral change unrelated to reranking.. An ablation shows that a change in the pipeline correlates with an accuracy drop, but correlation is not causation unless other differences are ruled out. A control condition with a neutral modification tests whether the pipeline is sensitive to arbitrary changes. If the control shows little effect while removing reranking causes a large drop, the causal claim about reranking is substantially strengthened.

Answer

Increase the number of retrieved documents to compensate for the missing reranker.

Answer

Replace the benchmark with a larger one to increase statistical power.

Answer

Re-run the ablation with a different random seed to see if the accuracy drop persists.

Question 15

A data scientist wants to track training loss, learning rate, and GPU utilization side by side across thousands of steps in an interactive dashboard that supports comparing multiple runs. Which tool is designed specifically for this experiment-tracking and interactive visualization workflow?

Accepted Answer

Weights & Biases. Experiment-tracking platforms are purpose-built to ingest scalar metrics emitted during training and present them in interactive dashboards. Weights & Biases provides this out of the box, including live updates, run grouping, and side-by-side comparison, which are the exact capabilities requested. Generic plotting libraries and spreadsheet tools can render a single chart but cannot manage multi-run, multi-metric tracking at scale.

Answer

Excel

Answer

Pandas

Answer

Matplotlib