NCA-GENL Practice Test 12 — 15 Questions

Question 1

A research team is running a hyperparameter sweep for LoRA fine-tuning of a 13B-parameter LLM on NVIDIA GPUs. They notice that runs with identical configurations sometimes produce noticeably different evaluation scores, and the variance is larger than the differences between some of the hyperparameter settings being compared. Which action best addresses this problem?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Run each configuration with multiple fixed random seeds and compare mean evaluation scores with their confidence intervals.. When run-to-run variance exceeds the differences between configurations, single-run comparisons cannot support a conclusion. Repeating each configuration with several fixed seeds and reporting mean scores with confidence intervals turns noise into a measurable uncertainty, so the team can tell whether a hyperparameter effect is real. This is the standard remedy for high-variance sweeps and prevents selecting configurations on the basis of lucky seeds.

Answer

Report only the single best score achieved by each configuration, since the best result shows the model's potential.

Answer

Increase the learning rate for all runs so the model converges faster and the noise is averaged out.

Answer

Reduce the number of sweep configurations so fewer comparisons are affected by variance.

Question 2

You are designing an experiment to measure how quantization (FP16 versus INT8) affects inference latency and answer quality for an LLM deployed with NVIDIA TensorRT-LLM. Which two practices are required for the comparison to be valid? (Choose two.)

Accepted Answer

Benchmark both configurations on the same hardware and under the same batch size and concurrency conditions.. A valid quantization comparison must control everything except precision. Using identical prompts and generation parameters ensures the workload is the same, and benchmarking on identical hardware under identical batch and concurrency conditions ensures the environment is the same. Together these practices let the team attribute measured differences in latency and quality to FP16 versus INT8 rather than to confounds.

Answer

Retrain the model after quantization so the weights adapt to the lower precision.

Answer

Measure latency only on the first generated token, since subsequent tokens are less affected by precision.

Answer

Apply the same random seed to both configurations so token sampling is identical.

Question 3

A global e-commerce company uses an NVIDIA-powered LLM to generate product descriptions. They notice that for certain regions, the model occasionally produces content that violates local advertising regulations. To ensure Trustworthy AI, what is the most effective approach to prevent such violations?

Accepted Answer

Implement a region-aware content moderation layer that applies jurisdiction-specific rules before output. A region-aware content moderation layer is the most effective because it dynamically applies local rules to the model's output, ensuring compliance without retraining. It centralizes policy updates and can be integrated with NeMo Guardrails or custom filters. This approach balances scalability, cost, and regulatory adherence, directly supporting Trustworthy AI in a global deployment.

Answer

Fine-tune the model separately for each region using local regulatory data

Answer

Deploy a separate LLM for each region, each trained on local data

Answer

Use prompt engineering to instruct the model to avoid prohibited terms for each region

Question 4

A data science team is fine-tuning a Llama 3 8B model on a proprietary customer-support corpus using NVIDIA NeMo. They need to run dozens of experiments with different learning rates and batch sizes. Because the dataset contains personally identifiable information, they cannot send any telemetry to an external tracking server, but they still need to compare runs later and reproduce the best configuration. Which approach best satisfies both the reproducibility and data-privacy requirements?

Accepted Answer

Use the built-in NeMo experiment manager with a local file store, logging configs, metrics, and checkpoints to a shared on-premises directory.. The team needs reproducibility without external telemetry. NeMo's experiment manager with a local file store writes configurations, metrics, and checkpoints to a path the team controls, so nothing leaves the secure environment. External SaaS trackers violate the privacy constraint, checkpoints alone lack the metrics needed for comparison, and manual logging is unreliable and incomplete.

Answer

Enable Weights & Biases integration inside the NeMo experiment manager and configure the project to log only aggregate metrics.

Answer

Disable all logging and rely on the saved .nemo checkpoint files, since the checkpoint embeds the full training configuration.

Answer

Run each experiment in a separate container and manually copy the console output into a spreadsheet after each run.

Question 5

During an ablation study, a team removes the instruction-tuning stage from their NeMo pipeline and observes that the model still answers factual questions but frequently ignores the requested output format. They want to attribute this change in behavior to the removed stage rather than to noise. Which experimental design element is most important for supporting that attribution?

Accepted Answer

Run both the full pipeline and the ablated pipeline with all other variables held constant, then compare on the same evaluation set.. An ablation is only interpretable when the manipulated component is the sole difference between conditions. Keeping data, hyperparameters, training budget, and evaluation protocol identical, and scoring both variants on the same held-out set, lets the team attribute the observed format-compliance change to removing instruction tuning instead of to unrelated variation.

Answer

Increase the ablated model's training steps so it receives more optimization than the full pipeline.

Answer

Retrain the ablated model several times with different random seeds and report the highest format-compliance score.

Answer

Evaluate the ablated model with a different, more format-focused prompt template than the full pipeline.

Question 6

A research team is running a series of controlled LLM fine-tuning experiments on NVIDIA DGX systems using NeMo Framework to compare two learning-rate schedules. They want the comparison to be scientifically valid and repeatable by another engineer next quarter. Which two practices are required to make the experiments reproducible? (Choose two.)

Accepted Answer

Record the exact dataset version, tokenizer configuration, and NeMo container image tag used.. Reproducibility in a controlled fine-tuning comparison requires controlling randomness and pinning the environment. Seeding all stochastic operations keeps data order, dropout, and initialization identical, while recording dataset version, tokenizer configuration, and container image tag lets another engineer rebuild the same setup. Performance-oriented choices such as GPU count, mixed precision, and inference benchmarking do not establish repeatability of the training comparison.

Answer

Deploy the resulting checkpoints to NVIDIA Triton Inference Server for latency benchmarking.

Answer

Increase the number of GPUs in every run so that throughput is maximized.

Answer

Enable mixed precision training to reduce memory usage on each DGX node.

Question 7

You are comparing the inference throughput of an LLM served with two different batching strategies across a range of request arrival rates. You want a single visualization that shows both the median throughput and the variability at each arrival rate. Which visualization is most appropriate?

Accepted Answer

A box plot of throughput for each batching strategy, with arrival rate on the x-axis. A box plot is designed to show median and spread simultaneously, and placing one box per arrival rate per strategy makes the comparison direct. It handles skewed throughput distributions better than mean-only line charts and avoids the clutter of plotting every individual measurement.

Answer

A line chart of mean throughput only, with one line per batching strategy

Answer

A stacked bar chart of total throughput summed across arrival rates

Answer

A scatter plot of individual throughput measurements colored by batching strategy

Question 8

A research team is using NVIDIA NeMo to experiment with a large language model for a summarization task. They observe that the model's ROUGE scores vary significantly across different runs even when using the same hyperparameters and dataset. They suspect that non-deterministic operations in the training pipeline are causing this variance. Which step should they take to improve reproducibility of their experiment results?

Accepted Answer

Enable deterministic algorithms and set a fixed random seed in the NeMo training configuration.. To achieve reproducibility, the team must eliminate non-deterministic sources. Setting a fixed random seed and enabling deterministic algorithms ensures that random number generation, data shuffling, and GPU operations are consistent across runs. This directly reduces variance in ROUGE scores and allows other researchers to replicate the experiment exactly, which is a cornerstone of rigorous LLM experimentation.

Answer

Run the experiment multiple times and average the ROUGE scores to report a mean value.

Answer

Use a different optimizer with adaptive learning rates to stabilize training.

Answer

Increase the batch size to reduce the number of gradient updates per epoch.

Question 9

A financial services firm deploys an LLM assistant that summarizes earnings calls for analysts. Legal requires that the firm be able to reconstruct, months later, exactly which model version and prompt template produced a given summary, and that any later model update not silently change historical outputs. Which practice best meets this requirement?

Accepted Answer

Log prompt text, model identifier, model version hash, decoding parameters, and a timestamp for every generation, and pin the model version used for each summary.. Reproducibility and change control require per-generation records that tie each summary to its prompt, model identity, decoding settings, and time, plus deliberate version pinning so later updates cannot rewrite history. Comprehensive logging plus pinned versions gives auditors the evidence chain they need. Aggregate metrics and public changelogs lack the granularity and control to reconstruct or stabilize individual outputs.

Answer

Store only the final generated summary text in the analyst-facing document repository.

Answer

Enable a monthly email digest that lists the total number of summaries generated per analyst.

Answer

Rely on the model provider's public changelog to determine which version was active on any given date.

Question 10

A team is fine-tuning an NVIDIA NIM-hosted Llama 3 8B model and wants a single visualization that tracks per-step training loss, learning rate, and GPU memory utilization together, so they can correlate a mid-run loss spike with resource pressure. They need a framework that integrates natively with the NVIDIA NeMo training stack and requires minimal custom plotting code. Which visualization approach best meets these requirements?

Accepted Answer

Use the NVIDIA NeMo training logs with TensorBoard, logging loss, learning rate, and GPU memory as scalars on the same step axis.. The scenario requires a single, integrated view that ties training loss, learning rate, and GPU memory together on the same step axis. NeMo's built-in TensorBoard logging emits these as scalars during training, letting the team visually align a loss spike with resource pressure without writing custom plotting code. Post-hoc CSV plotting, embedding projections, and confusion matrices either lack the required metrics or cannot correlate them in real time.

Answer

Render a t-SNE projection of the model's embedding layer each epoch to observe representation drift.

Answer

Generate a confusion matrix from the validation set after each checkpoint to detect training instability.

Answer

Export per-step metrics to CSV and build a custom Matplotlib multi-panel figure after training completes.

Question 11

While reviewing training logs from a multi-node NVIDIA DGX cluster running data-parallel fine-tuning, you plot per-step gradient norm alongside loss. The gradient norm shows sharp periodic spikes every N steps that align with evaluation checkpoints. Which action should you take to determine whether the spikes are an artifact of the evaluation loop or a genuine optimization problem?

Accepted Answer

Recompute gradient norms only on training batches and exclude evaluation steps from the same plot.. Periodic spikes aligned with evaluation checkpoints suggest the measurement or the evaluation loop itself is perturbing the logged gradient norm, for example through mode switches or extra all-reduce operations. Recomputing and plotting gradient norms only for training steps removes that confound. If the spikes disappear, training optimization is fine and the artifact is in the evaluation path rather than the optimizer.

Answer

Increase the gradient clipping threshold until the spikes disappear.

Answer

Reduce the learning rate by half and observe whether the periodicity changes.

Answer

Switch from data parallelism to pipeline parallelism to eliminate the periodic pattern.

Question 12

You are running a NeMo fine-tuning experiment where validation loss decreases for the first three epochs, then rises steadily while training loss keeps falling. You want to confirm overfitting and select the most appropriate intervention. Which experiment action should you take first?

Accepted Answer

Add regularization such as dropout or weight decay, then re-run the experiment and compare validation curves.. Rising validation loss alongside falling training loss indicates the model is fitting training-specific patterns that do not generalize. Adding regularization such as dropout or weight decay directly limits effective capacity and is a controlled, reversible change. Re-running with the same split and seed isolates the intervention, so the resulting validation curve tells you whether overfitting was indeed the dominant cause.

Answer

Switch the optimizer from Adam to SGD without changing any other hyperparameter.

Answer

Reduce the size of the validation set to lower evaluation noise.

Answer

Increase the number of training epochs and re-run to see if validation loss recovers.

Question 13

A data scientist is running a fine-tuning experiment with NVIDIA NeMo on a single A100 GPU. They want to establish a repeatable baseline before sweeping any hyperparameters, so that a later run can be compared fairly. Which practice best supports this goal?

Accepted Answer

Fix a random seed in the NeMo training configuration and record the exact dataset version, model checkpoint, and NeMo container tag used.. A trustworthy baseline requires controlling every input that affects the result, then recording those inputs so the run can be repeated. Fixing the random seed, freezing the dataset version, naming the starting checkpoint, and noting the NeMo container tag together make the run reproducible and comparable. Performance tweaks and best-of-N reporting change or obscure the result rather than establishing a stable reference point.

Answer

Increase the batch size to the maximum the GPU memory allows so the baseline finishes quickly.

Answer

Enable automatic mixed precision and Tensor Core acceleration to reduce training time.

Answer

Run the experiment three times and report the best validation loss observed.

Question 14

You are analyzing a dataset of 50,000 LLM training samples and want to visualize how sample lengths are distributed to decide on a maximum sequence length cutoff. The lengths range from 10 to 8,000 tokens with a long right tail. Which visualization should you use to best reveal the shape, central tendency, and outliers of this single continuous variable?

Accepted Answer

A histogram with 50 bins and a log-scaled x-axis. A histogram is the standard univariate visualization for a continuous variable because bin heights encode frequency and expose shape, center, and outliers. With a range spanning three orders of magnitude, a log-scaled x-axis prevents the low-token region from collapsing into a single bar. This combination lets the data scientist choose a cutoff where sample density drops, balancing truncation loss against compute cost.

Answer

A scatter plot of token length versus sample index

Answer

A line chart of cumulative sample count sorted by token length

Answer

A pie chart of samples grouped into five equal token-length ranges

Question 15

During a RAG evaluation, a data scientist computes cosine similarity between 40,000 query embeddings and 40,000 retrieved-chunk embeddings using an NVIDIA-accelerated pipeline. They then reduce the 4,096-dimensional vectors with t-SNE to 2D for a scatter plot, but the plot shows no separation between relevant and irrelevant retrievals. What is the MOST likely reason the visualization fails to reveal the retrieval quality signal?

Accepted Answer

t-SNE collapses high-dimensional structure into two dimensions and its perplexity and local-neighborhood objective can obscure the few dimensions that separate relevant from irrelevant chunks.. t-SNE is a nonlinear local-neighborhood method whose output depends on perplexity and on which local structure dominates. With 4,096-dimensional embeddings, the dimensions that separate relevant from irrelevant chunks may carry little of the local variance t-SNE preserves, so the two classes interleave in the projection. A supervised or linear method that targets the label would surface the signal that t-SNE hides.

Answer

Cosine similarity cannot be computed on 4,096-dimensional embeddings, so the input matrix was invalid.

Answer

t-SNE preserves global distances, so the relevant and irrelevant points should be separated and the plot must be mislabeled.

Answer

t-SNE is showing the raw cosine similarities rather than the embedding structure, so the relevant and irrelevant points overlap by construction.