NCA-GENL Multiple Select — 30 Questions

Question 1

A team is fine-tuning an LLM and wants to detect whether individual training examples are causing unusually large gradient updates. They plan to visualize per-example gradient norms alongside other diagnostics. Which two visualizations are most appropriate for identifying these influential examples? (Choose two.)

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

A sorted bar chart of per-example gradient norms with examples ranked from largest to smallest. Detecting influential examples requires per-example gradient information. A sorted bar chart ranks examples so the largest norms stand out, while a scatter plot of gradient norm versus loss adds context about why those norms are large. Together they let the team isolate and inspect the examples driving unusually large updates.

Answer

A line chart of the moving average of total training loss across epochs

Answer

A pie chart showing the proportion of examples in each loss decile

Answer

A heatmap of the model's attention weights for a single randomly chosen example

Question 2

Which THREE factors influence the reproducibility of an LLM experiment?

Accepted Answer

Random seed initialization. Reproducibility requires strict control over the experimental environment. Because LLM training involves non-deterministic factors, controlling the random seeds, the exact data splits, and the library versions is paramount. Professional experiment tracking requires the ability to recreate identical results, which is only possible when every component of the pipeline is version-controlled and explicitly defined in the experiment's configuration manifest.

Answer

The ambient temperature of the datacenter

Answer

The number of hours the researcher works

Question 3

When designing an experiment to evaluate the performance of an LLM on a downstream classification task, which THREE factors should be controlled to ensure the results are comparable across different model sizes?

Accepted Answer

Decoding strategy (temperature, top-p). To conduct valid cross-model comparisons, one must neutralize external variables. Inference-time settings like decoding parameters, the specific prompt template, and the evaluation dataset must remain constant. If these vary, the results reflect differences in the evaluation environment rather than the intrinsic capabilities of the models being tested, rendering the experimentation data inconclusive for determining which model size is truly optimal for the specific classification application.

Answer

The learning rate used during fine-tuning

Answer

The hardware GPU generation (e.g., A100 vs H100)

Question 4

Which TWO factors should be considered when evaluating the cost-benefit of an LLM experimentation strategy?

Accepted Answer

Total GPU hours per experiment. Evaluating the cost-benefit of LLM experimentation requires balancing the financial cost of compute with the performance gains observed. In professional environments, experiments must be scoped to maximize meaningful insights while minimizing wasted GPU cycles. This involves prioritizing experiments that are likely to yield the highest impact on model quality relative to the resources consumed by the training runs.

Answer

The color scheme of the monitoring dashboard

Answer

The popularity of the LLM framework used

Answer

The number of research papers published by the team

Question 5

Which THREE of the following factors are critical when choosing a foundation model for an enterprise generative AI application?

Accepted Answer

Model licensing and data privacy policies. Selecting a foundation model involves balancing technical performance, operational costs, and security requirements. Enterprise readiness requires understanding data privacy policies, licensing, and the ability to fine-tune the model for domain-specific tasks. Failing to evaluate these factors can lead to significant downstream issues, including legal risks, poor performance, or unmanageable infrastructure costs, making this selection process one of the most important steps in an AI project lifecycle.

Answer

The number of hidden layers in the model

Answer

The specific programming language used to build the model

Question 6

An engineer is evaluating different prompting strategies (Zero-shot, Few-shot, Chain-of-Thought) for an RAG pipeline. Which TWO metrics are most effective for quantifying the quality of the generative output during this experimentation?

Accepted Answer

Faithfulness score. Quantitative evaluation is essential to move beyond subjective intuition in LLM experiments. Faithfulness (grounding in context) and Answer Relevance (usefulness to the query) provide distinct, measurable dimensions of performance. By measuring these, developers can iterate on prompts with empirical data, ensuring that changes to the prompt template actually improve system utility rather than just altering the verbosity or style of the generated response.

Answer

Average token generation speed

Answer

Number of model parameters

Answer

Total GPU memory consumption

Question 7

A data scientist is preparing to perform hyperparameter tuning for a Retrieval-Augmented Generation (RAG) system. Which TWO parameters should be prioritized for experimentation to improve retrieval accuracy?

Accepted Answer

Chunk size. Retrieval accuracy in RAG systems is heavily influenced by the chunking strategy and the similarity search configuration. Experimenting with these two components allows developers to optimize the context provided to the LLM. Properly tuned retrieval ensures that the model has the most relevant information, which is the most critical factor in reducing hallucinations and improving the factual grounding of generated responses.

Answer

Model quantization bit-width

Answer

System prompt length

Answer

GPU clock speed

Question 8

In the context of NVIDIA NeMo, which THREE actions are part of a robust experiment tracking workflow for fine-tuning?

Accepted Answer

Logging hyperparameters for every run. Robust experiment tracking is the foundation of reproducibility in machine learning. By logging configurations, monitoring metrics in real-time using tools like Weights & Biases or TensorBoard, and saving versioned checkpoints, researchers can compare results across iterations. These actions are vital for ensuring that performance gains are attributable to specific hyperparameter changes rather than random chance or environmental variations during training.

Answer

Deleting logs to conserve disk space

Answer

Manually calculating gradients during training

Question 9

Which THREE of the following strategies are recommended by NVIDIA to mitigate data leakage in enterprise-grade LLM applications?

Accepted Answer

Applying PII masking techniques on raw data prior to the training phase.. Data leakage occurs when sensitive information is unintentionally included in the model's training set or emitted in its response. Mitigating this requires a defense-in-depth approach: sanitizing training data to remove PII, implementing strict access controls for users, and utilizing output guardrails to intercept sensitive patterns (like credit card numbers) before they reach the user. These measures collectively protect user privacy and corporate integrity, ensuring that sensitive data remains secure throughout the model lifecycle.

Answer

Sharing the entire training dataset publicly to increase model transparency.

Answer

Increasing the model parameter count to improve internal data encryption.

Question 10

An enterprise AI researcher is conducting ablation studies on a large language model using NVIDIA NeMo. To ensure the experimentation results are scientifically valid and statistically sound, which THREE practices must be enforced during the study?

Accepted Answer

Evaluate across multiple random initialization seeds to account for stochastic variance in training.. Rigorous ablation studies require isolating individual components while holding all other experimental variables constant. Enforcing multiple random seeds accounts for initialization variance, controlling confounding hyperparameters prevents attribution errors, and applying rigorous statistical testing confirms whether observed performance deltas are statistically significant.

Answer

Vary multiple architectural hyperparameters simultaneously in every single experimental iteration.

Answer

Discard all experimental runs that fail to meet performance expectations without logging the failure.

Question 11

Which THREE actions are recommended for establishing a robust 'Human-in-the-Loop' (HITL) system for an AI deployment?

Accepted Answer

Setting specific confidence thresholds that trigger human review.. An effective Human-in-the-Loop system balances AI speed with human judgment. The three critical steps are defining clear escalation triggers, providing tools for human intervention, and maintaining a feedback loop where human corrections improve the model. These actions ensure that humans remain the ultimate authority in high-stakes scenarios, directly supporting the Trustworthy AI goals of oversight, accountability, and continuous improvement through expert guidance.

Answer

Automating all processes to eliminate the potential for human error.

Answer

Restricting human access to the model's internal weights and architecture.

Question 12

When evaluating LLM output quality using human-in-the-loop data, which THREE metrics or techniques are most effective for detecting systemic hallucinations?

Accepted Answer

Consistency check across multiple temperature settings.. Detecting systemic hallucinations requires a combination of automated consistency checks and structured human evaluation. Consistency across multiple temperature settings, NLI (Natural Language Inference) against ground truth, and human-labeled factuality scores provide a robust framework. By triangulating these metrics, developers can quantify the frequency and severity of model fabrications, which is critical for safety and reliability in generative AI deployment, ensuring users receive accurate and trustworthy information rather than confident but incorrect model responses.

Answer

Word count distribution analysis.

Answer

Model training loss convergence tracking.

Question 13

An ML engineer is setting up an experiment log for a fine-tuning run and wants to record the metadata necessary to reproduce the resulting model later. Which two items are most essential to capture for reproducibility? (Choose two.)

Accepted Answer

The random seed used for data shuffling and initialization.. Reproducibility requires capturing the inputs that determine the trained weights: the randomness source and the exact training data. The seed governs shuffling and initialization, and the dataset version pins the examples used. Together they let another engineer rerun the same configuration and obtain a comparable model, which is the practical definition of a reproducible experiment.

Answer

The names of the engineers who reviewed the training logs.

Answer

The wall-clock duration of each training epoch.

Answer

The GPU model and driver version used during training.

Question 14

You are designing an experiment to measure how quantization (FP16 versus INT8) affects inference latency and answer quality for an LLM deployed with NVIDIA TensorRT-LLM. Which two practices are required for the comparison to be valid? (Choose two.)

Accepted Answer

Benchmark both configurations on the same hardware and under the same batch size and concurrency conditions.. A valid quantization comparison must control everything except precision. Using identical prompts and generation parameters ensures the workload is the same, and benchmarking on identical hardware under identical batch and concurrency conditions ensures the environment is the same. Together these practices let the team attribute measured differences in latency and quality to FP16 versus INT8 rather than to confounds.

Answer

Retrain the model after quantization so the weights adapt to the lower precision.

Answer

Measure latency only on the first generated token, since subsequent tokens are less affected by precision.

Answer

Apply the same random seed to both configurations so token sampling is identical.

Question 15

A research team is running a series of controlled LLM fine-tuning experiments on NVIDIA DGX systems using NeMo Framework to compare two learning-rate schedules. They want the comparison to be scientifically valid and repeatable by another engineer next quarter. Which two practices are required to make the experiments reproducible? (Choose two.)

Accepted Answer

Record the exact dataset version, tokenizer configuration, and NeMo container image tag used.. Reproducibility in a controlled fine-tuning comparison requires controlling randomness and pinning the environment. Seeding all stochastic operations keeps data order, dropout, and initialization identical, while recording dataset version, tokenizer configuration, and container image tag lets another engineer rebuild the same setup. Performance-oriented choices such as GPU count, mixed precision, and inference benchmarking do not establish repeatability of the training comparison.

Answer

Deploy the resulting checkpoints to NVIDIA Triton Inference Server for latency benchmarking.

Answer

Increase the number of GPUs in every run so that throughput is maximized.

Answer

Enable mixed precision training to reduce memory usage on each DGX node.

Question 16

A research team is running a hyperparameter sweep over learning rate and warmup steps for a NeMo fine-tuning job. They notice that runs with identical hyperparameters produce different final validation losses across repeated executions. Which two changes would most directly improve the reproducibility of these experiments? (Choose two.)

Accepted Answer

Enable deterministic kernels and disable nondeterministic cuDNN algorithms in the framework settings.. Run-to-run variance in identical configurations usually comes from uncontrolled randomness in data order, initialization, dropout, and nondeterministic GPU kernels. Fixing the random seed and enabling deterministic kernels address both the algorithmic and hardware-level sources. Adding GPUs, logging, or lowering the learning rate change the experiment or its observability but do not make repeated runs converge.

Answer

Log the loss curve to TensorBoard so the team can visually compare runs.

Answer

Increase the number of GPUs per run so each experiment finishes faster.

Answer

Reduce the learning rate by a factor of ten across all sweep configurations.

Question 17

An engineer is setting up an automated experiment sweep over temperature and top-p for a NeMo-served LLM, and wants the results to be comparable and reproducible. Which two practices should be applied? (Choose two.)

Accepted Answer

Use the same prompt set and maximum token count across all temperature and top-p combinations.. Comparable decoding sweeps require controlling both stochasticity and inputs. Fixing the sampler seed makes each configuration repeatable, while holding prompts and maximum token count constant ensures the only varying factors are temperature and top-p. Changing the model, the prompt template, or removing parameter logs introduces confounds or destroys traceability, so those practices undermine the experiment.

Answer

Allow each trial to choose its own prompt template so the model can show its best behavior.

Answer

Increase the model's parameter count for trials with higher temperature to compensate for randomness.

Answer

Disable logging of per-trial parameters to reduce storage overhead during the sweep.

Question 18

A research team is comparing three fine-tuning recipes for a NeMo LLM and wants the comparison to be defensible in a later review. Which two practices most improve the credibility of the reported comparison? (Choose two.)

Accepted Answer

Evaluate every recipe on the same held-out evaluation set with identical decoding settings.. Credible comparisons rest on two pillars: full provenance of each run so results are reproducible, and a shared, uncontaminated evaluation protocol so scores are measured identically. Provenance lets reviewers attribute differences to the recipes, while a fixed held-out set with consistent decoding settings ensures the numbers being compared actually measure the same thing.

Answer

Tune each recipe's learning rate separately until its score is maximized.

Answer

Reuse the evaluation set during training as an additional validation signal.

Answer

Report only the single best run for each recipe to keep the summary concise.

Question 19

An AI researcher is designing an experiment to compare two prompt templates for a customer-support LLM using NVIDIA NeMo. To ensure the comparison is fair and reproducible, which two practices should they follow? (Choose two.)

Accepted Answer

Use the same underlying LLM checkpoint and decoding parameters for both prompt templates.. A fair prompt-template comparison requires isolating the template as the only independent variable. Keeping the model checkpoint, decoding parameters, evaluation dataset, and scoring criteria identical across both conditions ensures that observed differences are caused by the prompt itself. This controlled design is the foundation of reproducible LLM experimentation and supports confident deployment decisions.

Answer

Vary the temperature for each template to explore a wider range of outputs.

Answer

Fine-tune the LLM separately for each prompt template before comparing them.

Answer

Use a different evaluation metric for each template to capture their unique strengths.

Question 20

A research team is designing an experiment to measure how prompt phrasing affects the factuality of an LLM in a retrieval-augmented question-answering pipeline. Which two design choices are necessary to attribute observed factuality differences to the prompt rather than to other pipeline components? (Choose two.)

Accepted Answer

Hold the retrieved context, model checkpoint, and decoding parameters constant across prompt variants. Attributing factuality differences to prompt phrasing requires isolating the prompt as the only changed variable. Holding retrieval, model, and decoding settings constant removes competing explanations, while a shared evaluation set and rubric make scores comparable. Altering the retrieval index, temperature, or datasets per variant introduces confounds that make any observed improvement impossible to credit to the prompt itself.

Answer

Evaluate each prompt variant on a different dataset tailored to its strengths

Answer

Increase the temperature for variants that produce shorter answers to equalize response length

Answer

Allow the retrieval index to be rebuilt with different embedding models for each prompt variant