NCA-GENL Experimentation Practice Test 3 — 15 Questions

Question 1

A researcher is using NVIDIA's 'TensorRT-LLM' to optimize an LLM. During the experimentation phase, they observe the model's accuracy drops significantly after quantization. What is the most appropriate next step?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Use a representative calibration dataset for quantization.. Post-training quantization often introduces errors that degrade model accuracy. To mitigate this, techniques like 'Quantization-Aware Training' (QAT) or using a calibration dataset are essential. These methods help the model adapt to the lower precision format during or after the process. Mastering these techniques is critical for delivering high-performance, resource-efficient models that maintain their accuracy in production environments.

Answer

Revert to full FP32 training without any optimization.

Answer

Increase the model's hidden dimension size.

Answer

Switch to a smaller base model architecture.

Question 2

Which of the following is a primary objective of 'Ablation Studies' in LLM experimentation?

Accepted Answer

To determine the contribution of individual components.. Ablation studies systematically remove components (like layers, heads, or data sources) to measure their specific contribution to model performance. This process is essential for understanding the model's architecture and optimizing it by removing redundant or inefficient parts. For NVIDIA-certified professionals, ablation studies provide the empirical evidence needed to defend design choices and justify model architectural simplifications in complex projects.

Answer

To increase the total number of parameters in the model.

Answer

To accelerate the training speed by using less data.

Answer

To debug and fix errors in the model's training code.

Question 3

Refer to the exhibit. The experiment fails with an Out-of-Memory (OOM) error during the second epoch. Given the configuration, which change is most effective for immediate stabilization?

Accepted Answer

Reduce batch_size to 16.. The OOM error indicates the memory footprint exceeds the available VRAM, likely due to activation growth or KV cache accumulation. Reducing the batch size is the most direct way to lower memory consumption per iteration. Experimentation requires iterative adjustment of hyperparameters; scaling back memory-intensive settings allows the job to complete successfully, providing a baseline from which performance optimizations like gradient accumulation can then be systematically applied to restore throughput.

Answer

Switch precision to FP32.

Answer

Increase batch_size to 64.

Answer

Increase max_seq_len to 8192.

Question 4

An engineer is evaluating different prompting strategies (Zero-shot, Few-shot, Chain-of-Thought) for an RAG pipeline. Which TWO metrics are most effective for quantifying the quality of the generative output during this experimentation?

Accepted Answer

Faithfulness score. Quantitative evaluation is essential to move beyond subjective intuition in LLM experiments. Faithfulness (grounding in context) and Answer Relevance (usefulness to the query) provide distinct, measurable dimensions of performance. By measuring these, developers can iterate on prompts with empirical data, ensuring that changes to the prompt template actually improve system utility rather than just altering the verbosity or style of the generated response.

Answer

Average token generation speed

Answer

Number of model parameters

Answer

Total GPU memory consumption

Question 5

During LLM experimentation, what is the primary purpose of maintaining a consistent 'seed' value across different runs?

Accepted Answer

To ensure reproducibility of experimental results.. Consistency is the cornerstone of empirical science. By fixing the random seed, researchers ensure that weight initialization and data shuffling occur identically across runs. This allows them to isolate the impact of specific hyperparameter changes, such as learning rate or batch size, without the confounding variable of stochastic randomness, which is vital for reproducible and meaningful comparative analysis in complex machine learning workflows.

Answer

To increase the training speed of the model.

Answer

To reduce the VRAM usage during training.

Answer

To prevent overfitting on the training set.

Question 6

A researcher is experimenting with prompt-tuning and finds that the model output is repetitive. They decide to adjust the sampling hyperparameters. Which combination of changes is most likely to increase the diversity of the output?

Accepted Answer

Increase temperature and increase top-p.. Sampling parameters control the trade-off between coherence and creativity. Increasing temperature shifts the probability distribution, allowing for less likely tokens to be selected, while increasing top-p (nucleus sampling) expands the set of tokens considered. Balancing these allows researchers to explore the model's creative range during experimentation, ensuring the output is varied enough to be useful while maintaining sufficient logical coherence for the application's specific requirements.

Answer

Decrease temperature and decrease top-p.

Answer

Set temperature to 0 and top-p to 1.

Answer

Increase frequency penalty and decrease top-p.

Question 7

When designing an experiment to evaluate the performance of an LLM on a downstream classification task, which THREE factors should be controlled to ensure the results are comparable across different model sizes?

Accepted Answer

Decoding strategy (temperature, top-p). To conduct valid cross-model comparisons, one must neutralize external variables. Inference-time settings like decoding parameters, the specific prompt template, and the evaluation dataset must remain constant. If these vary, the results reflect differences in the evaluation environment rather than the intrinsic capabilities of the models being tested, rendering the experimentation data inconclusive for determining which model size is truly optimal for the specific classification application.

Answer

The learning rate used during fine-tuning

Answer

The hardware GPU generation (e.g., A100 vs H100)

Question 8

You are experimenting with RAG and notice the model is frequently ignoring the provided context. Which of the following is the most likely culprit to investigate first?

Accepted Answer

The prompt instruction strength.. LLMs often exhibit 'pre-training bias,' where they prioritize their internal knowledge base over the provided context. If the context is ignored, the retrieval quality or the prompt's instruction is usually insufficient to override this behavior. Experimentation should focus on prompt engineering—specifically strengthening the system instructions to enforce context usage—before attempting more complex architectural changes to the retrieval process or the model itself.

Answer

The GPU clock speed.

Answer

The number of training epochs.

Answer

The total number of parameters in the model.

Question 9

Refer to the exhibit. You are running a multi-node distributed fine-tuning experiment and receive this error. What does this indicate about your experimentation environment?

Accepted Answer

There is an issue with the cluster interconnect.. This NCCL (NVIDIA Collective Communications Library) error signifies a breakdown in inter-node communication. In distributed training, nodes must synchronize gradients; a reset connection implies that the network fabric is failing to maintain the persistent connections required for this synchronization. This is a critical infrastructure issue that prevents the experiment from proceeding, requiring a check of the interconnects, such as InfiniBand or Ethernet switches, before any further experimentation can occur.

Answer

The model weights are corrupted.

Answer

The learning rate is too high.

Answer

The batch size is too large.

Question 10

Which of the following best describes the role of 'AB testing' in Generative AI experimentation?

Accepted Answer

It validates performance improvements with real user feedback.. AB testing is a controlled method for comparing two variations of a generative system—such as different prompt templates or model versions—using real-world user interactions. By splitting traffic, researchers can gather empirical evidence on which variation yields superior user metrics. This is the gold standard for validating whether experimental improvements in a lab environment translate into actual value for the end-users in a production setting.

Answer

It is used to calculate the model's perplexity.

Answer

It automates the hyperparameter tuning process.

Answer

It eliminates the need for any offline evaluation.

Question 11

When fine-tuning a model for domain-specific tasks, which THREE metrics should you monitor during the training phase to ensure the experiment is progressing healthily?

Accepted Answer

Training Loss. Monitoring training dynamics is crucial for detecting issues early. Training loss confirms the optimizer is reducing error, while validation loss catches overfitting before it becomes permanent. Gradient norm tracking is the most reliable way to identify instability—such as the infamous 'exploding gradients'—which is common in complex LLM training. Together, these metrics form the dashboard necessary for a data scientist to make informed decisions about when to stop, adjust, or continue their experiments.

Answer

Model disk usage

Answer

Total number of GPUs used

Question 12

You are running an experiment comparing two different fine-tuning methods (Full Fine-tuning vs. PEFT). Why is it crucial to keep the dataset and evaluation benchmark identical?

Accepted Answer

To ensure that performance differences reflect the tuning method.. Scientific rigor demands that all variables except the independent variable be controlled. If the dataset or evaluation criteria differ, it becomes impossible to determine if the performance variance is caused by the fine-tuning method or by the inherent differences in the evaluation content. Consistent evaluation acts as a 'control' for the experiment, ensuring that the findings regarding the effectiveness of the fine-tuning technique are statistically valid and replicable.

Answer

To minimize the computational cost of the experiment.

Answer

To prevent the model from crashing due to memory issues.

Answer

To speed up the training time of the PEFT approach.

Question 13

When experimenting with synthetic data generation to improve model performance, what is the most important risk to monitor?

Accepted Answer

The introduction of model bias or hallucinations.. Synthetic data can inadvertently contain artifacts or biases present in the teacher model. Over time, 'model collapse'—where the model learns from its own generated noise—can occur, leading to a degradation in performance. Regular evaluation on a held-out, human-verified dataset is essential to ensure that the synthetic data is actually improving the model's ability to reason, rather than just forcing it to mirror the stylistic flaws of the synthetic data source.

Answer

The increase in training time.

Answer

The file format of the training data.

Answer

The cost of disk storage.

Question 14

An AI engineer at an automotive enterprise is running an LLM experimentation pipeline using NeMo. The primary objective is to evaluate how different prompt engineering strategies affect the model's spatial reasoning capabilities across diverse spatial datasets. Which foundational workflow step should be prioritized to ensure reproducible experimental results?

Accepted Answer

Fixing random seeds across all libraries and versioning the evaluation datasets.. Establishing strict deterministic seeds and dataset versioning is foundational for reliable LLM experimentation. Without fixed random seeds and tracked dataset states, variability in generation outputs prevents meaningful comparison between distinct prompt strategies, undermining the scientific validity of the enterprise validation process.

Answer

Maximizing GPU batch size to accelerate throughput during the prompt evaluation phase.

Answer

Utilizing mixed precision FP16 computations to reduce memory footprint across nodes.

Answer

Upgrading the cluster interconnect fabric to InfiniBand for faster inter-GPU communication.

Question 15

During an LLM experimentation phase using NVIDIA NeMo, an ML engineer needs to systematically track hyperparameters, dataset lineage, and evaluation artifacts to meet strict auditing standards. Which TWO actions should the engineer take to achieve comprehensive experiment tracking?

Accepted Answer

Integrate an experiment tracking framework like MLflow to log hyperparameters and metrics automatically.. Effective experiment tracking in enterprise generative AI requires systematic logging of both code configurations and artifact lineage. Integrating MLflow or NeMo-native logging captures hyperparameters, while artifact stores maintain dataset versions and model weights, ensuring full traceability and regulatory compliance during model evaluation.

Answer

Rely solely on manual terminal output logs stored in local temporary directories.

Answer

Disable all logging mechanisms to maximize GPU execution speed and prevent I/O bottlenecks.

Answer

Store all intermediate checkpoints in volatile system memory without persisting to disk.