NCA-GENL Practice Test 24 — 15 Questions

Question 1

When implementing RLHF (Reinforcement Learning from Human Feedback), why is diversity in the human rater pool essential for Trustworthy AI?

Accepted Answer

To ensure the model aligns with a wide range of human values and reduces bias.. Diversity in the rater pool prevents the model from aligning solely with the cultural or subjective preferences of a single demographic. If the raters are not diverse, the model may inadvertently learn biases that alienate specific user groups or reinforce narrow worldviews. Ensuring a broad range of perspectives during the feedback phase is critical for creating an AI that is universally helpful, respectful, and reflective of a global user base.

Answer

To increase the total number of data points, thereby lowering training costs.

Answer

To allow the model to learn multiple languages more efficiently during fine-tuning.

Answer

To optimize the GPU memory usage during the reinforcement learning phase.

Question 2

A team is deploying a large language model for real-time text generation and notices that inference latency is too high. They want to reduce latency without retraining the model. Which technique is most appropriate?

Accepted Answer

Apply quantization to reduce the model's precision from FP32 to FP16 or INT8.. Quantization reduces the precision of model parameters, which lowers memory bandwidth and speeds up computation. It can be applied without retraining and is a common technique for reducing inference latency. Other options either increase latency or do not affect it. Thus, quantization is the most appropriate choice.

Answer

Increase the batch size for inference requests.

Answer

Use a larger model with more parameters.

Answer

Train the model for additional epochs to improve its efficiency.

Question 3

When evaluating an LLM for factual accuracy, which metric is most effective at detecting hallucinations compared to simple word-overlap metrics?

Accepted Answer

NLI-based entailment scoring. Hallucinations in LLMs are semantic errors that word-overlap metrics like BLEU or ROUGE fail to capture, as they only measure token matching. Model-based evaluation, using frameworks like RAGAS or NLI (Natural Language Inference) models, assesses the truthfulness of generated content against reference contexts. This is critical for building trustworthy enterprise AI, where factual accuracy is paramount and simple text similarity does not suffice for assessing the validity of generated information.

Answer

BLEU Score

Answer

ROUGE-L

Answer

Perplexity

Question 4

Which THREE of the following are valid methods for improving the inference performance of a deployed deep learning model?

Accepted Answer

Quantization. Optimizing inference is critical for real-time AI applications. Techniques like quantization reduce precision, lowering memory footprint and increasing throughput. Model pruning removes redundant parameters, reducing computational cost without significant loss in accuracy. Finally, knowledge distillation transfers the capabilities of a large, high-performance 'teacher' model into a smaller, faster 'student' model, providing a highly optimized deployment candidate that maintains the intelligence of its larger predecessor while being significantly more efficient.

Answer

Increasing the number of hidden layers

Answer

Data Augmentation

Question 5

Which of the following best describes the role of 'AB testing' in Generative AI experimentation?

Accepted Answer

It validates performance improvements with real user feedback.. AB testing is a controlled method for comparing two variations of a generative system—such as different prompt templates or model versions—using real-world user interactions. By splitting traffic, researchers can gather empirical evidence on which variation yields superior user metrics. This is the gold standard for validating whether experimental improvements in a lab environment translate into actual value for the end-users in a production setting.

Answer

It is used to calculate the model's perplexity.

Answer

It automates the hyperparameter tuning process.

Answer

It eliminates the need for any offline evaluation.

Question 6

In the context of generative AI, what is the 'mode collapse' problem in GANs, and why is it a significant challenge?

Accepted Answer

The generator fails to produce diverse samples.. Mode collapse occurs when the generator in a Generative Adversarial Network learns to map several input noise vectors to the same output or a very limited set of outputs. This prevents the generator from capturing the full diversity of the target data distribution. It is a major challenge because it defeats the purpose of generative modeling, which is to produce a wide range of realistic, diverse samples that represent the training distribution.

Answer

The generator output becomes purely random noise.

Answer

The discriminator becomes too powerful to train.

Answer

Training loss increases monotonically to infinity.

Question 7

Refer to the exhibit. The experiment fails with an Out-of-Memory (OOM) error during the second epoch. Given the configuration, which change is most effective for immediate stabilization?

Accepted Answer

Reduce batch_size to 16.. The OOM error indicates the memory footprint exceeds the available VRAM, likely due to activation growth or KV cache accumulation. Reducing the batch size is the most direct way to lower memory consumption per iteration. Experimentation requires iterative adjustment of hyperparameters; scaling back memory-intensive settings allows the job to complete successfully, providing a baseline from which performance optimizations like gradient accumulation can then be systematically applied to restore throughput.

Answer

Switch precision to FP32.

Answer

Increase batch_size to 64.

Answer

Increase max_seq_len to 8192.

Question 8

Which THREE practices are recommended to minimize 'Data Leakage' in generative AI applications?

Accepted Answer

Automated PII redaction during the data pre-processing phase.. Data leakage occurs when sensitive or private information is inadvertently included in training data or revealed through model outputs. Minimizing this requires PII redaction, strict access control, and output filtering. These measures are essential for Trustworthy AI because protecting user privacy is a foundational responsibility, and failure to control data flow can lead to significant legal, ethical, and reputational damage to an organization.

Answer

Using public, unverified data sources to maximize training diversity.

Answer

Increasing the learning rate to ensure faster convergence and data masking.

Question 9

Which component of an NVIDIA AI stack is primarily responsible for providing a low-level API for high-performance collective communication primitives across multi-GPU nodes?

Accepted Answer

NCCL. NCCL (NVIDIA Collective Communications Library) is the industry standard for multi-GPU communication. It is designed to provide high-performance primitives such as All-Reduce, All-Gather, and Broadcast, which are essential for distributed training and inference. Understanding NCCL is fundamental for scaling models across nodes, as it directly impacts the efficiency of gradient synchronization and model parallel strategies in high-performance computing environments.

Answer

cuBLAS

Answer

cuDNN

Answer

TensorRT

Question 10

A developer is debugging a TensorRT-LLM generation that intermittently produces truncated responses when many users submit long prompts concurrently. Logs show requests completing without errors, but outputs stop mid-sentence. Which configuration change is most likely to resolve this?

Accepted Answer

Increase the maximum sequence length setting so the combined prompt and generated tokens fit within the engine's configured limit.. Silent mid-sentence termination with no error typically means generation reached the engine's maximum sequence length, which caps the sum of prompt and generated tokens. Concurrent long prompts consume that budget, leaving little room for completion. Raising the maximum sequence length removes the artificial boundary. Parallelism, sampling settings, and batch size influence other dimensions of behavior and do not extend the token budget.

Answer

Enable pipeline parallelism across the available GPUs to distribute the long prompts across multiple devices.

Answer

Lower the sampling temperature and top-p values so the model produces shorter, more deterministic completions.

Answer

Reduce the maximum batch size so each concurrent request receives a larger share of the KV cache blocks.

Question 11

An ML engineer is comparing two fine-tuned variants of the same base LLM on an internal question-answering benchmark. Variant A scores higher on exact-match accuracy, while Variant B produces answers that human reviewers rate as more helpful and better formatted. The team must choose one variant to ship to production. Which evaluation approach is most appropriate for making this decision?

Accepted Answer

Combine automated metrics with a structured human-preference evaluation, and choose based on the product's primary success criterion.. For generative LLM tasks, automated metrics and human preference capture different quality dimensions. The right decision combines both and weights them by the product's primary success criterion. Shipping on accuracy alone ignores user-perceived helpfulness, shipping on preference alone ignores objective regressions, and averaging without justified weights hides the real trade-off.

Answer

Average the exact-match score and a normalized human-preference score into a single number and pick the higher value.

Answer

Ship Variant A because exact-match accuracy is an objective metric and human ratings are subjective.

Answer

Ship Variant B because human preference is the only metric that matters for generative models.

Question 12

A developer wants an LLM application to answer questions about an internal knowledge base that changes daily. Rather than retraining the model, they plan to retrieve relevant passages at query time and place them into the prompt. Which approach are they implementing?

Accepted Answer

Retrieval-augmented generation, where an embedding index supplies context for the prompt.. Answering over a frequently changing corpus without retraining is the defining use case for retrieval-augmented generation: an embedding index is refreshed as documents change, and retrieved passages are injected into the prompt at query time. Fine-tuning variants and few-shot prompting all fix knowledge at build time or prompt-authoring time.

Answer

Few-shot prompting, where several worked examples are prepended to every request.

Answer

Supervised fine-tuning, where labeled question-answer pairs update the model weights.

Answer

Parameter-efficient fine-tuning with LoRA adapters trained on the knowledge base.

Question 13

A team runs a NeMo fine-tuning experiment and observes that validation loss decreases for several epochs and then steadily rises while training loss keeps falling. They want to confirm whether the checkpoint from the best validation epoch is genuinely better than the final checkpoint. Which action provides the strongest evidence?

Accepted Answer

Evaluate both checkpoints on a separate held-out test set and compare task metrics. When a checkpoint is selected using validation loss, that metric becomes optimistically biased for the chosen model. The strongest evidence comes from an untouched test set that was not involved in any selection decision. Evaluating both checkpoints there with task-relevant metrics yields an unbiased comparison, whereas training loss and repeated validation checks cannot settle the question.

Answer

Compare the training loss values of the two checkpoints

Answer

Continue training the final checkpoint for more epochs and recheck validation loss

Answer

Pick whichever checkpoint has the lower validation loss

Question 14

An ML engineer is using NVIDIA NeMo to evaluate a retrieval-augmented generation pipeline. They want to measure whether adding a reranker improves answer faithfulness, but they must ensure the experiment is reproducible and comparable across runs. Which practice best supports a valid comparison between the pipeline with and without the reranker?

Accepted Answer

Use the same evaluation dataset, same base LLM checkpoint, and same decoding parameters for both pipeline variants, changing only the reranker component.. To attribute a change in faithfulness to the reranker, every other element of the RAG pipeline must be held constant. Using the same evaluation set, base checkpoint, and decoding parameters for both variants ensures the only difference is the presence or absence of the reranker. This controlled design yields a clean, reproducible comparison and avoids confounding variables that would invalidate the result.

Answer

Increase the temperature of the LLM in the reranker variant to encourage more diverse answers and better faithfulness.

Answer

Evaluate the reranker variant on a larger and more diverse dataset than the baseline so the results are more statistically significant.

Answer

Run the baseline and reranker variants on different GPU models to test robustness across hardware configurations.

Question 15

Which of the following describes the purpose of a 'Validation Set' during the model experimentation cycle?

Accepted Answer

It provides a mechanism to tune hyperparameters iteratively.. The validation set is used during training to monitor performance and tune hyperparameters without leaking information from the test set. It acts as an objective checkpoint for model improvement, allowing the researcher to stop training if overfitting occurs. Proper use of this set is a hallmark of robust experimentation, ensuring that the final model generalizes to unseen data in real-world production environments.

Answer

It is used for the final performance evaluation of the model.

Answer

It replaces the training set to reduce compute usage.

Answer

It acts as a buffer to store temporary model weights.