NCP-GENL Practice Test 7 — 10 Questions

Question 1

When assessing the quality of a dataset for instruction fine-tuning, which TWO metrics or methods are considered most reliable for measuring dataset diversity?

Accepted Answer

Embedding-based clustering to visualize topical coverage across the corpus.. Dataset diversity ensures that the model encounters a broad range of topics and linguistic patterns, which is critical for generalization. Using embedding-based clustering allows for the identification of thematic coverage, while perplexity distribution analysis helps assess whether the dataset contains a balance of common and complex structures. These methods together provide a quantitative view of the data's breadth, reducing the risk of bias or overfitting.

Answer

Calculating the total number of words in the dataset.

Answer

Measuring the average response length for every instruction.

Answer

Checking if the dataset is solely in ASCII format.

Question 2

Why is it important to perform 'domain-specific' data cleaning when preparing a corpus for fine-tuning a medical LLM?

Accepted Answer

Generic cleaning often removes or modifies critical domain-specific nomenclature.. Medical text contains specific jargon, abbreviations, and relationships that generic cleaning might misinterpret. Generic tools often remove or alter terms that are critical to medical context, such as drug names or procedural codes. By using domain-specific cleaning, you ensure that the model retains the precise terminology necessary for high-stakes, accurate clinical reasoning in medical applications.

Answer

It helps to anonymize the patient data by replacing all medical terms with generic labels.

Answer

It ensures that the dataset size is significantly reduced to fit in GPU cache.

Answer

It automatically corrects all scientific inaccuracies present in the original documents.

Question 3

Which metric is most critical to monitor for identifying 'bottlenecks' in a high-throughput LLM deployment?

Accepted Answer

GPU Utilization percentage.. GPU utilization is the primary metric for understanding how well the model is saturating the compute resources. If GPU utilization is low while latency is high, it suggests a bottleneck elsewhere, such as CPU preprocessing, data transfer, or synchronization issues. Monitoring this metric allows engineers to determine if they need to increase batch sizes, optimize the pipeline, or scale the infrastructure to maintain performance.

Answer

Total number of files in the model repository.

Answer

The name of the backend framework used for training.

Answer

The number of times the server was restarted.

Question 4

An LLM application deployed on NVIDIA Triton Inference Server is experiencing intermittent latency spikes. Which metric is most critical to monitor to determine if the issue stems from GPU compute saturation?

Accepted Answer

SM (Streaming Multiprocessor) occupancy. Monitoring GPU utilization alone is insufficient because it does not distinguish between compute saturation and memory bandwidth bottlenecks. NVML-based metrics specifically tracking SM (Streaming Multiprocessor) occupancy provide the granular insight required to identify whether the execution cores are fully utilized. This metric is essential for capacity planning and ensuring that the inference throughput meets strict service level agreements under high concurrent request loads in production environments.

Answer

Total system CPU usage

Answer

Network interface throughput

Answer

Total GPU memory capacity

Question 5

A team is testing a new LLM application. During red-teaming, the model consistently leaks sensitive internal project codenames. How should the team address this systematically?

Accepted Answer

Deploy a guardrail that filters output against a list of sensitive terms.. Systematic mitigation of data leakage requires a multi-layered approach. Modifying the base model is rarely sufficient; instead, one must implement output-side guardrails that perform pattern matching and dictionary-based filtering. This ensures that even if the model attempts to generate sensitive info, the guardrail intercepts it. This process protects intellectual property and maintains compliance with corporate confidentiality agreements by ensuring that protected information remains strictly within the secure environment.

Answer

Increase the number of training epochs on the existing dataset.

Answer

Add a disclaimer at the end of every response stating that the content is confidential.

Answer

Randomize the model's weights during every inference run.

Question 6

An enterprise fine-tunes a Llama-3-70B model using NVIDIA NeMo for automated technical support ticketing. The development team needs an automated evaluation pipeline that measures semantic similarity against human-curated reference answers without relying on costly human annotators. Which metric provides the most robust embedding-based semantic similarity assessment for this scenario?

Accepted Answer

BERTScore computes token similarity matrices using contextual embeddings to capture deep semantic meaning regardless of surface-level phrasing variations.. BERTScore leverages contextual embeddings from transformer models to evaluate token-level semantic overlap rather than exact string matching, making it ideal for technical support text where phrasing varies. This automated metric correlates strongly with human judgment, significantly accelerating iteration cycles during enterprise model development workflows on NVIDIA infrastructure.

Answer

ROUGE-1 measures unigram overlap but fails to capture semantic synonyms and contextual nuances common in enterprise technical support documentation.

Answer

BLEU evaluates n-gram precision with a brevity penalty, heavily penalizing valid creative paraphrasing typically found in conversational AI support responses.

Answer

Perplexity measures how well a probability distribution predicts a sample, reflecting language fluency rather than semantic alignment with a specific reference answer.

Question 7

Which TWO of the following practices are recommended for ensuring ethical AI development when using NVIDIA NIMs in an enterprise environment?

Accepted Answer

Perform periodic bias audits on model responses using diverse, representative evaluation datasets.. Ensuring ethical AI involves both technical validation and process transparency. Regular bias auditing helps detect unintended stereotyping, while implementing robust logging ensures accountability for every generated output. These practices allow organizations to monitor for discriminatory patterns and maintain an audit trail for compliance, which is essential for building trust with users and adhering to global ethical standards for AI deployment in sensitive business sectors.

Answer

Bypass local logging to optimize inference latency for high-throughput applications.

Answer

Use the model in a closed-loop system where users cannot report offensive content.

Answer

Exclude all metadata from model outputs to prevent potential privacy leaks.

Question 8

An engineer is designing a Transformer-based model for long-context document summarization. They decide to replace the standard dense self-attention mechanism with a sliding window attention approach. What is the primary architectural implication of this change?

Accepted Answer

The computational complexity of the self-attention mechanism is reduced from quadratic to linear.. Sliding window attention restricts the receptive field of each token to a local neighborhood, drastically reducing the quadratic memory complexity of standard attention to linear. This is critical for scaling LLMs to long contexts, as it prevents the O(n²) memory growth that typically causes GPU out-of-memory errors on large input sequences while maintaining local coherence.

Answer

The model's total parameter count increases due to added positional embedding layers.

Answer

The model loses the ability to perform parallel training across multiple GPU nodes.

Answer

The model is no longer compatible with standard softmax normalization functions.

Question 9

An enterprise deployment of NeMo Guardrails is experiencing hallucinations where the model provides medical advice despite strict system prompts. What is the most effective approach to mitigate this risk?

Accepted Answer

Implement NeMo Guardrails 'dialogue rails' to detect and redirect queries related to medical diagnosis.. NeMo Guardrails provides a structured way to intercept model input and output to enforce safety boundaries. By defining specific 'rails' that detect non-compliant topics, the system can pivot the conversation or block the response entirely. This mechanism is critical in high-stakes environments where adherence to strict safety protocols is mandatory to prevent liability and ensure that generative AI tools do not cross into domains requiring human expertise.

Answer

Increase the temperature parameter of the LLM to provide more creative, diverse outputs.

Answer

Retrain the entire foundation model on a curated dataset of medical textbooks.

Answer

Change the model architecture to a smaller parameter size to reduce knowledge density.

Question 10

A team is pretraining a 13B-parameter decoder-only LLM on English text using byte-pair encoding with a 50,000-token vocabulary. They observe that the model produces fluent but repetitive continuations and that the average log-probability assigned to ground-truth tokens plateaus early. The training loss curve shows the model is underfitting rather than overfitting. Which architectural change is most likely to improve the model's capacity to capture long-range dependencies?

Accepted Answer

Increase the number of transformer layers and the hidden dimension proportionally.. Underfitting in a large decoder-only LLM indicates the model lacks sufficient capacity for the data distribution. Scaling both depth and width increases the parameter count and the number of nonlinear transformations applied to the residual stream, which improves the model's ability to capture long-range dependencies. The other options either rebalance existing capacity, change positional encoding without adding parameters, or reduce computation without improving fit.

Answer

Reduce the vocabulary size to 32,000 tokens to lower the softmax computation cost.

Answer

Increase the number of attention heads while keeping the hidden dimension fixed.

Answer

Replace the learned positional embeddings with sinusoidal absolute positional encodings.