NCP-GENL Practice Test 3 — 30 Questions

Question 1

When assessing the quality of a dataset for instruction fine-tuning, which TWO metrics or methods are considered most reliable for measuring dataset diversity?

Accepted Answer

Embedding-based clustering to visualize topical coverage across the corpus.. Dataset diversity ensures that the model encounters a broad range of topics and linguistic patterns, which is critical for generalization. Using embedding-based clustering allows for the identification of thematic coverage, while perplexity distribution analysis helps assess whether the dataset contains a balance of common and complex structures. These methods together provide a quantitative view of the data's breadth, reducing the risk of bias or overfitting.

Answer

Calculating the total number of words in the dataset.

Answer

Measuring the average response length for every instruction.

Answer

Checking if the dataset is solely in ASCII format.

Question 2

Why is it important to perform 'domain-specific' data cleaning when preparing a corpus for fine-tuning a medical LLM?

Accepted Answer

Generic cleaning often removes or modifies critical domain-specific nomenclature.. Medical text contains specific jargon, abbreviations, and relationships that generic cleaning might misinterpret. Generic tools often remove or alter terms that are critical to medical context, such as drug names or procedural codes. By using domain-specific cleaning, you ensure that the model retains the precise terminology necessary for high-stakes, accurate clinical reasoning in medical applications.

Answer

It helps to anonymize the patient data by replacing all medical terms with generic labels.

Answer

It ensures that the dataset size is significantly reduced to fit in GPU cache.

Answer

It automatically corrects all scientific inaccuracies present in the original documents.

Question 3

Which metric is most critical to monitor for identifying 'bottlenecks' in a high-throughput LLM deployment?

Accepted Answer

GPU Utilization percentage.. GPU utilization is the primary metric for understanding how well the model is saturating the compute resources. If GPU utilization is low while latency is high, it suggests a bottleneck elsewhere, such as CPU preprocessing, data transfer, or synchronization issues. Monitoring this metric allows engineers to determine if they need to increase batch sizes, optimize the pipeline, or scale the infrastructure to maintain performance.

Answer

Total number of files in the model repository.

Answer

The name of the backend framework used for training.

Answer

The number of times the server was restarted.

Question 4

An LLM application deployed on NVIDIA Triton Inference Server is experiencing intermittent latency spikes. Which metric is most critical to monitor to determine if the issue stems from GPU compute saturation?

Accepted Answer

SM (Streaming Multiprocessor) occupancy. Monitoring GPU utilization alone is insufficient because it does not distinguish between compute saturation and memory bandwidth bottlenecks. NVML-based metrics specifically tracking SM (Streaming Multiprocessor) occupancy provide the granular insight required to identify whether the execution cores are fully utilized. This metric is essential for capacity planning and ensuring that the inference throughput meets strict service level agreements under high concurrent request loads in production environments.

Answer

Total system CPU usage

Answer

Network interface throughput

Answer

Total GPU memory capacity

Question 5

A team is testing a new LLM application. During red-teaming, the model consistently leaks sensitive internal project codenames. How should the team address this systematically?

Accepted Answer

Deploy a guardrail that filters output against a list of sensitive terms.. Systematic mitigation of data leakage requires a multi-layered approach. Modifying the base model is rarely sufficient; instead, one must implement output-side guardrails that perform pattern matching and dictionary-based filtering. This ensures that even if the model attempts to generate sensitive info, the guardrail intercepts it. This process protects intellectual property and maintains compliance with corporate confidentiality agreements by ensuring that protected information remains strictly within the secure environment.

Answer

Increase the number of training epochs on the existing dataset.

Answer

Add a disclaimer at the end of every response stating that the content is confidential.

Answer

Randomize the model's weights during every inference run.

Question 6

An enterprise fine-tunes a Llama-3-70B model using NVIDIA NeMo for automated technical support ticketing. The development team needs an automated evaluation pipeline that measures semantic similarity against human-curated reference answers without relying on costly human annotators. Which metric provides the most robust embedding-based semantic similarity assessment for this scenario?

Accepted Answer

BERTScore computes token similarity matrices using contextual embeddings to capture deep semantic meaning regardless of surface-level phrasing variations.. BERTScore leverages contextual embeddings from transformer models to evaluate token-level semantic overlap rather than exact string matching, making it ideal for technical support text where phrasing varies. This automated metric correlates strongly with human judgment, significantly accelerating iteration cycles during enterprise model development workflows on NVIDIA infrastructure.

Answer

ROUGE-1 measures unigram overlap but fails to capture semantic synonyms and contextual nuances common in enterprise technical support documentation.

Answer

BLEU evaluates n-gram precision with a brevity penalty, heavily penalizing valid creative paraphrasing typically found in conversational AI support responses.

Answer

Perplexity measures how well a probability distribution predicts a sample, reflecting language fluency rather than semantic alignment with a specific reference answer.

Question 7

Which TWO of the following practices are recommended for ensuring ethical AI development when using NVIDIA NIMs in an enterprise environment?

Accepted Answer

Perform periodic bias audits on model responses using diverse, representative evaluation datasets.. Ensuring ethical AI involves both technical validation and process transparency. Regular bias auditing helps detect unintended stereotyping, while implementing robust logging ensures accountability for every generated output. These practices allow organizations to monitor for discriminatory patterns and maintain an audit trail for compliance, which is essential for building trust with users and adhering to global ethical standards for AI deployment in sensitive business sectors.

Answer

Bypass local logging to optimize inference latency for high-throughput applications.

Answer

Use the model in a closed-loop system where users cannot report offensive content.

Answer

Exclude all metadata from model outputs to prevent potential privacy leaks.

Question 8

An engineer is designing a Transformer-based model for long-context document summarization. They decide to replace the standard dense self-attention mechanism with a sliding window attention approach. What is the primary architectural implication of this change?

Accepted Answer

The computational complexity of the self-attention mechanism is reduced from quadratic to linear.. Sliding window attention restricts the receptive field of each token to a local neighborhood, drastically reducing the quadratic memory complexity of standard attention to linear. This is critical for scaling LLMs to long contexts, as it prevents the O(n²) memory growth that typically causes GPU out-of-memory errors on large input sequences while maintaining local coherence.

Answer

The model's total parameter count increases due to added positional embedding layers.

Answer

The model loses the ability to perform parallel training across multiple GPU nodes.

Answer

The model is no longer compatible with standard softmax normalization functions.

Question 9

An enterprise deployment of NeMo Guardrails is experiencing hallucinations where the model provides medical advice despite strict system prompts. What is the most effective approach to mitigate this risk?

Accepted Answer

Implement NeMo Guardrails 'dialogue rails' to detect and redirect queries related to medical diagnosis.. NeMo Guardrails provides a structured way to intercept model input and output to enforce safety boundaries. By defining specific 'rails' that detect non-compliant topics, the system can pivot the conversation or block the response entirely. This mechanism is critical in high-stakes environments where adherence to strict safety protocols is mandatory to prevent liability and ensure that generative AI tools do not cross into domains requiring human expertise.

Answer

Increase the temperature parameter of the LLM to provide more creative, diverse outputs.

Answer

Retrain the entire foundation model on a curated dataset of medical textbooks.

Answer

Change the model architecture to a smaller parameter size to reduce knowledge density.

Question 10

A team is pretraining a 13B-parameter decoder-only LLM on English text using byte-pair encoding with a 50,000-token vocabulary. They observe that the model produces fluent but repetitive continuations and that the average log-probability assigned to ground-truth tokens plateaus early. The training loss curve shows the model is underfitting rather than overfitting. Which architectural change is most likely to improve the model's capacity to capture long-range dependencies?

Accepted Answer

Increase the number of transformer layers and the hidden dimension proportionally.. Underfitting in a large decoder-only LLM indicates the model lacks sufficient capacity for the data distribution. Scaling both depth and width increases the parameter count and the number of nonlinear transformations applied to the residual stream, which improves the model's ability to capture long-range dependencies. The other options either rebalance existing capacity, change positional encoding without adding parameters, or reduce computation without improving fit.

Answer

Reduce the vocabulary size to 32,000 tokens to lower the softmax computation cost.

Answer

Increase the number of attention heads while keeping the hidden dimension fixed.

Answer

Replace the learned positional embeddings with sinusoidal absolute positional encodings.

Question 11

After fine-tuning a code-generation model with NVIDIA NeMo, an engineer notices the model now produces correct domain-specific function calls but has started emitting malformed JSON in about 15 percent of structured-output requests. The fine-tuning dataset contained no structured-output examples. Which evaluation action best explains and catches this regression?

Accepted Answer

Add a format-compliance check plus a general-capability regression suite to the evaluation, and compare against the base model on the same structured-output prompts.. Fine-tuning on a narrow dataset can degrade capabilities absent from that data, and structured output is a classic casualty. The right response is to measure it: add format-compliance scoring to the harness, rerun the same structured prompts against the base and tuned models, and include a general regression suite. Retraining harder, raising temperature, or dismissing the failures all skip the measurement step the situation demands.

Answer

Increase the temperature of the structured-output requests so the model has more freedom to produce valid JSON.

Answer

Conclude that the fine-tune succeeded on its target task and that the JSON failures are acceptable collateral given the domain gains.

Answer

Retrain the model with a larger learning rate so the structured-output behavior is reinforced more strongly during fine-tuning.

Question 12

A team is using an NVIDIA NIM-hosted Llama model to generate product descriptions from a list of technical specifications. The descriptions sometimes omit key specifications or include invented features. The team wants to improve reliability without changing the model. Which prompt engineering change is most likely to reduce these errors?

Accepted Answer

Add a system prompt that instructs the model to act as a technical writer and to include every specification exactly as given, and provide a few-shot example of a correct description.. To reduce omissions and inventions without changing the model, the most effective prompt engineering approach is to provide explicit instructions and few-shot examples. A system prompt that defines the role and constraints, combined with an example that demonstrates including all specifications and avoiding fabrication, guides the model to produce more reliable outputs. Other parameter changes or chain-of-thought do not directly address content fidelity.

Answer

Increase the max_tokens parameter to allow the model to generate longer descriptions, ensuring all specifications are covered.

Answer

Use a chain-of-thought prompt that asks the model to list each specification and then write the description.

Answer

Set the temperature to 0.0 to make the model deterministic and reduce creativity.

Question 13

You are preparing a large instruction-tuning dataset for an NVIDIA NeMo-based LLM. The raw data consists of user queries and assistant responses collected from a customer support system, stored as JSON lines. During preprocessing, you notice that many responses contain personally identifiable information (PII) such as names, email addresses, and phone numbers. You need to ensure the dataset is safe for training while preserving as much semantic content as possible. Which approach is most appropriate for handling PII in this dataset?

Accepted Answer

Use a named entity recognition (NER) model to detect PII and replace each entity with a generic placeholder (e.g., [NAME]).. Replacing PII with generic placeholders using an NER model is the most balanced approach: it removes sensitive information while preserving the linguistic patterns and context needed for instruction tuning. This method aligns with NVIDIA NeMo's data preprocessing recommendations for responsible AI. It also avoids the pitfalls of discarding data or introducing unintelligible tokens.

Answer

Leave the PII intact but exclude the dataset from production use and train only in a sandbox environment.

Answer

Hash each detected PII string with SHA-256 and substitute the hash into the text.

Answer

Apply a rule-based regex filter to remove any line containing PII, then train on the remaining data.

Question 14

Refer to the exhibit. An engineer receives this timeout error during a CUDA kernel execution. What is the most appropriate first step to diagnose the resource contention?

Accepted Answer

Use NVIDIA Nsight Compute to profile occupancy.. CUDA kernel timeouts are typically caused by long-running kernels that exceed the GPU's watchdog timer or by excessive resource usage (registers/shared memory) that limits occupancy. Using the NVIDIA Nsight Compute profiler allows the engineer to see the exact occupancy metrics and register pressure, enabling targeted optimizations. This step is crucial for identifying if the kernel is over-provisioned for the specific GPU architecture being targeted.

Answer

Increase the block size to 2048 threads.

Answer

Disable the watchdog timer in the OS.

Answer

Switch the kernel to run on the CPU.

Question 15

Refer to the exhibit. The deployment is facing memory allocation errors during peak load. Based on the error log, what is the most effective configuration change to resolve the issue while keeping the model architecture constant?

Accepted Answer

Enable PagedAttention in the TensorRT-LLM runtime.. The error log indicates that the system is running out of memory because the KV cache is allocated statically, which often leads to fragmentation or over-allocation. Enabling PagedAttention is the standard NVIDIA-recommended solution for this scenario. It allows the system to manage KV cache memory dynamically, reclaiming space from finished requests and efficiently allocating blocks for new tokens, thereby preventing the out-of-memory (OOM) errors caused by static allocation.

Answer

Switch to a smaller model size.

Answer

Increase the GPU clock frequency.

Answer

Reduce the batch size to one.

Question 16

Refer to the exhibit. An engineer observes that GPU memory utilization is high, but the GPU is frequently idling. How does the provided Triton configuration optimize the inference pipeline?

Accepted Answer

It enables concurrent execution on the GPU. By setting the instance count to 2 on a single GPU, the configuration enables concurrent model execution. This allows Triton to schedule multiple inference requests to be processed simultaneously on the same hardware. This overlap helps to hide memory latency and pipeline stalls, effectively increasing the utilization of the GPU and raising the overall throughput of the deployment.

Answer

It forces the model to use INT8 precision

Answer

It partitions the GPU for dedicated memory

Answer

It enables dynamic batching for the requests

Question 17

When using the NVIDIA Collective Communications Library (NCCL), what does 'AllReduce' specifically optimize for in a multi-GPU training configuration?

Accepted Answer

Gradient aggregation across all devices.. AllReduce is critical for distributed training because it synchronizes the gradient updates from all GPUs across the cluster. It aggregates data from all devices and distributes the result back to each one, allowing models to train concurrently on massive datasets. By optimizing this collective operation, NCCL minimizes the time GPUs spend waiting for synchronization, which is the primary hurdle in scaling training to hundreds or thousands of GPUs.

Answer

Local GPU thread synchronization.

Answer

Direct CPU-to-GPU data copy.

Answer

Increasing GPU clock frequency.

Question 18

Refer to the exhibit. What is the effective batch size for this fine-tuning job?

Accepted Answer

16. The effective batch size is calculated by multiplying the per-device batch size by the gradient accumulation steps. In this case, 1 * 16 equals 16. Gradient accumulation allows the model to simulate a larger batch size by running multiple forward and backward passes before performing a single optimizer step, which helps in achieving more stable gradient updates without increasing the immediate memory overhead for activations.

Answer

1

Answer

15

Answer

17

Question 19

Which metric is the most reliable indicator that an LLM is overfitting during the fine-tuning phase?

Accepted Answer

A decrease in training loss with a simultaneous increase in validation loss.. An increasing validation loss alongside a decreasing training loss is the classic indicator of overfitting. The model is essentially memorizing the training samples rather than learning generalized language patterns. Monitoring this divergence is critical in NVIDIA-based training pipelines, as it allows researchers to implement early stopping or adjust regularization strategies before the model's performance on unseen tasks degrades significantly.

Answer

A consistent increase in both training and validation loss.

Answer

A plateau in the training loss while the validation loss remains stagnant.

Answer

The model generates text that matches the training set exactly.

Question 20

You are preparing a massive dataset for training a NeMo-based LLM. Which TWO data preprocessing steps are critical to prevent data leakage and ensure model quality?

Accepted Answer

Performing temporal or content-based splitting to isolate the validation set.. Data leakage occurs when test data is inadvertently included in the training set, leading to inflated performance metrics. Deduplication is equally vital, as repetitive data causes the model to memorize samples rather than generalize. In NVIDIA workflows, these steps are typically performed via distributed scripts on the cluster before tokenization, ensuring that the model learns unique, non-overlapping information across all shards.

Answer

Increasing the frequency of data augmentation in the training pipeline.

Answer

Adding metadata tags to all training sequences for classification.

Answer

Converting all text to lowercase to increase vocabulary efficiency.