NCP-GENL Practice Test 18 — 15 Questions

Question 1

You are using NVIDIA NeMo Evaluation to assess a summarization model. The model produces summaries that are grammatically correct but omit key information from the source. Which metric should you use to quantify the amount of missing content?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

ROUGE-1 recall. The problem is omission of key information from summaries. ROUGE-1 recall directly measures how much of the reference unigrams are present in the generated summary. Low recall indicates missing content. Precision, BLEU, and perplexity focus on relevance, precision, or fluency, not on capturing all reference content, making recall the correct choice.

Answer

Perplexity

Answer

BLEU score

Answer

ROUGE-1 precision

Question 2

An enterprise team is preparing a supervised fine-tuning job in NVIDIA NeMo for a 20B LLM. They want to reduce GPU memory consumption during training without changing the model architecture or the dataset. Which two configuration changes should they apply? (Choose two.)

Accepted Answer

Enable gradient checkpointing to recompute activations during the backward pass.. Gradient checkpointing and bfloat16 mixed precision both reduce memory during training without modifying model architecture or dataset. Checkpointing lowers stored activation memory by recomputation, while bfloat16 halves tensor memory and is supported natively in NeMo. Architectural changes, data expansion, and sequence truncation either violate the constraints or fail to address memory consumption per training step.

Answer

Expand the training dataset with additional synthetic examples.

Answer

Reduce the maximum sequence length by truncating all samples to 64 tokens.

Answer

Increase the number of attention heads in each transformer block.

Question 3

A team is preparing a customer-support chat dataset for instruction fine-tuning of an LLM. The raw data contains HTML tags, inconsistent date formats, and emoji. Which data preparation step should be performed first to make the text usable for tokenization?

Accepted Answer

Normalize and clean the text. Cleaning and normalizing the raw chat data first removes HTML, standardizes dates, and handles emoji, producing consistent text for tokenization. This order prevents noisy artifacts from becoming tokens and ensures both training and validation splits receive the same treatment. It is the foundational step before tokenization and dataset splitting.

Answer

Tokenize the text with the model's tokenizer

Answer

Convert the text to lowercase

Answer

Split the dataset into train and validation sets

Question 4

You are curating instruction-tuning data for an NVIDIA NIM-deployed LLM. The raw dataset contains many near-duplicate instruction-response pairs that differ only by punctuation and whitespace. Which data preparation step is most appropriate to remove these before fine-tuning?

Accepted Answer

Apply fuzzy deduplication using MinHash LSH over normalized instruction-response text.. Fuzzy deduplication with MinHash LSH is the standard approach for near-duplicate removal in instruction-tuning datasets. It normalizes text for comparison, generates MinHash signatures, and uses locality-sensitive hashing to find similar pairs efficiently at scale. This directly removes the redundant examples described, improving training efficiency and reducing overfitting without altering the semantic content of the retained data.

Answer

Increase the batch size during fine-tuning to average out duplicate examples.

Answer

Convert all instructions to lowercase and remove all punctuation globally.

Answer

Use NVIDIA Triton Inference Server dynamic batching to filter duplicates at serving time.

Question 5

You are preparing a multilingual corpus for pretraining with NVIDIA NeMo. The dataset contains documents in 40 languages, but the tokenizer was trained primarily on English. Which data preparation action best ensures that non-English text is represented efficiently during tokenization?

Accepted Answer

Train a new tokenizer on a balanced sample of all 40 languages using SentencePiece or Hugging Face tokenizers before pretraining.. Training a new tokenizer on a balanced multilingual sample directly solves the vocabulary coverage problem. It ensures that frequent subwords in each language are included in the vocabulary, reducing token fragmentation and improving training efficiency. The other options either mask the symptom, alter the data inappropriately, or do not address tokenizer vocabulary at all.

Answer

Increase the model's maximum sequence length to 8192 tokens to accommodate longer tokenized non-English text.

Answer

Apply Unicode normalization form NFKC to all text and rely on the existing English tokenizer.

Answer

Translate all non-English documents to English using an NVIDIA NIM translation model before tokenization.

Question 6

A team is fine-tuning a 70B-parameter LLM with NVIDIA NeMo on a multi-node cluster and wants to reduce the memory footprint per GPU without changing the model architecture. They are already using mixed precision and a reasonable micro-batch size. Which two techniques should they apply? (Choose two.)

Accepted Answer

Enable ZeRO-style optimizer state partitioning across data-parallel ranks.. Partitioning optimizer states across data-parallel ranks and recomputing activations during the backward pass both reduce per-GPU memory without changing the model architecture. The first shrinks optimizer memory, and the second shrinks activation memory, which together address the dominant contributors for a 70B model. Changing attention heads, switching optimizers, or shrinking the global batch do not achieve the same effect.

Answer

Switch the optimizer to SGD with momentum and remove weight decay.

Answer

Reduce the global batch size by a factor of eight and keep the micro-batch size the same.

Answer

Increase the number of attention heads to spread the computation across more GPUs.

Question 7

A team has fine-tuned a Llama-3-70B model with NVIDIA NeMo and now must decide whether the tuned checkpoint actually improves on the base model for their domain. They run both models over the same 500-prompt held-out set and compute ROUGE-L against reference answers. The fine-tuned model scores 0.41 and the base model scores 0.44. What is the most technically sound conclusion the evaluation lead should draw?

Accepted Answer

ROUGE-L is insensitive to instruction-following style, so a 0.03 gap cannot be used to judge the fine-tune and the team should switch to a pairwise LLM-as-judge with human spot-checking.. ROUGE-L compares token overlap to a single reference, so it cannot distinguish a correct paraphrase from an incorrect answer, and a 0.03 gap is within that measurement noise. Since the team's goal is domain-appropriate behavior rather than lexical imitation, a pairwise preference evaluation with human adjudication gives a defensible signal. Discarding the checkpoint, enlarging the sample, or claiming forgetting all misread what the metric actually measures.

Answer

The base model should be kept because a fine-tune that does not raise ROUGE-L proves catastrophic forgetting of the pre-training corpus.

Answer

The fine-tuned model is worse and should be discarded, since a lower ROUGE-L on 500 held-out prompts is a statistically significant regression.

Answer

The evaluation is valid but the held-out set is too small; expanding it to 5,000 prompts would make ROUGE-L reliable enough to select the better checkpoint.

Question 8

You are preparing a dataset for pretraining an LLM using NVIDIA NeMo's Megatron-LM. The dataset consists of JSONL files where each line contains a 'text' field. You need to convert these files into the binary format required by Megatron for efficient training. Which tool or method should you use to perform this conversion?

Accepted Answer

Use the `preprocess_data.py` script from the Megatron repository to tokenize and create indexed binary files.. Megatron-LM provides the `preprocess_data.py` script specifically for converting JSONL text data into indexed binary files that its data loader can read efficiently. This script handles tokenization, document concatenation, and sharding, ensuring compatibility with Megatron's training pipeline. It is the recommended and standard method for preparing data for Megatron-based pretraining.

Answer

Use Apache Arrow to serialize the JSONL data into a columnar format and load it with a custom data loader.

Answer

Use the `jsonl_to_bin` utility provided by the Hugging Face Transformers library.

Answer

Use the `nemo_curator` library to convert JSONL to TFRecord format, then train directly on TFRecords.

Question 9

You are curating a 200 GB instruction-tuning corpus for an NVIDIA NeMo fine-tuning job on a Llama-based model. Post-training evaluation reveals the model regurgitates exact validation-set passages verbatim. An audit shows that near-duplicate instruction/response pairs were split randomly at the record level across train and validation partitions. Which data preparation change most directly eliminates this leakage while preserving the maximum amount of usable training data?

Accepted Answer

Perform MinHash-based near-duplicate detection across the full corpus, cluster records by similarity, and assign whole clusters to either the train or validation split using a deterministic hash of the cluster ID.. The leakage arises because near-duplicate instruction/response pairs were split at the record level, allowing paraphrased twins to appear in both partitions. Clustering near-duplicates and assigning entire clusters to one split closes that pathway while retaining all unique content for training. Deterministic cluster-to-split hashing also makes the partition reproducible, which matters for auditing and for comparing fine-tuning runs fairly.

Answer

Increase the validation split ratio from 5% to 20% so that fewer training records overlap with the validation partition.

Answer

Apply aggressive token-level cleaning to strip boilerplate phrases and punctuation from every record before splitting the dataset randomly.

Answer

Deduplicate only the validation partition by removing any record whose exact text appears in the training partition, then keep the original random split.

Question 10

You are preparing a dataset for instruction fine-tuning an LLM using NVIDIA NeMo. The dataset contains pairs of instructions and responses, but you notice that some responses are significantly longer than others, and a few are extremely short (e.g., 'Yes' or 'No'). You want to ensure the model learns to generate appropriate-length responses. Which data preparation technique is most effective?

Accepted Answer

Ensure the dataset has a balanced distribution of response lengths by sampling or resampling.. A balanced distribution of response lengths allows the model to learn when to generate concise versus detailed answers. Overrepresentation of short responses may lead to truncation, while too many long responses can cause verbosity. Resampling to achieve a more uniform distribution helps the model internalize the relationship between instruction complexity and response length.

Answer

Augment the dataset by duplicating short responses to increase their frequency.

Answer

Filter out all responses shorter than 10 tokens to remove trivial answers.

Answer

Include a length token or metadata in the input to indicate the desired response length.

Question 11

A developer is building a decoder-only generative model and wants to prevent the model from attending to future tokens during training so that each position can only use information from itself and earlier positions. Which architectural mechanism should they implement in the self-attention layer?

Accepted Answer

A causal attention mask that sets attention scores for future positions to negative infinity before the softmax.. Decoder-only generative models must not see future tokens during training, otherwise next-token prediction becomes trivial and the model will not generalize to autoregressive inference. A causal attention mask accomplishes this by making future positions invisible to the softmax, ensuring each position only conditions on itself and earlier tokens.

Answer

A padding mask that ignores tokens added to equalize sequence lengths across a batch.

Answer

A learned positional embedding added to the token embeddings at the input layer.

Answer

A residual connection that adds the attention output back to the input hidden state.

Question 12

A developer is preparing to fine-tune a model with NVIDIA NeMo and wants a quantitative baseline before training begins. They plan to score the base model on a 300-question multiple-choice reasoning set and report accuracy. Which evaluation setup gives the most defensible baseline number?

Accepted Answer

Score the base model on the same held-out set with a fixed prompt template and deterministic decoding, then record the exact configuration alongside the accuracy.. A trustworthy baseline requires a held-out set, a fixed prompt template, deterministic decoding, and a recorded configuration. Those choices make the number reproducible and ensure later deltas reflect training rather than evaluation noise. Best-of-runs reporting, training-set scoring, and self-reported confidence all produce numbers that either inflate performance or cannot be compared to ground-truth labels.

Answer

Run the base model repeatedly at high temperature and report the highest accuracy observed across runs as the baseline.

Answer

Use the training split as the baseline evaluation set so the number reflects how well the model already covers the fine-tuning material.

Answer

Ask the model to self-report a confidence percentage for each question and average those values as the baseline accuracy.

Question 13

You are preparing a dataset of support tickets for a RAG system using NVIDIA NeMo. Many tickets are short and contain little context, which hurts retrieval quality. Which data preparation technique best improves retrieval by enriching each ticket with related information before embedding?

Accepted Answer

Use an LLM to generate contextual summaries or metadata for each ticket and prepend them to the text before embedding.. Short tickets embed poorly because they carry little semantic signal. Using an LLM to generate contextual summaries or metadata and prepending that text before embedding enriches each record, so the vector captures more of the ticket's intent and domain. This improves matching with user queries. Deduplication, larger embeddings, and lower thresholds do not add missing context, so they cannot resolve the sparsity problem in this scenario.

Answer

Increase the embedding dimension of the retrieval model.

Answer

Lower the similarity threshold used during retrieval.

Answer

Apply deduplication to remove similar tickets from the corpus.

Question 14

An LLM inference service on NVIDIA Triton Inference Server uses TensorRT-LLM as the backend. During peak load, you notice that the first token latency (time to first token) is high, but subsequent tokens are generated quickly. Which metric should you monitor to diagnose this issue?

Accepted Answer

Time to first token (TTFT). Time to first token (TTFT) is the key metric for diagnosing prefill latency in LLM inference. Since the issue is high TTFT but fast subsequent tokens, monitoring TTFT allows you to isolate the prefill phase and optimize it, for example by reducing batch size or using more efficient attention kernels.

Answer

GPU utilization

Answer

Requests per second (RPS)

Answer

Inter-token latency (ITL)

Question 15

A financial institution is evaluating a fine-tuned GPT-3 model for generating investment advice summaries. They must ensure the model does not produce harmful or biased recommendations. Which evaluation methodology should they implement using NVIDIA NeMo Guardrails and NeMo Evaluator to systematically detect and quantify such issues?

Accepted Answer

Use NeMo Guardrails to define a set of safety policies, then run the model's outputs through the guardrails and measure the percentage of violations flagged.. To systematically detect and quantify harmful or biased outputs, the team should leverage NeMo Guardrails to encode safety policies and measure violation rates. This provides a direct, quantifiable assessment of safety compliance. Other metrics like perplexity, BLEU, or F1 do not target harmful content and therefore cannot fulfill the evaluation requirement.

Answer

Use NeMo Evaluator to compute the F1 score of the model's outputs against a set of correct summaries.

Answer

Compute the model's perplexity on a validation set of financial texts.

Answer

Fine-tune the model further on a dataset of unbiased financial advice and then evaluate with BLEU score.