NCP-GENL Data Preparation Practice Test 3 — 15 Questions

Question 1

You are curating instruction-tuning data for an NVIDIA NIM-deployed LLM. The raw dataset contains many near-duplicate instruction-response pairs that differ only by punctuation and whitespace. Which data preparation step is most appropriate to remove these before fine-tuning?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Apply fuzzy deduplication using MinHash LSH over normalized instruction-response text.. Fuzzy deduplication with MinHash LSH is the standard approach for near-duplicate removal in instruction-tuning datasets. It normalizes text for comparison, generates MinHash signatures, and uses locality-sensitive hashing to find similar pairs efficiently at scale. This directly removes the redundant examples described, improving training efficiency and reducing overfitting without altering the semantic content of the retained data.

Answer

Increase the batch size during fine-tuning to average out duplicate examples.

Answer

Convert all instructions to lowercase and remove all punctuation globally.

Answer

Use NVIDIA Triton Inference Server dynamic batching to filter duplicates at serving time.

Question 2

You are preparing a dataset for pretraining an LLM using NVIDIA NeMo's Megatron-LM. The dataset consists of JSONL files where each line contains a 'text' field. You need to convert these files into the binary format required by Megatron for efficient training. Which tool or method should you use to perform this conversion?

Accepted Answer

Use the `preprocess_data.py` script from the Megatron repository to tokenize and create indexed binary files.. Megatron-LM provides the `preprocess_data.py` script specifically for converting JSONL text data into indexed binary files that its data loader can read efficiently. This script handles tokenization, document concatenation, and sharding, ensuring compatibility with Megatron's training pipeline. It is the recommended and standard method for preparing data for Megatron-based pretraining.

Answer

Use Apache Arrow to serialize the JSONL data into a columnar format and load it with a custom data loader.

Answer

Use the `jsonl_to_bin` utility provided by the Hugging Face Transformers library.

Answer

Use the `nemo_curator` library to convert JSONL to TFRecord format, then train directly on TFRecords.

Question 3

You are preparing a large instruction-tuning dataset for an LLM using NVIDIA NeMo. You need to ensure the dataset supports efficient training and evaluation. Which TWO steps are essential in the data preparation pipeline? (Choose two.)

Accepted Answer

Convert the dataset into a format compatible with NeMo, such as JSONL with input and output fields. Formatting the dataset into a NeMo-compatible structure such as JSONL with input and output fields ensures the data loader can consume it, while splitting into training, validation, and test sets enables proper model evaluation and hyperparameter tuning. These two steps are foundational; other activities like tokenizer training or augmentation are optional and context-dependent.

Answer

Train a tokenizer from scratch on the dataset

Answer

Apply data augmentation using back-translation

Answer

Remove all punctuation from the text

Question 4

You are preparing a multilingual corpus for pretraining with NVIDIA NeMo. The dataset contains documents in 40 languages, but the tokenizer was trained primarily on English. Which data preparation action best ensures that non-English text is represented efficiently during tokenization?

Accepted Answer

Train a new tokenizer on a balanced sample of all 40 languages using SentencePiece or Hugging Face tokenizers before pretraining.. Training a new tokenizer on a balanced multilingual sample directly solves the vocabulary coverage problem. It ensures that frequent subwords in each language are included in the vocabulary, reducing token fragmentation and improving training efficiency. The other options either mask the symptom, alter the data inappropriately, or do not address tokenizer vocabulary at all.

Answer

Increase the model's maximum sequence length to 8192 tokens to accommodate longer tokenized non-English text.

Answer

Apply Unicode normalization form NFKC to all text and rely on the existing English tokenizer.

Answer

Translate all non-English documents to English using an NVIDIA NIM translation model before tokenization.

Question 5

You are preparing a dataset for instruction fine-tuning an LLM using NVIDIA NeMo. The dataset contains pairs of instructions and responses, but you notice that some responses are significantly longer than others, and a few are extremely short (e.g., 'Yes' or 'No'). You want to ensure the model learns to generate appropriate-length responses. Which data preparation technique is most effective?

Accepted Answer

Ensure the dataset has a balanced distribution of response lengths by sampling or resampling.. A balanced distribution of response lengths allows the model to learn when to generate concise versus detailed answers. Overrepresentation of short responses may lead to truncation, while too many long responses can cause verbosity. Resampling to achieve a more uniform distribution helps the model internalize the relationship between instruction complexity and response length.

Answer

Augment the dataset by duplicating short responses to increase their frequency.

Answer

Filter out all responses shorter than 10 tokens to remove trivial answers.

Answer

Include a length token or metadata in the input to indicate the desired response length.

Question 6

You are preparing a multilingual corpus for pre-training an LLM. The corpus contains documents in English, Spanish, and German, but the English portion is 80% of the data. You want to avoid the model becoming biased toward English. Which data preparation technique should you apply?

Accepted Answer

Oversample the non-English documents or undersample the English documents. Balancing the language distribution by oversampling minority languages or undersampling the dominant one directly addresses the 80% English skew. This ensures the model sees a more representative mix during training, reducing bias toward English and improving performance on Spanish and German. It is a standard data preparation technique for multilingual corpora.

Answer

Remove all English documents to force multilingual learning

Answer

Translate all documents into English

Answer

Apply a language-specific tokenizer for each language

Question 7

You are curating a 200 GB instruction-tuning corpus for an NVIDIA NeMo fine-tuning job on a Llama-based model. Post-training evaluation reveals the model regurgitates exact validation-set passages verbatim. An audit shows that near-duplicate instruction/response pairs were split randomly at the record level across train and validation partitions. Which data preparation change most directly eliminates this leakage while preserving the maximum amount of usable training data?

Accepted Answer

Perform MinHash-based near-duplicate detection across the full corpus, cluster records by similarity, and assign whole clusters to either the train or validation split using a deterministic hash of the cluster ID.. The leakage arises because near-duplicate instruction/response pairs were split at the record level, allowing paraphrased twins to appear in both partitions. Clustering near-duplicates and assigning entire clusters to one split closes that pathway while retaining all unique content for training. Deterministic cluster-to-split hashing also makes the partition reproducible, which matters for auditing and for comparing fine-tuning runs fairly.

Answer

Increase the validation split ratio from 5% to 20% so that fewer training records overlap with the validation partition.

Answer

Apply aggressive token-level cleaning to strip boilerplate phrases and punctuation from every record before splitting the dataset randomly.

Answer

Deduplicate only the validation partition by removing any record whose exact text appears in the training partition, then keep the original random split.

Question 8

You are building a pretraining dataset from a large collection of source-code repositories for an NVIDIA NeMo LLM. The data includes many files with licenses, generated code, and minified JavaScript. Which NeMo Curator-based approach best improves code data quality before tokenization?

Accepted Answer

Use language-specific filters that detect and remove minified files, license headers, and auto-generated code, then apply deduplication.. Language-specific filters combined with deduplication directly target the described quality issues. Minified files, license headers, and generated code are common in code corpora and can be detected with heuristics such as average line length, ratio of whitespace, and presence of standard license text. Deduplication removes repeated generated files. This preprocessing ensures the pretraining corpus contains meaningful code, which improves the model's code capabilities.

Answer

Convert all code to a single programming language using an automated transpiler before tokenization.

Answer

Increase the model's context window to 32k tokens so that entire repositories fit in one sequence.

Answer

Tokenize all files with a byte-level BPE tokenizer and skip any further filtering.

Question 9

You are preparing a customer-support dataset for fine-tuning an LLM with NVIDIA NeMo. The raw data includes personally identifiable information such as names, email addresses, and phone numbers. Which data preparation step must be performed before training to comply with privacy requirements?

Accepted Answer

Apply PII detection and redaction to replace sensitive entities with placeholders.. Privacy compliance requires removing or masking personally identifiable information before training. PII detection and redaction replaces sensitive entities with placeholders, preserving the dataset's instructional value while preventing the model from memorizing real customer details. Tokenization, batch size, and shuffling do not alter the presence of PII and therefore cannot satisfy the requirement to protect sensitive data during fine-tuning.

Answer

Shuffle the dataset to break associations between PII and responses.

Answer

Increase the batch size during fine-tuning to average out PII exposure.

Answer

Tokenize the dataset with a custom vocabulary that includes PII patterns.

Question 10

A team is preparing a mixed-language corpus for continued pretraining of an NVIDIA NeMo Megatron model. The corpus contains English, Japanese, and Arabic documents. Tokenizer analysis shows the current English-centric BPE vocabulary produces very long token sequences for Japanese and Arabic, inflating sequence length and compute cost. The team wants to reduce sequence length for non-English text without retraining the tokenizer from scratch and without degrading English performance. Which data preparation action best achieves this?

Accepted Answer

Extend the existing BPE vocabulary with additional merges learned from a balanced multilingual sample, then retrain only the embedding and output layers while freezing the rest of the model.. The high token counts for Japanese and Arabic stem from a vocabulary that lacks subword units for those scripts. Extending the BPE vocabulary with multilingual merges shortens their token sequences while preserving English merges, and retraining only the embedding and output layers adapts the new entries without perturbing the pretrained transformer. This lowers compute cost and keeps English behavior stable.

Answer

Lowercase and strip diacritics from all Japanese and Arabic text so that the English-centric vocabulary can represent it more efficiently.

Answer

Transliterate all Japanese and Arabic documents into Latin script using a standard romanization scheme before tokenization.

Answer

Increase the model's maximum sequence length and positional embedding size so that long token sequences for Japanese and Arabic fit without truncation.

Question 11

You are preparing a dataset for supervised fine-tuning (SFT) of an NVIDIA NeMo LLM to follow instructions. Which TWO data preparation practices are essential to ensure the model learns to generalize rather than memorize? (Choose two.)

Accepted Answer

Include a diverse set of instruction types and domains in the training data rather than repeating a few templates.. Disjoint data splits prevent leakage and give honest evaluation, while diverse instruction types and domains teach the model to follow instructions generally rather than memorize templates. Together, these practices reduce overfitting and improve generalization. The other options either fail to address memorization, actively encourage it, or unnecessarily restrict the data distribution.

Answer

Increase the learning rate significantly to force the model to escape memorization of individual examples.

Answer

Duplicate the most common instruction-response pairs to reinforce the desired behavior.

Answer

Use only English instructions to simplify tokenization and avoid multilingual complexity.

Question 12

You are preparing a dataset of support tickets for a RAG system using NVIDIA NeMo. Many tickets are short and contain little context, which hurts retrieval quality. Which data preparation technique best improves retrieval by enriching each ticket with related information before embedding?

Accepted Answer

Use an LLM to generate contextual summaries or metadata for each ticket and prepend them to the text before embedding.. Short tickets embed poorly because they carry little semantic signal. Using an LLM to generate contextual summaries or metadata and prepending that text before embedding enriches each record, so the vector captures more of the ticket's intent and domain. This improves matching with user queries. Deduplication, larger embeddings, and lower thresholds do not add missing context, so they cannot resolve the sparsity problem in this scenario.

Answer

Increase the embedding dimension of the retrieval model.

Answer

Lower the similarity threshold used during retrieval.

Answer

Apply deduplication to remove similar tickets from the corpus.

Question 13

You are building the data preparation stage for an NVIDIA NeMo retrieval-augmented generation pipeline that will ingest millions of internal wiki pages. The ingestion team reports that the same policy text appears in dozens of pages with minor edits, and that some pages contain copied tables from external sources. You need to produce a clean, deduplicated chunk store that supports accurate citation and avoids returning redundant passages. Which TWO actions best address these requirements? (Choose two.)

Accepted Answer

Run near-duplicate detection at the chunk level using MinHash or SimHash, and retain one canonical chunk per duplicate group while preserving a mapping from the canonical chunk to all source page identifiers.. Redundant retrieval from repeated policy text is best solved by chunk-level near-duplicate detection that keeps one canonical copy while mapping it to all source pages, and accurate citation requires source metadata to travel with each chunk through the embedding and indexing stages. Together these actions reduce duplicate results and preserve provenance, which fixed-size chunking, page-level embedding, or blanket exclusion of external tables cannot achieve.

Answer

Apply a fixed-size token window with no overlap to every page so that chunk boundaries are uniform across the corpus.

Answer

Tag chunks that contain tables copied from external sources as low-quality and exclude them from the index entirely.

Answer

Embed every page as a single vector and store the full page text as the retrieval unit, avoiding chunking entirely.

Question 14

A data engineer is preparing a JSONL instruction dataset for an NVIDIA NeMo supervised fine-tuning run. Each line currently contains a free-form 'text' field with the instruction, context, and response concatenated. The training configuration expects the standard NeMo instruction-tuning schema with separate fields for the task instruction, optional context, and the expected response. What is the most appropriate data preparation step?

Accepted Answer

Parse each record and rewrite it into the expected fields, such as instruction, input, and output, while preserving the original text content and escaping any embedded quotes.. NeMo's supervised fine-tuning loader expects separate instruction, context, and response fields, so the free-form text must be parsed into that schema with content preserved and quotes escaped. This produces a dataset the standard training configuration can consume directly, keeping the run reproducible and allowing loss to be computed on the response portion alone.

Answer

Keep the free-form text field and modify the NeMo training configuration to treat the entire line as the response.

Answer

Convert the JSONL file to plain text with one example per paragraph and let the tokenizer infer the instruction and response boundaries.

Answer

Duplicate each record and label one copy as instruction and the other as response so the loader sees two fields per example.

Question 15

You are preparing a large corpus of customer support transcripts for continued pretraining of an NVIDIA NeMo Megatron model. The transcripts contain personally identifiable information such as names, email addresses, and account numbers, and company policy requires that this information be removed before training while preserving as much linguistic context as possible for the model to learn from. Which data preparation approach best satisfies both requirements?

Accepted Answer

Detect PII spans with a combination of regular expressions and a named-entity recognition model, then replace each detected span with a consistent placeholder token that preserves the surrounding sentence structure.. Combining regex detection for structured identifiers with named-entity recognition for names and locations covers the PII surface area, and replacing detected spans with consistent placeholders removes sensitive values while preserving the sentence context that continued pretraining depends on. Dropping records, masking all digits and capitals, or hashing every token either destroys usable language data or fails to anonymize reliably.

Answer

Hash every token in the corpus so that the original text cannot be reconstructed, and train on the hashed sequences.

Answer

Replace all digits and capitalized words with a generic mask token across the entire corpus before training.

Answer

Drop every transcript that contains any detected PII so that no sensitive information reaches the training pipeline.