Data Preparation practice questions

Maximum possible volume of raw, uncleaned web-scraped data.

Why it fails: Raw, uncleaned data is often noisy, redundant, and potentially harmful. Including high volumes of such data without filtering introduces biases and lowers the signal-to-noise ratio, which significantly degrades model performance and safety. Quality over quantity is a standard best practice in professional dataset development for LLMs.

Removing all professional terminology to ensure the model remains accessible to general users.

Why it fails: Removing professional terminology reduces the model's utility in specialized domains. Fine-tuned models must understand domain-specific language to be effective assistants. Stripping this data limits the model's ability to perform expert-level tasks, rendering it less useful for the professional use cases it was specifically designed to support.

Including non-English data to maximize the model's multilingual capabilities.

Why it fails: While multilingual capability is beneficial, adding non-English data without a clear strategy or validation can dilute the model's performance in its primary target language. Effective dataset curation requires focused selection criteria rather than simply gathering as much multilingual data as possible without considering the specific project goals.

Implementing a custom GPU-based real-time sentiment analysis layer.

Why it fails: Sentiment analysis is an application-level task, not a data preparation step for pre-training. Pre-training focuses on learning language structure, not on classifying the sentiment of the training data. Including such an unnecessary layer introduces overhead and complexity that is irrelevant to the core objective of language modeling.

Replacing all punctuation marks with unique tokens.

Why it fails: Replacing punctuation with unique tokens unnecessarily expands the vocabulary size without providing significant semantic benefits. Standard tokenization methods handle punctuation effectively without needing such drastic transformations. This approach would likely degrade performance by breaking standard language structures that the model expects to encounter during training.

Encoding all data into 1-bit quantized vectors to save space.

Why it fails: While quantization is a valid technique to reduce memory footprint, 1-bit quantization usually leads to massive loss of semantic information. It is too extreme for most production RAG systems, as it drastically reduces the precision of the vector similarity search, making the retrieval process far less accurate.

Indexing only the top 10% of the most popular documents.

Why it fails: Indexing only a subset of data creates a 'blind spot' in the retrieval system. A RAG system needs access to the full relevant corpus to provide comprehensive and accurate answers. Limiting the index to popular documents reduces the system's ability to answer niche or specific user queries effectively.

Filtering documents that contain any named entity recognized by an NER model.

Why it fails: Named entities are common in high-quality factual text, so removing documents that contain them would discard valuable content and bias the corpus against informative material. NER is not a quality heuristic in NeMo Curator, and this criterion would harm rather than improve the dataset. It does not address the scenario's need to remove low-quality documents.

Filtering documents that contain more than one language.

Why it fails: Multilingual documents are not low quality, and code-switching or quoted foreign phrases appear in legitimate text. NeMo Curator offers language identification to route documents, not to discard them as noise. Removing multilingual content would unnecessarily shrink the corpus and reduce diversity, failing to address the actual goal of eliminating low-quality records.

Filtering documents that exceed a maximum token count of 512 tokens.

Why it fails: Long documents are not inherently low quality; many valuable articles and books exceed 512 tokens. Truncating or removing them based on length would discard useful pretraining signal and distort the corpus distribution. NeMo Curator's quality heuristics target content characteristics, not arbitrary length caps, so this criterion does not fit the scenario's objective.

Converting all text to lowercase to ensure uniformity in vector embeddings.

Why it fails: Lowercase conversion often destroys semantic meaning in technical documentation, such as distinguishing between specific command flags or variable names that are case-sensitive. This approach is detrimental to retrieval precision for technical manuals where specific syntax or product model identifiers rely on exact casing for correct parsing.

Performing aggressive stop-word removal to reduce the dimensionality of the vector space.

Why it fails: Removing stop-words frequently strips away essential linguistic cues that define relationships between technical components. In complex manuals, small words often link functional requirements, and removing them can alter the semantic intent, leading the vector database to retrieve irrelevant context blocks during the retrieval-augmented generation process.

Applying basic sentence tokenization based strictly on periods to create uniform chunks.

Why it fails: Strict period-based tokenization fails to account for technical formatting like bulleted lists, code blocks, or nested sections common in NVIDIA documentation. This simplistic approach often breaks logical units, resulting in fragmented context windows that lack the necessary information for the model to generate accurate, contextually aware responses.

The use of 'nv-embed-v1' will cause OOM errors during the indexing phase.

Why it fails: NVIDIA embedding models are optimized for performance and memory efficiency. Specifying a model name in a configuration file does not inherently lead to out-of-memory errors. Resource allocation is handled by the underlying infrastructure and container configuration, not the choice of the embedding model itself during the indexing stage.

The chunk size of 4096 tokens is too small for modern log analysis.

Why it fails: 4096 tokens is a significant chunk size, generally adequate for most log-based analysis tasks. The issue is not the size itself, but the lack of contextual continuity between these chunks. Log data requires contextual awareness of surrounding events, which is sacrificed by the lack of overlap, not by the size.

The 'regex_mask_all' policy will cause the embedding model to fail during vectorization.

Why it fails: Masking sensitive data via regex is a standard preprocessing step that is independent of the vectorization process. The embedding model generates high-dimensional representations of the sanitized text. While masking might remove some data, it does not interrupt the pipeline execution or cause the vectorization model to fail.

Random shuffling of all documents regardless of their timestamps.

Why it fails: Randomly shuffling documents in temporal datasets is the primary cause of data leakage. It allows the model to observe future information during the training phase, which is impossible in real-world scenarios. This invalidates performance evaluation metrics and leads to models that perform poorly once deployed in actual environments.

Oversampling the minority class in the training set to improve balance.

Why it fails: Oversampling can introduce duplicate entries that skew the distribution and lead to overfitting. While it addresses class imbalance, it does not mitigate the temporal data leakage caused by training on future information. It is a technique for balancing distributions, not for maintaining chronological integrity during dataset preparation.

Applying aggressive data normalization to all numerical values.

Why it fails: Normalization is a standard preprocessing step that ensures model stability by scaling inputs, but it does not address the temporal sequence leakage. It ensures that inputs are mathematically comparable, but it has no effect on whether the model is seeing future labels or temporal patterns during its training phase.

It specifically targets the removal of personally identifiable information (PII).

Why it fails: The configuration settings focus on text length, coherence (perplexity), and document similarity. None of these parameters are designed to detect or sanitize personally identifiable information. PII removal requires specialized regex patterns or entity recognition models, which are not represented in this configuration file or its associated policies.

It enforces a strict length-based chunking strategy for all documents.

Why it fails: The configuration sets a minimum word count as a filtering policy, not as a chunking strategy. Filtering removes entire documents that fail to meet the criteria, whereas chunking would split long documents into smaller segments. These are distinct processes in data preparation that serve different purposes.

It converts all text to a vector space representation before filtering.

Why it fails: The policy operates on the raw text corpus (perplexity, word count, document similarity). While MinHash uses hashing, it is a technique for efficient similarity estimation of text sets, not a general vectorization process. The filtering happens prior to any full-scale embedding or vectorization performed during training.

It guarantees that the model will always generate the same output for a given prompt.

Why it fails: Tokenization stability is about consistent input representation, not output generation. Output determinism depends on other factors, such as seed control, temperature settings, and model architecture. While stable tokenization is a prerequisite for predictable inputs, it cannot control or guarantee the output consistency of the underlying probabilistic LLM.

It ensures that the GPU memory usage remains constant during the training process.

Why it fails: Tokenization happens in the data preparation or pre-processing stage and does not impact GPU memory usage during training. Memory usage is primarily influenced by model parameters, activation tensors, and batch sizes. The stability of tokenization is unrelated to the operational memory management of the GPU hardware.

It reduces the total number of parameters required for the embedding layer.

Why it fails: The stability of tokenization does not influence the size of the embedding layer or the total parameter count of the model. The embedding layer is determined by the vocabulary size defined during the model architecture phase. Tokenization is simply a translation process that occurs prior to entering the model.

Using extremely large chunks to ensure that every document is a single vector.

Why it fails: Using single-document vectors creates an index that is too coarse for granular retrieval. The model loses the ability to pinpoint specific information, leading to retrieval failures when the query targets a small detail within a large document. This approach drastically degrades the precision of the retrieval system.

Storing every sentence as an individual chunk in the vector database.

Why it fails: Storing individual sentences creates an extremely sparse and noisy index. The system loses the broad context required to understand the query's intent. While this is granular, it fails to provide sufficient information to the LLM for high-quality generation, as each chunk lacks the necessary surrounding contextual information.

Removing all overlaps to keep the index size at the absolute minimum possible.

Why it fails: Removing all overlaps creates 'boundary effects' where critical information is split between two chunks, making it inaccessible for retrieval. While this minimizes the index size, it destroys the utility of the retrieval system. The performance impact of missing information far outweighs the minor savings in index storage space.

To artificially inflate the model's perplexity scores during evaluation.

Why it fails: Increasing perplexity is an indicator of poor model quality, not a desired outcome. Adversarial training aims to improve the model's robustness and safety, not to manipulate evaluation metrics like perplexity. A model should ideally have lower perplexity while maintaining high safety standards across various input types.

To increase the model's fluency in generating complex technical instructions.

Why it fails: Adversarial training focuses on safety and constraint adherence, not on improving the model's fluency or instructional capability. While the model must remain fluent, the primary goal of including adversarial examples is to harden the model's refusal mechanisms and constraint compliance against malicious user inputs.

To allow the model to learn and reproduce the adversarial techniques during generation.

Why it fails: The objective of including adversarial examples is to train the model to recognize and refuse, not to replicate or perform these techniques. Reproducing adversarial techniques would be a failure of the safety training, potentially making the model a tool for generating harmful or insecure content.

Calculating the total number of words in the dataset.

Why it fails: Total word count is a measure of size, not diversity. A dataset could have millions of words but repeat the same concept, resulting in poor diversity. Relying on raw size metrics without considering content variety is a common mistake that leads to models with poor generalization capabilities.

Measuring the average response length for every instruction.

Why it fails: Response length is a formatting metric, not a measure of content diversity. While having consistent length can be useful for training, it says nothing about the topical or conceptual variety within the instruction set. A dataset could have uniform lengths but be entirely redundant in its subject matter.

Checking if the dataset is solely in ASCII format.

Why it fails: Character encoding (like ASCII) is a matter of data hygiene and format, not content diversity. Whether a dataset is in ASCII or UTF-8 has no bearing on whether it covers diverse topics or linguistic styles. Focus should be on the semantic content, not the storage encoding format.

Increasing the complexity of the input text to help the model learn more features.

Why it fails: Normalization aims to simplify and standardize inputs, not increase their complexity. Adding unnecessary complexity would make it harder for the model to identify patterns, essentially wasting computational resources on noise. The goal is to maximize the signal-to-noise ratio in the input data provided to the model.

Encrypting the dataset to protect sensitive information during training.

Why it fails: Normalization is not an encryption process. Encryption involves complex cryptographic algorithms to secure data, whereas normalization is a preprocessing technique used to clean and standardize inputs. Confusing these two distinct domains would lead to an inability to actually train the model on the prepared dataset.

Compressing the dataset to reduce storage space on the GPU disk.

Why it fails: While data normalization might occasionally result in slightly smaller file sizes due to cleaned whitespace, its primary purpose is not compression. Compression techniques are designed for storage efficiency, whereas normalization is designed for model performance by ensuring consistent semantic inputs. They are fundamentally different engineering operations.

It helps to anonymize the patient data by replacing all medical terms with generic labels.

Why it fails: Replacing medical terms with generic labels would destroy the model's ability to reason about clinical conditions, rendering it useless for medical purposes. Anonymization should be handled by de-identification tools that remove PII while keeping the clinical terminology intact to maintain the functional integrity of the dataset.

It ensures that the dataset size is significantly reduced to fit in GPU cache.

Why it fails: Domain-specific cleaning is about content quality and accuracy, not reducing data size. While cleaning might reduce size, the primary motivation is the preservation of technical context. Relying on cleaning simply to shrink a dataset is the wrong approach, as it prioritizes performance over the model's accuracy.

It automatically corrects all scientific inaccuracies present in the original documents.

Why it fails: No cleaning pipeline can automatically fix scientific inaccuracies in source text. Cleaning is intended to manage formatting and noise; it does not replace the need for expert curation or ground-truth verification of the medical information itself. Relying on cleaning to fix accuracy is a dangerous assumption.

It limits the vocabulary size to 20% of the original content.

Why it fails: Lexical diversity is a measure of the variety of words used in a text, not a cap on the vocabulary size. The threshold of 0.2 indicates a minimum level of language variety required for a document to be included, not a limitation on the number of unique tokens.

It forces the model to use 20% more computation for tokenization.

Why it fails: The filter operates on the text during the data preparation stage, long before tokenization or model training occur. It has absolutely no impact on the computational resources required for tokenization or model training. It is purely a data selection step to improve the quality of the training set.

It reduces the training corpus size by exactly 20%.

Why it fails: A threshold of 0.2 refers to the lexical diversity score, not the percentage of data to be removed. The actual reduction in corpus size depends entirely on how much of the dataset fails to meet this diversity threshold, which is determined by the quality of the raw data.

Randomly masking 50% of the words in the training documents.

Why it fails: Masking 50% of words destroys the semantic structure of technical documents. This would make it impossible for the model to learn relationships between concepts. Such aggressive masking is used for training masked language models, not as a general augmentation strategy for fine-tuning in specialized domains.

Using a smaller, unverified LLM to generate completely new technical scenarios.

Why it fails: Using an unverified LLM introduces a high risk of producing factually incorrect, hallucinated technical information. In a specialized domain, this error-prone data will degrade the model's performance and compromise its reliability. Augmentation must be based on verified information, not generated blindly by potentially flawed models.

Translating the dataset into multiple languages using a generic online translator.

Why it fails: Online translators often lack the specialized knowledge required to translate technical terminology accurately. This results in poor-quality data that introduces inaccuracies and linguistic errors, which will negatively impact the model's performance in the target domain. It is an unreliable and dangerous augmentation strategy for specialized technical tasks.

Increasing the batch size during the training phase.

Why it fails: Batch size impacts memory utilization and gradient estimation stability but does not resolve structural inconsistencies in the input corpus. Data preparation must occur before training begins to ensure the model learns from clean, standardized representations that accurately reflect the desired domain vocabulary and context.

Expanding the model's depth with additional transformer layers.

Why it fails: Adding layers increases model capacity but exacerbates the issue if the input data remains noisy. Training on unnormalized data leads to fragmented tokenization, where similar concepts are represented as disjointed inputs, preventing the model from effectively capturing the underlying linguistic patterns regardless of its size.

Reducing the learning rate to prevent overfitting on the noise.

Why it fails: Lowering the learning rate primarily helps with convergence stability in the presence of noisy gradients but does not correct the root cause of inconsistent terminology. The model will still be forced to treat variant tokens as distinct entities, hindering its ability to generalize across the document set.

Increasing the frequency of data augmentation in the training pipeline.

Why it fails: While augmentation can diversify data, it does not address data leakage or inherent redundancies. Excessive augmentation might actually introduce synthetic artifacts that confuse the model, and it is not a substitute for rigorous dataset sanitization and splitting protocols required to ensure scientific validity in model training.

Adding metadata tags to all training sequences for classification.

Why it fails: Metadata tagging is useful for instruct-tuning or multi-task learning but does not prevent leakage or deduplication errors. Including metadata without proper filtering might actually lead to unintentional leakage if the tags provide hints about the content that is meant to be held out for testing.

Converting all text to lowercase to increase vocabulary efficiency.

Why it fails: Lowercasing is a normalization technique, not a strategy for preventing leakage. While it reduces the vocabulary size, it can result in the loss of important semantic information, such as distinguishing between proper nouns and common nouns, which may be detrimental for specific downstream LLM applications.

It limits the memory footprint of the dataloader.

Why it fails: The filter primarily impacts the quality and distribution of the training data rather than the memory footprint of the dataloader. Memory usage is governed more by sequence length and batch size parameters, whereas filtering short sequences focuses on removing low-entropy tokens that hinder the model's learning process.

It enables faster tokenization by reducing the number of input files.

Why it fails: Filtering does not inherently accelerate the tokenization process itself, which is a compute-intensive task. Instead, it improves the quality of the tokens generated by ensuring that the tokenizer processes long-form, context-rich sequences, which prevents the generation of too many padding tokens for short inputs.

It forces the model to ignore PII-heavy documents.

Why it fails: PII redaction is handled by the 'pii_redaction' flag. The 'min_words' filter is strictly about document length and information density. Using a length filter to control PII would be an ineffective, indirect strategy that would inadvertently remove valid, high-quality documents that happen to be concise.

Apply aggressive regex-based whitespace removal across all extracted string buffers to normalize sentence boundaries prior to embedding generation.

Why it fails: Aggressive whitespace removal collapses the column delimiters and cell boundaries that encode a table's spatial structure, worsening the scrambling rather than recovering it. Regex normalisation suits prose cleanup where sentence segmentation matters. Preserving tabular integrity requires layout-aware extraction, such as table-structure parsing, before any tokenisation or embedding step runs.

Increase the chunk size parameter in the tokenizer configuration so that entire table blocks fit inside a single oversized context window.

Why it fails: Increasing the tokenizer chunk size does not fix structural extraction failures. If the underlying extraction tool outputs scrambled strings, a larger context window simply ingests larger amounts of corrupted, unformatted text without restoring table semantics.

Convert all PDF pages into low-resolution JPEG images to bypass text extraction errors and feed raw pixels directly into a text-only causal language model.

Why it fails: Feeding raw pixels into a text-only causal language model discards the textual tokens it expects and cannot preserve table structure. Image conversion suits vision-language models or OCR pipelines, not NeMo Curator's text extraction and tokenisation stage.