NCP-GENL Data Preparation Practice Question

Answer analysis

Option-by-option breakdown

For each option: why learners choose it and why it is or isn't the right answer here.

• ✗

  Convert all code to a single programming language using an automated transpiler before tokenization.

  Why it's wrong here

  Transpiling all code to one language is infeasible and would destroy language-specific syntax and idioms that the model needs to learn. It also introduces transpilation errors and loses information. The goal is to pretrain on diverse code, so preserving the original languages is essential. This approach does not remove minified or generated files and adds unnecessary complexity.

• ✗

  Increase the model's context window to 32k tokens so that entire repositories fit in one sequence.

  Why it's wrong here

  A larger context window does not clean the data; it only allows longer sequences. Minified files and license headers would still be present and would consume context with low-value tokens. The scenario is about data quality, not sequence length. Increasing context also raises memory and compute costs significantly, without addressing the root issue of noisy or repetitive code.

• ✗

  Tokenize all files with a byte-level BPE tokenizer and skip any further filtering.

  Why it's wrong here

  Byte-level BPE handles any character, but it does not remove low-quality or repetitive content. Minified files would still be tokenized into long sequences of short tokens, and license headers would remain. Skipping filtering means the model trains on noise, which can degrade performance. Tokenization is necessary but not sufficient; filtering and deduplication are also required.

• ✓

  Use language-specific filters that detect and remove minified files, license headers, and auto-generated code, then apply deduplication.

  Why this is correct

  Minified JavaScript, license headers, and generated code are low-value or repetitive patterns that harm code pretraining. NeMo Curator supports custom filters for line length, ratio of alphanumeric characters, and detection of boilerplate. Deduplication removes repeated generated files. Applying these before tokenization ensures the model learns from meaningful code rather than noise, improving downstream code generation and understanding.