NCP-GENL Data Preparation Practice Question

Answer analysis

Option-by-option breakdown

For each option: why learners choose it and why it is or isn't the right answer here.

• ✗

  HeuristicFilter stage with a quality threshold

  Why it's wrong here

  HeuristicFilter removes low-quality documents based on rules such as word count, perplexity, or symbol ratios, but it does not compare documents against each other. It cannot detect exact or near-duplicate content, so it leaves redundancy in the corpus and does not address the memorization and compute concerns described.

• ✗

  DocumentDownloader stage followed by DocumentResolver

  Why it's wrong here

  DocumentDownloader and DocumentResolver are ingestion stages that fetch and resolve source documents; they do not perform deduplication. Running them would at best re-fetch the same content, and they provide no exact or fuzzy matching capability, so the near-duplicate and exact-duplicate removal goal would not be met.

• ✓

  ExactDuplicates and FuzzyDuplicates stages

  Why this is correct

  NeMo Curator provides ExactDuplicates and FuzzyDuplicates stages that identify redundant documents; ExactDuplicates catches byte-identical texts, while FuzzyDuplicates uses MinHash/LSH to find near-duplicates. Applying both in sequence on the 600 GB corpus removes redundant content, lowering memorization risk and reducing the effective training token count.

• ✗

  ClassifierFilter stage using a domain classifier

  Why it's wrong here

  ClassifierFilter scores documents with a trained model to keep or drop them by domain relevance or quality. It evaluates each document independently and has no pairwise comparison mechanism, so it will not identify duplicate or near-duplicate documents, leaving the redundancy problem unsolved.