NCP-GENL Evaluation Practice Question

Answer analysis

Option-by-option breakdown

For each option: why learners choose it and why it is or isn't the right answer here.

• ✗

  Compute BLEU score on the generated answers.

  Why it's wrong here

  BLEU score measures n-gram overlap and is designed for machine translation. In medical QA, correct answers may use different phrasing, so BLEU would unfairly penalize valid responses. It also does not assess factual correctness or hallucination. Thus, BLEU is not suitable for this evaluation goal.

• ✗

  Calculate perplexity of the model on a held-out set of medical questions.

  Why it's wrong here

  Perplexity measures how well the model predicts the next token and is not a direct measure of factual correctness or hallucination. A model can have low perplexity yet still generate false statements. Therefore, perplexity alone does not satisfy the need to evaluate correctness or detect hallucinations.

• ✓

  Use a fact-checking module that cross-references generated statements with a trusted medical knowledge base.

  Why this is correct

  A fact-checking module can verify each claim in the model's output against a curated medical knowledge base, directly detecting hallucinations or unsupported facts. This approach provides a granular, evidence-based assessment of factual accuracy, which is critical in healthcare. Integrating such a module with NeMo Evaluator allows automated flagging of unsupported statements.

• ✓

  Compare model outputs against gold-standard answers using exact match and F1 score.

  Why this is correct

  Exact match and F1 score provide a quantitative measure of answer correctness relative to ground truth. In medical QA, where precise terminology matters, these metrics can indicate how often the model produces the expected answer. While they may not capture semantic equivalence perfectly, they are standard and efficient for large-scale evaluation, making them a valid component of the assessment.

• ✗

  Run the model through NeMo Guardrails with a policy that blocks any output containing numbers.

  Why it's wrong here

  Blocking outputs with numbers is an arbitrary rule that would not effectively detect hallucinations; many correct medical answers include numbers (e.g., dosages). This approach would introduce false positives and fail to assess factual accuracy. It does not address the core evaluation goals of correctness and hallucination detection.