NCP-GENL Evaluation Practice Question

Answer analysis

Option-by-option breakdown

For each option: why learners choose it and why it is or isn't the right answer here.

• ✗

  ROUGE-1 measures unigram overlap but fails to capture semantic synonyms and contextual nuances common in enterprise technical support documentation.

  Why it's wrong here

  ROUGE-1 counts exact unigram matches, so paraphrases and synonyms in support answers score poorly despite identical meaning, making it unsuitable for embedding-based semantic assessment. It is tempting because it is a cheap, widely used reference-based metric, and it would be correct where surface lexical overlap with the reference genuinely reflects answer quality.

• ✗

  BLEU evaluates n-gram precision with a brevity penalty, heavily penalizing valid creative paraphrasing typically found in conversational AI support responses.

  Why it's wrong here

  BLEU was originally designed for rigid machine translation tasks and focuses heavily on exact n-gram precision. It lacks semantic understanding and penalizes valid alternative phrasings, making it a poor choice for evaluating modern generative support assistants.

• ✓

  BERTScore computes token similarity matrices using contextual embeddings to capture deep semantic meaning regardless of surface-level phrasing variations.

  Why this is correct

  BERTScore aligns tokens between candidate and reference using contextual embeddings, computing precision, recall and F1 over cosine similarity. This satisfies the stem's requirement for embedding-based semantic assessment that captures meaning despite surface-level phrasing variation, without human annotators.

• ✗

  Perplexity measures how well a probability distribution predicts a sample, reflecting language fluency rather than semantic alignment with a specific reference answer.

  Why it's wrong here

  Perplexity scores how fluently a model predicts token sequences, with no reference answer involved, so it cannot measure alignment with curated responses. It is tempting because it is a standard intrinsic language-model metric requiring no labelled data, and it would be correct for comparing base and fine-tuned model fluency, not semantic similarity evaluation.