AIF-C01 Practice Question: Evaluate the quality of a text generation model…

Answer analysis

Option-by-option breakdown

For each option: why learners choose it and why it is or isn't the right answer here.

• ✗

  BLEU

  Why it's wrong here

  BLEU measures precision of n-gram overlap, penalising length via brevity penalty, but it does not account for recall of the reference content, so it under-rewards summaries that capture all key points. It is designed for machine translation, where precision-oriented scoring of short outputs against references is the standard evaluation approach.

• ✗

  Perplexity

  Why it's wrong here

  Perplexity scores how fluently a language model predicts a token sequence, requiring no reference summary at all, so it cannot compare output against human references. It is tempting because it is a standard intrinsic metric for language models, and would be correct when judging a model's fluency or comparing trained checkpoints.

• ✓

  ROUGE

  Why this is correct

  ROUGE measures n-gram overlap between generated and reference summaries, directly satisfying the stem's requirement for an automated metric with human reference summaries. Recall-oriented variants such as ROUGE-N and ROUGE-L quantify how much reference content the generated summary captures, making it the standard choice for summarization evaluation.

• ✗

  BERTScore

  Why it's wrong here

  BERTScore embeds tokens and computes cosine similarity between contextual vectors, so it never counts n-gram overlap; it captures semantic equivalence instead. It is tempting because it handles paraphrasing and synonymy that BLEU penalises, making it the right pick when generated text differs in wording but matches reference meaning.

Go deeper

Related to this question