AI-900 Practice Question: Describe features of Natural Language Processing workloads on Azure

Answer analysis

Option-by-option breakdown

For each option: why learners choose it and why it is or isn't the right answer here.

• ✗

  Measuring summary quality by counting how many sentences were preserved from the original

  Why it's wrong here

  Counting how many sentences were preserved from the original simply measures an extractive overlap rate, indicating that the summary is largely a verbatim excerpt rather than a condensed synthesis. This metric fails to judge whether those sentences are the most salient in the document or whether the generated text properly resembles a reference summary. ROUGE is agnostic to extractive versus abstractive generation; it computes n-gram overlap between the output and reference, allowing both styles to be evaluated fairly on content quality.

• ✓

  ROUGE scores measuring n-gram overlap between generated and reference summaries

  Why this is correct

  ROUGE (Recall-Oriented Understudy for Gisting Evaluation) is the standard automated metric for summarisation; it compares a generated summary against human-written reference summaries by computing n-gram overlap. ROUGE-1/2/L capture unigrams, bigrams, and the longest common subsequence, respectively, offering a reproducible and objective way to gauge content fidelity. This reference-based approach avoids the cost of human annotation while correlating reasonably with human judgments, making it the conventional quality metric for summarisation tasks.

• ✗

  Asking users to rate summary quality on a 1-10 scale in production

  Why it's wrong here

  Collecting 1-10 user ratings in production is a form of human evaluation that is subjective, costly, and difficult to replicate, since each rater's interpretation of the scale can vary. Automated evaluation with ROUGE instead compares the generated summary directly to reference summaries via n-gram overlap, requiring no manual annotation during model iteration. While user ratings can reflect satisfaction, they do not provide an objective, automated measure of summarisation quality and are not a standard benchmark metric.

• ✗

  Measuring how much shorter the summary is compared to the original document

  Why it's wrong here

  Measuring how much shorter the summary is than the original document assesses only the compression ratio, which is a length efficiency metric rather than a quality metric. A summary can achieve a large reduction in length while omitting crucial information or producing garbled content, so compression alone cannot validate whether the key ideas were preserved. ROUGE, by contrast, scores the lexical correspondence between the generated summary and a reference summary, directly evaluating content retention rather than size reduction.

Go deeper

Related to this question