Generative AI Leader Practice Question: During a proof-of-concept for a GenAI document…

Answer analysis

Option-by-option breakdown

For each option: why learners choose it and why it is or isn't the right answer here.

• ✗

  Measure latency and cost as the primary evaluation metrics

  Why it's wrong here

  Latency and cost measure operational efficiency, not whether summaries preserve key facts or avoid fabrication — the accuracy goal of this proof-of-concept. They become the right focus later, when tuning throughput and budget for a deployed pipeline, but here they cannot detect hallucinated or missing content.

• ✗

  Deploy to all users and collect feedback via a survey

  Why it's wrong here

  A full rollout exposes every user to unvalidated output before accuracy is established, and survey feedback measures satisfaction rather than factual retention. This suits post-deployment monitoring at scale, not a proof-of-concept needing controlled, reference-based accuracy measurement.

• ✓

  Run an A/B test with a small user group and have domain experts manually review a sample of summaries for accuracy

  Why this is correct

  At proof-of-concept stage, accuracy and key-information retention matter more than scale metrics. Expert manual review of a sampled subset directly measures those qualities, whereas a small A/B test captures user preference, not factual fidelity, and is premature before quality is validated.

• ✗

  Use ROUGE scores exclusively to compare summaries against human-written ones

  Why it's wrong here

  ROUGE measures n-gram overlap with reference summaries, so it cannot detect fabricated content or omitted key facts, which is precisely what the proof-of-concept must verify. It is tempting because ROUGE is cheap, reproducible and standard for summarisation benchmarking, and would suit large-scale regression testing once accuracy has already been established.

Go deeper

Related to this question