AIF-C01 Fundamentals of Generative AI Practice Question

A startup is choosing between two foundation models for a summarization feature. Model X is a large general-purpose model with strong benchmark scores; Model Y is a smaller domain-specialized model with lower general benchmarks but excellent results on the startup's own document samples. Cost per token for Model Y is roughly one third of Model X. Which evaluation practice should the team follow?

⚠ Common exam trap

The trap here is equating leaderboard rankings or lowest price with fitness for a specific task, when only task-specific evaluation reveals the real trade-off.

Answer choices

Why each option matters

Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.

Correct answer & explanation

✓

Evaluate both models on a held-out set of representative documents, then weigh measured quality against cost and latency.

Model selection should be driven by measured performance on data that resembles the production workload, combined with operational constraints such as cost and latency. Public benchmarks and price alone are weak proxies, while a held-out evaluation set gives direct evidence of whether the cheaper specialized model meets the quality bar for summarization.

Answer analysis

Option-by-option breakdown

For each option: why learners choose it and why it is or isn't the right answer here.

✗
Select Model Y solely because its cost per token is lower, since summarization quality is equivalent across models.
Why it's wrong here
Assuming quality parity across models is unsupported; the scenario shows different results on the same samples. Cost alone is a valid tiebreaker only after quality requirements are met, so choosing purely on price without validating that Model Y clears the accuracy bar risks shipping a feature that underperforms on real documents.
✗
Deploy both models and let end users vote on which summaries they prefer before any offline evaluation.
Why it's wrong here
User preference testing is valuable but expensive and slow, and it exposes production traffic to an unvetted system before basic quality is established. Offline evaluation on representative samples should gate which models reach users, so starting with live voting inverts the correct order of validation activities.
✗
Select Model X because higher public benchmark scores reliably predict performance on every downstream task.
Why it's wrong here
Public benchmarks measure broad capabilities on standardized tasks and frequently do not correlate with performance on a narrow, domain-specific workload. Ignoring the startup's own sample results discards the most relevant evidence available and may lead to paying three times more for no accuracy benefit on the actual summarization task.
✓
Evaluate both models on a held-out set of representative documents, then weigh measured quality against cost and latency.
Why this is correct
Task-specific evaluation on representative, held-out data is the most reliable signal of downstream performance, and combining it with cost and latency reflects the real trade-offs of production. This approach uses the startup's own evidence rather than generic benchmarks or price alone, which is the sound engineering practice for model selection.

Go deeper

AIF-C01 Fundamentals of Generative AI Practice Question

Why each option matters

Option-by-option breakdown

Related to this question