NCP-GENL Evaluation Practice Question

Answer analysis

Option-by-option breakdown

For each option: why learners choose it and why it is or isn't the right answer here.

• ✗

  Let the same generative model that powers the assistant also grade its own answers, since it understands the domain best.

  Why it's wrong here

  Self-grading introduces a strong bias toward the model's own phrasing and blind spots, and it cannot detect errors the model systematically makes. Using the production model as judge also couples the evaluator to the artifact under test, so a regression in the model silently degrades the measurement. An independent judge or human review is required for credible scoring.

• ✓

  Hold out a prompt set that was never used during fine-tuning or prompt engineering, and keep it frozen across model versions.

  Why this is correct

  A frozen, genuinely unseen set prevents leakage and version-to-version drift in the test data itself. If prompts were recycled during prompt engineering or fine-tuning, scores inflate because the model has effectively seen the answers. Keeping the set immutable also makes run-over-run deltas interpretable, since any change in score can be attributed to the model rather than to a shifting benchmark.

• ✗

  Average the outputs of several automatic metrics into one composite score so the product owner receives a single number.

  Why it's wrong here

  Collapsing several metrics into one weighted average hides which dimension failed and lets strong performance on a weak metric mask a real regression. It also invites arbitrary weighting choices that are hard to defend. A single headline number is convenient for stakeholders but destructive for diagnosis, and it does not address leakage, contamination, or retrieval attribution.

• ✗

  Increase decoding temperature during evaluation so the model explores more of the answer space and scores reflect average-case behavior.

  Why it's wrong here

  Raising temperature adds variance that is not part of the deployed configuration, so scores no longer represent production behavior. It also makes comparisons across versions noisy because each run samples a different distribution. Evaluation should mirror the inference settings used in production, with controlled sampling, and only vary temperature deliberately when studying robustness.

• ✓

  Include a retrieval ablation that runs the same prompts with and without retrieved context to isolate the contribution of the retriever.

  Why this is correct

  In a RAG system, a good score can come from the generator's parametric knowledge or from the retrieved passages. Running the identical prompt set with retrieval disabled separates those effects, revealing whether the retriever is actually helping and whether failures are retrieval misses or generation errors. Without this ablation, improvements attributed to retrieval quality may really be model memorization.