NCA-GENL Experimentation Practice Question

Answer analysis

Option-by-option breakdown

For each option: why learners choose it and why it is or isn't the right answer here.

• ✓

  Evaluate both variants on the same held-out test set that neither saw during training

  Why this is correct

  Holding the evaluation data constant isolates the effect of the training-set size from the effect of the evaluation data. If each variant were scored on a different test set, differences in difficulty or domain mix would confound the comparison. A shared, untouched held-out set gives both models the same challenge, so observed differences can be attributed to training choices rather than evaluation noise.

• ✗

  Report the training loss of each variant at the final step

  Why it's wrong here

  Training loss measures fit to data the model has already seen and is heavily influenced by dataset size. The variant trained on fewer examples will typically show higher training loss regardless of which generalizes better, so this metric cannot answer the comparison question and may actively mislead the team into preferring the wrong variant.

• ✗

  Compare the number of parameters each variant updated during fine-tuning

  Why it's wrong here

  Parameter counts describe the training procedure, not the resulting quality on the summarization task. Two variants could update the same number of parameters yet differ substantially in output quality. This metric offers no signal about generalization and cannot substitute for measuring task performance on data the models have not memorized.

• ✗

  Let each variant be scored on its own randomly split test set

  Why it's wrong here

  Separate random splits create different evaluation samples with different difficulty and topic composition. A model could appear superior simply because its test split happened to contain easier or more repetitive summaries. To attribute performance differences to the training regime, both variants must face an identical, fixed evaluation set.