NCA-GENL Experimentation Practice Question

Answer analysis

Option-by-option breakdown

For each option: why learners choose it and why it is or isn't the right answer here.

• ✗

  The two runs cannot be compared because LoRA rank changes the base model architecture.

  Why it's wrong here

  LoRA injects trainable low-rank matrices alongside frozen base weights; it does not alter the underlying architecture or the forward computation of the pretrained layers. Both runs use the same base model family and identical other hyperparameters, so the comparison is valid. Claiming incomparability would discard a legitimate and informative experimental result about adapter capacity.

• ✗

  Higher LoRA rank always improves downstream accuracy, so the evaluation set must be mislabeled.

  Why it's wrong here

  LoRA rank controls the capacity of the injected low-rank update matrices; it does not guarantee better generalization. Assuming the evaluation set is faulty because results contradict a rule of thumb inverts the evidence. The observed pattern, lower training loss with worse held-out accuracy, is a classic sign of overfitting, so the rank-64 configuration likely memorized training data rather than learning transferable patterns.

• ✗

  The alpha value should be doubled for Run 2 to restore the intended scaling ratio.

  Why it's wrong here

  Alpha scales the LoRA update relative to rank, but changing it does not address the core observation that extra capacity hurt held-out accuracy. Raising alpha for the larger-rank run would likely amplify the overfit update rather than fix it. The defensible conclusion is about capacity and generalization, not about a scaling constant, and the scenario gives no evidence that alpha scaling was miscalibrated.

• ✓

  The rank-64 run overfits the training data, so the lower training loss does not translate to better generalization.

  Why this is correct

  Increasing LoRA rank raises the number of trainable parameters in the adapter, giving the model more capacity to fit the training set. When training loss drops but held-out accuracy worsens, the extra capacity has been used to memorize rather than generalize. The rank-8 adapter is more constrained and therefore generalizes better on this evaluation set. The experiment supports the overfitting interpretation rather than a labeling error.