NCA-GENL Experimentation Practice Question

Answer analysis

Option-by-option breakdown

For each option: why learners choose it and why it is or isn't the right answer here.

• ✗

  Stop any trial whose training loss has not decreased in the last 100 steps and discard its results.

  Why it's wrong here

  A fixed patience window ignores the fact that learning rate and batch size affect loss-curve shape; some valid trials have plateaus. Discarding results also prevents later analysis. This heuristic is arbitrary and can eliminate configurations that would have performed well with more budget.

• ✗

  Run all trials to full length but evaluate them on a smaller validation set to save time.

  Why it's wrong here

  Running every trial to completion defeats the purpose of pruning and consumes the full compute budget. A smaller validation set adds noise and may misrank trials. This approach saves time only on evaluation, not on the dominant cost of training.

• ✗

  Eliminate trials whose first-epoch loss is above the median of all trials, without further evaluation.

  Why it's wrong here

  First-epoch loss is a weak predictor of final performance, especially across different learning rates and batch sizes. Median-based elimination at epoch one can discard slow-starting configurations that would converge well. It is an aggressive heuristic without the budget-promotion structure that makes early stopping statistically meaningful.

• ✓

  Use a successive halving or Hyperband-style scheduler that allocates small budgets to many trials and promotes only the best performers to larger budgets.

  Why this is correct

  Successive halving and Hyperband evaluate many configurations with small resource budgets, then promote the top performers to larger budgets. This concentrates compute on promising trials while still exploring broadly early on. It is a principled early-stopping strategy that preserves the ability to identify strong hyperparameter regions.