A data science team trains several machine learning models for a regression task. They observe that Model A has low training error and low test error. Model B has low training error but high test error. Model C has high training error and high test error. Which model would most likely benefit from an ensemble technique that averages the predictions of multiple models?
Model B has low training error but high test error, a classic sign of overfitting caused by high variance: the model memorizes noise in the training set and fails to generalize. Ensemble techniques such as bagging train multiple models on different bootstrap samples and average their predictions, which cancels out independent errors and stabilizes the decision boundary. This variance reduction directly targets the gap between training and test error, making Model B the clearest candidate for improvement.
Why this answer
Model B exhibits low training error but high test error, which is a classic sign of overfitting. Ensemble techniques like averaging predictions from multiple models reduce variance and improve generalization, making them most beneficial for overfit models. In Azure Machine Learning, you can use an ensemble pipeline or AutoML's VotingEnsemble to combine diverse models and lower test error.
Exam trap
The trap here is that candidates often assume ensembles always improve accuracy, but they are most effective for high-variance (overfit) models, not for underfit or already well-generalized models.
How to eliminate wrong answers
Option A is wrong because Model A already generalizes well (low training and test error), so an ensemble would provide minimal improvement and might add unnecessary complexity. Option C is wrong because Model C has high training error, indicating underfitting; ensembles primarily reduce variance, not bias, so they would not fix the underlying high bias. Option D is wrong because Model B clearly suffers from high variance, and ensemble techniques are specifically designed to address this issue by averaging predictions to smooth out overfitting.