Mitigating Outliers in Linear Regression

Quick Answer

Outliers disproportionately affect linear regression because the standard squared-error loss function penalizes large residuals quadratically, meaning a single extreme point can pull the fitted line toward itself far more than its actual representativeness in the data would justify, so mitigating outliers means either changing how the model responds to large residuals, or changing what data it sees in the first place. Using a loss function that's robust to outliers, such as Huber loss, addresses the first path: Huber loss behaves like squared error for small residuals but switches to a linear penalty for large residuals, which caps how much influence any single extreme point can exert on the fitted coefficients compared to ordinary least squares. Removing observations with outlier values addresses the problem from the data side instead, directly eliminating points that would otherwise disproportionately skew the regression coefficients, which can be done as part of preprocessing before the model ever sees that data, for example in a SageMaker Processing job. These represent two different philosophies for the same goal, adapt the model's sensitivity to extreme values, or curate the data so extreme values aren't present to begin with, and a well-rounded approach to an outlier-heavy dataset often uses more than one such technique together, which is why this is framed as a 'choose three' question. Whenever a scenario describes outliers degrading a regression model and asks for mitigation techniques, expect the correct answers to include both loss-function changes and data-level interventions, since both are standard, complementary approaches.