Handling Right-Skewed Data with Log Transformation

Quick Answer

The signal to catch here is heavily right-skewed, which describes a distribution with a long tail of unusually large values stretching out to the right, a pattern income data commonly follows, since most people earn within a moderate range while a small number earn far more. A log transformation compresses that long tail of large values much more than it compresses the smaller values near the bulk of the distribution, because the logarithm function grows more slowly as its input increases; the practical effect is that extreme high values get pulled closer to the rest of the data, while the low end of the distribution is stretched out relatively more, making the overall shape more symmetric and closer to normal. This matters for downstream modeling because many algorithms, particularly linear models and anything assuming roughly normal or homoscedastic errors, perform better and produce more interpretable coefficients when their input features aren't dominated by a small number of extreme outlying values. Log transformation is especially well suited to data like income precisely because such data often approximates a log-normal distribution to begin with, meaning the log of the values is closer to normally distributed even though the raw values are not. When exploratory data analysis reveals a heavily right-skewed numeric feature, especially one involving monetary amounts, counts, or other naturally multiplicative quantities, a log transformation is usually the first and most standard technique to reach for to reduce that skew.