Regularisation: Charging for Complexity
Add a penalty on the size of the coefficients to the thing being minimised, and the fit pulls back from chasing noise. A squared penalty shrinks everything smoothly, an absolute one can drive some coefficients exactly to zero and selects features that way.
What a learner can do afterwards
- Add a penalty term and show what happens to the coefficients as it grows
- Say why the two common penalties behave differently at zero
- Explain why features have to be scaled before a penalty is fair
1 · Read
You already know a flexible model can chase noise and fail on new data. Regularisation fights that by adding a penalty on coefficient size to the thing being minimised. As the penalty grows, the fit pulls back from wild coefficients.
A regression fits training points almost perfectly with huge coefficients, then flops on new points. You add a penalty, refit, and watch the coefficients shrink. The training error rises a little and the new-data error falls a lot.
The two common penalties differ at zero. A squared penalty shrinks the coefficients smoothly and usually stops just short of zero. An absolute penalty has a sharp corner at zero, so it can park a coefficient exactly on zero, and a zeroed feature drops out of the model. How many land there depends on the data and the penalty strength: a small penalty may zero nothing, and a large one zeros everything.
Scale your features before penalising, because the size of a coefficient depends on the units of its feature. Take one length as 2 m or as 0.002 km. To add 10 to a prediction, the metre coefficient must be 5 but the kilometre coefficient must be 5000, so the kilometre version pays a far bigger penalty for the same job. Small numbers force big coefficients, so without scaling the units decide the charge instead of usefulness.
A penalty on coefficient size calms overfitting. The absolute form can zero features out, while the squared form usually only shrinks them, and both need scaled features to charge fairly.
2 · Watch
Take it off screen
Where it sits
8 questions wait behind this lesson, each with its answer explained. Every answer feeds the sky: stars light as they are learned, and dim when it is time to come back.