Regularisation: Charging for Complexity
Add a penalty on the size of the coefficients to the thing being minimised, and the fit pulls back from chasing noise. A squared penalty shrinks everything smoothly, an absolute one drives some coefficients to zero and selects features on the way.
What a learner can do afterwards
- Add a penalty term and show what happens to the coefficients as it grows
- Say why the two common penalties behave differently at zero
- Explain why features have to be scaled before a penalty is fair
1 · Read
You already know a flexible model can chase noise and fail on new data. Regularisation fights that by adding a penalty on coefficient size to the thing being minimised. As the penalty grows, the fit pulls back from wild coefficients.
A regression fits training points almost perfectly with huge coefficients, then flops on new points. You add a penalty, refit, and watch the coefficients shrink. The training error rises a little and the new-data error falls a lot.
The two common penalties differ at zero. A squared penalty shrinks every coefficient smoothly toward zero but never quite lands on it. An absolute penalty has a sharp corner at zero, so it drives some coefficients exactly to zero and selects features on the way.
Scale your features before penalising. Without scaling, a feature measured in thousands looks guilty just for its units and takes an unfair share of the penalty.
A penalty on coefficient size calms overfitting, and the absolute form zeroes features while the squared form only shrinks them.
2 · Watch
Take it off screen
Where it sits
8 questions wait behind this lesson, each with its answer explained. Every answer feeds the sky: stars light as they are learned, and dim when it is time to come back.