--- title: "Regularisation: Charging for Complexity" description: "Add a penalty on the size of the coefficients to the thing being minimised, and the fit pulls back from chasing noise. A squared penalty shrinks everything smoothly, an absolute one can drive some coe" canonical: https://lightmysky.com/learn/computing/regularisation-charging-for-complexity-mt_RVgsDP0XzP source: https://lightmysky.com/learn/computing/regularisation-charging-for-complexity-mt_RVgsDP0XzP.md retrieved: 2026-11-12 --- > **Agent view.** This is the Markdown twin of the page, for tools and assistants. > When to use this site, and the call that answers each job: https://lightmysky.com/agent-instructions.md > API description (OpenAPI 3.1): https://lightmysky.com/openapi.json · Authentication: https://lightmysky.com/auth.md > Pricing: https://lightmysky.com/pricing.md · Catalog: https://lightmysky.com/llms.txt · Full catalog: https://lightmysky.com/llms-full.txt > Every machine-readable file on this domain: https://lightmysky.com/.well-known/ai-catalog.json > Ask for Markdown with `Accept: text/markdown`, a `.md` address, or `?mode=agent`. # Regularisation: Charging for Complexity Add a penalty on the size of the coefficients to the thing being minimised, and the fit pulls back from chasing noise. A squared penalty shrinks everything smoothly, an absolute one can drive some coefficients exactly to zero and selects features that way. Subject: Computing · Area: Machine Learning · Ages 20 to 21 Page: https://lightmysky.com/learn/computing/regularisation-charging-for-complexity-mt_RVgsDP0XzP ## Ready when they can - Add a penalty term and show what happens to the coefficients as it grows - Say why the two common penalties behave differently at zero - Explain why features have to be scaled before a penalty is fair ## Lesson: A charge for big numbers You already know a flexible model can chase noise and fail on new data. Regularisation fights that by adding a penalty on coefficient size to the thing being minimised. As the penalty grows, the fit pulls back from wild coefficients. **Example.** A regression fits training points almost perfectly with huge coefficients, then flops on new points. You add a penalty, refit, and watch the coefficients shrink. The training error rises a little and the new-data error falls a lot. The two common penalties differ at zero. A squared penalty shrinks the coefficients smoothly and usually stops just short of zero. An absolute penalty has a sharp corner at zero, so it can park a coefficient exactly on zero, and a zeroed feature drops out of the model. How many land there depends on the data and the penalty strength: a small penalty may zero nothing, and a large one zeros everything. Scale your features before penalising, because the size of a coefficient depends on the units of its feature. Take one length as 2 m or as 0.002 km. To add 10 to a prediction, the metre coefficient must be 5 but the kilometre coefficient must be 5000, so the kilometre version pays a far bigger penalty for the same job. Small numbers force big coefficients, so without scaling the units decide the charge instead of usefulness. **Recap.** A penalty on coefficient size calms overfitting. The absolute form can zero features out, while the squared form usually only shrinks them, and both need scaled features to charge fairly. ## Practice 8 questions on this page, each with its working shown. ## Needs first - [Overfitting and the Bias-Variance Trade-off](https://lightmysky.com/learn/computing/overfitting-and-the-bias-variance-trade-off-mt_xznyNxeNNw) ## Optional connections - [Data as a Matrix: Rows, Features and the Target](https://lightmysky.com/learn/computing/data-as-a-matrix-rows-features-and-the-target-mt_KEN-TU1Gpn) ## Opens up - [Adapting a Pretrained Model: Fine-Tuning and Low-Rank Updates](https://lightmysky.com/learn/computing/adapting-a-pretrained-model-fine-tuning-and-low-rank-updates-mt_NaA2iGtMPc) - [Cross-Validation and Honest Model Selection](https://lightmysky.com/learn/computing/cross-validation-and-honest-model-selection-mt_YG7X3HYCrT)