---
title: "Regularisation: Charging for Complexity"
description: "Add a penalty on the size of the coefficients to the thing being minimised, and the fit pulls back from chasing noise. A squared penalty shrinks everything smoothly, an absolute one drives some coeffi"
canonical: https://lightmysky.com/learn/computing/regularisation-charging-for-complexity-mt_RVgsDP0XzP
source: https://lightmysky.com/learn/computing/regularisation-charging-for-complexity-mt_RVgsDP0XzP.md
retrieved: 2026-09-12
---

> **Agent view.** This is the Markdown twin of the page, for tools and assistants.
> When to use this site, and the call that answers each job: https://lightmysky.com/agent-instructions.md
> API description (OpenAPI 3.1): https://lightmysky.com/openapi.json · Authentication: https://lightmysky.com/auth.md
> Pricing: https://lightmysky.com/pricing.md · Catalog: https://lightmysky.com/llms.txt · Full catalog: https://lightmysky.com/llms-full.txt
> Every machine-readable file on this domain: https://lightmysky.com/.well-known/ai-catalog.json
> Ask for Markdown with `Accept: text/markdown`, a `.md` address, or `?mode=agent`.

# Regularisation: Charging for Complexity

Add a penalty on the size of the coefficients to the thing being minimised, and the fit pulls back from chasing noise. A squared penalty shrinks everything smoothly, an absolute one drives some coefficients to zero and selects features on the way.

Subject: Computing · Area: Machine Learning · Ages 20 to 21
Page: https://lightmysky.com/learn/computing/regularisation-charging-for-complexity-mt_RVgsDP0XzP

## Ready when they can

- Add a penalty term and show what happens to the coefficients as it grows
- Say why the two common penalties behave differently at zero
- Explain why features have to be scaled before a penalty is fair

## Lesson: A charge for big numbers

You already know a flexible model can chase noise and fail on new data. Regularisation fights that by adding a penalty on coefficient size to the thing being minimised. As the penalty grows, the fit pulls back from wild coefficients.

**Example.** A regression fits training points almost perfectly with huge coefficients, then flops on new points. You add a penalty, refit, and watch the coefficients shrink. The training error rises a little and the new-data error falls a lot.

The two common penalties differ at zero. A squared penalty shrinks every coefficient smoothly toward zero but never quite lands on it. An absolute penalty has a sharp corner at zero, so it drives some coefficients exactly to zero and selects features on the way.

**Tip.** Scale your features before penalising. Without scaling, a feature measured in thousands looks guilty just for its units and takes an unfair share of the penalty.

**Recap.** A penalty on coefficient size calms overfitting, and the absolute form zeroes features while the squared form only shrinks them.

## Practice

8 questions on this page, each with its working shown.

## Needs first

- [Data as a Matrix: Rows, Features and the Target](https://lightmysky.com/learn/computing/data-as-a-matrix-rows-features-and-the-target-mt_KEN-TU1Gpn)
- [Inner Products, Length and Orthogonality](https://lightmysky.com/learn/mathematics/inner-products-length-and-orthogonality-mt_X0HPRGto4W)
- [Overfitting and the Bias-Variance Trade-off](https://lightmysky.com/learn/computing/overfitting-and-the-bias-variance-trade-off-mt_xznyNxeNNw)

## Opens up

- [Adapting a Pretrained Model: Fine-Tuning and Low-Rank Updates](https://lightmysky.com/learn/computing/adapting-a-pretrained-model-fine-tuning-and-low-rank-updates-mt_NaA2iGtMPc)
- [Cross-Validation and Honest Model Selection](https://lightmysky.com/learn/computing/cross-validation-and-honest-model-selection-mt_YG7X3HYCrT)