---
title: "Orthogonal Projection and Least Squares"
description: "Project a vector onto a subspace to get the closest point in it. Applied to an inconsistent system, this is the least squares solution, which is where regression comes from."
canonical: https://lightmysky.com/learn/mathematics/orthogonal-projection-and-least-squares-mt_kGvRrYJrat
source: https://lightmysky.com/learn/mathematics/orthogonal-projection-and-least-squares-mt_kGvRrYJrat.md
retrieved: 2026-09-12
---

> **Agent view.** This is the Markdown twin of the page, for tools and assistants.
> When to use this site, and the call that answers each job: https://lightmysky.com/agent-instructions.md
> API description (OpenAPI 3.1): https://lightmysky.com/openapi.json · Authentication: https://lightmysky.com/auth.md
> Pricing: https://lightmysky.com/pricing.md · Catalog: https://lightmysky.com/llms.txt · Full catalog: https://lightmysky.com/llms-full.txt
> Every machine-readable file on this domain: https://lightmysky.com/.well-known/ai-catalog.json
> Ask for Markdown with `Accept: text/markdown`, a `.md` address, or `?mode=agent`.

# Orthogonal Projection and Least Squares

Project a vector onto a subspace to get the closest point in it. Applied to an inconsistent system, this is the least squares solution, which is where regression comes from.

Subject: Mathematics · Area: Linear Algebra · Ages 20 to 21
Page: https://lightmysky.com/learn/mathematics/orthogonal-projection-and-least-squares-mt_kGvRrYJrat

## Ready when they can

- Project a vector onto a subspace and verify the residual is orthogonal to it
- Set up the normal equations for an inconsistent system
- Explain why the least squares solution minimises the residual length

## Lesson: Dropping straight down onto the data sheet

Each column of your matrix A is a vector, and together the columns span a flat sheet called the column space. The data vector b usually sticks out of that sheet, so no exact solution to A times x equals b exists. The fix is to drop b straight down onto the sheet. That landing point is the projection, and it is the closest reachable point to b.

**Example.** Project (3, 4) onto the direction (1, 0). The dot product of the two is 3, and the squared length of (1, 0) is 1, so the scalar is 3 and the projection is (3, 0). In general you divide the dot product by the squared length of the target direction, then scale the direction by that number. Try (2, 6) onto (1, 1): the dot is 8, the squared length is 2, so the scalar is 4 and the answer is 4 times (1, 1).

The gap between b and its projection is the residual. Because the drop is perpendicular, the residual stands at right angles to everything in the column space. Solving the normal equations finds the x that produces exactly this landing point. That is why the least squares answer is the best possible compromise: it minimises the length of the residual.

**Tip.** Fitting a line is the same story in disguise. Each miss between a point and the line is a residual, and squaring stops positive and negative misses from cancelling. The least squares line is the one with the smallest total of squared misses, and every data set has exactly one such line.

**Recap.** Drop b straight onto the column space and solve the normal equations for the closest reachable point.

## Practice

14 questions on this page, each with its working shown.

## Needs first

- [The Regression Line and Making Predictions](https://lightmysky.com/learn/mathematics/the-regression-line-and-making-predictions-mt_tsjctRmkyE)
- [Inner Products, Length and Orthogonality](https://lightmysky.com/learn/mathematics/inner-products-length-and-orthogonality-mt_X0HPRGto4W)

## Opens up

- [The Gram-Schmidt Process](https://lightmysky.com/learn/mathematics/the-gram-schmidt-process-mt_nkp231EA_E)
- [The Singular Value Decomposition](https://lightmysky.com/learn/mathematics/the-singular-value-decomposition-mt_UsUZbM9VRK)
- [Simple Linear Regression and Inference on the Slope](https://lightmysky.com/learn/mathematics/simple-linear-regression-and-inference-on-the-slope-mt_W2huNTKsMj)
- [Hilbert Spaces and Orthogonal Projection in Infinite Dimensions](https://lightmysky.com/learn/mathematics/hilbert-spaces-and-orthogonal-projection-in-infinite-dimensions-mt_xmKq8SAgvy)