Orthogonal Projection and Least Squares
Project a vector onto a subspace to get the closest point in it. Applied to an inconsistent system, this is the least squares solution, which is where regression comes from.
What a learner can do afterwards
- Project a vector onto a subspace and verify the residual is orthogonal to it
- Set up the normal equations for an inconsistent system
- Explain why the least squares solution minimises the residual length
1 · Read
Each column of your matrix A is a vector, and together the columns span a flat sheet called the column space. The data vector b usually sticks out of that sheet, so no exact solution to A times x equals b exists. The fix is to drop b straight down onto the sheet. That landing point is the projection, and it is the closest reachable point to b.
Project (3, 4) onto the direction (1, 0). The dot product of the two is 3, and the squared length of (1, 0) is 1, so the scalar is 3 and the projection is (3, 0). In general you divide the dot product by the squared length of the target direction, then scale the direction by that number. Try (2, 6) onto (1, 1): the dot is 8, the squared length is 2, so the scalar is 4 and the answer is 4 times (1, 1).
The gap between b and its projection is the residual. Because the drop is perpendicular, the residual stands at right angles to everything in the column space. Solving the normal equations finds the x that produces exactly this landing point. That is why the least squares answer is the best possible compromise: it minimises the length of the residual.
Fitting a line is the same story in disguise. Each miss between a point and the line is a residual, and squaring stops positive and negative misses from cancelling. The least squares line is the one with the smallest total of squared misses, and every data set has exactly one such line.
Drop b straight onto the column space and solve the normal equations for the closest reachable point.
2 · Watch
Take it off screen
Where it sits
8 questions wait behind this lesson, each with its answer explained. Every answer feeds the sky: stars light as they are learned, and dim when it is time to come back.