--- title: "Orthogonal Projection and Least Squares" description: "For real matrices with the standard dot product, project b onto the column space of A. Solve the normal equations to minimise the squared residual length, and apply the method to straight-line regress" canonical: https://lightmysky.com/learn/mathematics/orthogonal-projection-and-least-squares-mt_kGvRrYJrat source: https://lightmysky.com/learn/mathematics/orthogonal-projection-and-least-squares-mt_kGvRrYJrat.md retrieved: 2026-11-12 --- > **Agent view.** This is the Markdown twin of the page, for tools and assistants. > When to use this site, and the call that answers each job: https://lightmysky.com/agent-instructions.md > API description (OpenAPI 3.1): https://lightmysky.com/openapi.json · Authentication: https://lightmysky.com/auth.md > Pricing: https://lightmysky.com/pricing.md · Catalog: https://lightmysky.com/llms.txt · Full catalog: https://lightmysky.com/llms-full.txt > Every machine-readable file on this domain: https://lightmysky.com/.well-known/ai-catalog.json > Ask for Markdown with `Accept: text/markdown`, a `.md` address, or `?mode=agent`. # Orthogonal Projection and Least Squares For real matrices with the standard dot product, project b onto the column space of A. Solve the normal equations to minimise the squared residual length, and apply the method to straight-line regression. Subject: Mathematics · Area: Linear Algebra · Ages 20 to 21 Page: https://lightmysky.com/learn/mathematics/orthogonal-projection-and-least-squares-mt_kGvRrYJrat ## Ready when they can - Project a vector onto a subspace and verify the residual is orthogonal to it - Set up the normal equations for an inconsistent system - Explain why the least squares solution minimises the residual length ## Lesson: Projection and least squares Use real vectors and the standard dot product. The column space of A is the subspace spanned by its columns: all vectors Ax. It need not be a plane. The orthogonal projection p of b is the unique closest vector in this subspace. The residual is r = b - p. Compute it by subtracting components, then check that its dot product with every column is zero. If b belongs to the column space, p = b and r = 0. For a nonzero direction u, project b onto the line of all multiples of u using p = ((b dot u)/(u dot u))u. Compute the two dot products, divide, then scale u. A negative coefficient points opposite to u. The formula cannot use u = 0. Rescaling u by a nonzero number leaves its line and projection unchanged, though the coefficient may change. *(drawing: Check the residual: (1,-1) dot (1,1) = 0.)* For an m by n matrix A, x has n entries and Ax has m. A^T is the transpose: swap rows and columns. Each entry of A^T r is a column's dot product with r. Thus a least squares fit has A^T(b - Ax) = 0. Rearrange to obtain the normal equations A^T A x = A^T b. Every solution gives Ax equal to the projection, whether or not Ax = b is consistent. To check a candidate, dot its residual with every column. **Example.** Build A^T A by taking dot products of pairs of columns. Build A^T b by dotting each column with b. For A with rows (1,0), (1,1), (1,3) and b = (1,2,3), the columns are (1,1,1) and (0,1,3). Their dot products give A^T A = [[3,4],[4,10]], and their dots with b give A^T b = (6,11). The normal equations are 3x1 + 4x2 = 6 and 4x1 + 10x2 = 11. *(drawing: Each matrix entry is a dot product of columns.)* Write ||v|| for Euclidean length. Its square is the sum of squared components. Let p be the projection and r = b - p. For any Ax, p - Ax lies in the column space and is orthogonal to r. Pythagoras gives ||b - Ax|| squared = ||r|| squared + ||p - Ax|| squared. The last term is nonnegative, so the minimum occurs exactly when Ax = p. Minimising length or squared length gives the same solutions. The fitted vector p is unique. The coefficients x are unique when A's columns are independent. With dependent columns, different x values can give the same p. For A = [[1,1],[1,1]] and b = (2,6), the normal equations reduce to x1 + x2 = 4. Both x = (0,4) and x = (1,3) give p = (4,4). The residual (-2,2) has zero dot product with both columns. Fit y = a + ct to observations (ti,yi), with intercept a and slope c. Put row (1,ti) in A and yi in b. Then x = (a,c) and Ax contains predictions. Each residual yi - a - cti is a vertical difference, not the perpendicular distance to the plotted line. Least squares minimises their squared sum; signed residuals can cancel. The line is unique if at least two predictor values differ. If all ti agree, different lines can give the same fitted values. For n observations, sum means add over all observations. The line's normal equations are n a + (sum ti)c = sum yi and (sum ti)a + (sum ti^2)c = sum(ti yi). Here sum ti^2 means add the squared predictor values. Compute the sums to form the equations. Multiply equations if needed to match the coefficients of a. Subtract to eliminate a, solve for c, then substitute to find a. **Example.** For (0,2), (1,3), (2,7), the sums are n = 3, sum t = 3, sum y = 12, sum t^2 = 5 and sum ty = 17. The equations are 3a + 3c = 12 and 3a + 5c = 17. Subtracting gives 2c = 5, so c = 2.5. Substituting in the first equation gives a = 1.5. The fitted line is y = 1.5 + 2.5t. *(drawing: The fitted line need not pass through every observation.)* **Recap.** Solve A^T A x = A^T b to find a closest fit. Its residual is orthogonal to every column. The fitted vector is unique; coefficients need independent columns for uniqueness. *Lesson, practice questions and flashcards written for this dataset.* ## Practice 14 questions on this page, each with its working shown. ## Needs first - [Inner Products, Length and Orthogonality](https://lightmysky.com/learn/mathematics/inner-products-length-and-orthogonality-mt_X0HPRGto4W) ## Optional connections - [The Regression Line and Making Predictions](https://lightmysky.com/learn/mathematics/the-regression-line-and-making-predictions-mt_tsjctRmkyE) ## Opens up - [The Gram-Schmidt Process](https://lightmysky.com/learn/mathematics/the-gram-schmidt-process-mt_nkp231EA_E) - [The Singular Value Decomposition](https://lightmysky.com/learn/mathematics/the-singular-value-decomposition-mt_UsUZbM9VRK) - [Simple Linear Regression and Inference on the Slope](https://lightmysky.com/learn/mathematics/simple-linear-regression-and-inference-on-the-slope-mt_W2huNTKsMj) - [Hilbert Spaces and Orthogonal Projection in Infinite Dimensions](https://lightmysky.com/learn/mathematics/hilbert-spaces-and-orthogonal-projection-in-infinite-dimensions-mt_xmKq8SAgvy)