Orthogonal Projection and Least Squares

Use real vectors and the standard dot product. The column space of A is the subspace spanned by its columns: all vectors Ax. It need not be a plane. The orthogonal projection p of b is the unique closest vector in this subspace. The residual is r = b - p. Compute it by subtracting components, then check that its dot product with every column is zero. If b belongs to the column space, p = b and r = 0.

For a nonzero direction u, project b onto the line of all multiples of u using p = ((b dot u)/(u dot u))u. Compute the two dot products, divide, then scale u. A negative coefficient points opposite to u. The formula cannot use u = 0. Rescaling u by a nonzero number leaves its line and projection unchanged, though the coefficient may change.

For an m by n matrix A, x has n entries and Ax has m. A^T is the transpose: swap rows and columns. Each entry of A^T r is a column's dot product with r. Thus a least squares fit has A^T(b - Ax) = 0. Rearrange to obtain the normal equations A^T A x = A^T b. Every solution gives Ax equal to the projection, whether or not Ax = b is consistent. To check a candidate, dot its residual with every column.

Write ||v|| for Euclidean length. Its square is the sum of squared components. Let p be the projection and r = b - p. For any Ax, p - Ax lies in the column space and is orthogonal to r. Pythagoras gives ||b - Ax|| squared = ||r|| squared + ||p - Ax|| squared. The last term is nonnegative, so the minimum occurs exactly when Ax = p. Minimising length or squared length gives the same solutions.

The fitted vector p is unique. The coefficients x are unique when A's columns are independent. With dependent columns, different x values can give the same p. For A = [[1,1],[1,1]] and b = (2,6), the normal equations reduce to x1 + x2 = 4. Both x = (0,4) and x = (1,3) give p = (4,4). The residual (-2,2) has zero dot product with both columns.

Fit y = a + ct to observations (ti,yi), with intercept a and slope c. Put row (1,ti) in A and yi in b. Then x = (a,c) and Ax contains predictions. Each residual yi - a - cti is a vertical difference, not the perpendicular distance to the plotted line. Least squares minimises their squared sum; signed residuals can cancel. The line is unique if at least two predictor values differ. If all ti agree, different lines can give the same fitted values.

For n observations, sum means add over all observations. The line's normal equations are n a + (sum ti)c = sum yi and (sum ti)a + (sum ti^2)c = sum(ti yi). Here sum ti^2 means add the squared predictor values. Compute the sums to form the equations. Multiply equations if needed to match the coefficients of a. Subtract to eliminate a, solve for c, then substitute to find a.