The Singular Value Decomposition
Every matrix, square or not, factors into a rotation, a scaling and another rotation. The singular values rank the directions by how much the map stretches them, which is how low-rank approximation works.
What a learner can do afterwards
- Relate singular values to eigenvalues of the matrix times its transpose
- Read the rank of a matrix off its singular values
- Explain how truncating the decomposition gives a best low-rank approximation
1 · Read
Every matrix, square or not, factors into a rotation, a scaling, and another rotation. That is the singular value decomposition. The scalings are the singular values, and they rank the directions by how much the map stretches them. The largest singular value tells you the biggest stretch factor the map applies to any direction.
Take the diagonal matrix with rows (3, 0) and (0, 4). Form A transpose times A, which is diagonal with 9 and 16. The eigenvalues are 9 and 16, and their square roots are 3 and 4, so the largest singular value is 4. A wilder one: rows (3, 4) and (0, 0) give A transpose times A with trace 25 and determinant 0, hence eigenvalues 25 and 0, so the largest singular value is 5.
The recipe generalises: singular values are the square roots of the eigenvalues of A transpose times A, which are always real and nonnegative. Because of that link, you can read the rank straight off the singular values: count the nonzero ones. A zero singular value marks a direction the map flattens completely. Keep your transpose fluent, since one sign slip there ruins the whole chain.
Truncation is the payoff. Dropping the small singular values and keeping the big ones gives the best low-rank approximation of the matrix, which is how compression works. Review rank and symmetric diagonalisation first, and the whole decomposition reads as those two ideas combined.
Square root the eigenvalues of A transpose times A to rank every stretch direction of the map.
2 · Watch
Take it off screen
Where it sits
8 questions wait behind this lesson, each with its answer explained. Every answer feeds the sky: stars light as they are learned, and dim when it is time to come back.