Singular Value Decomposition
Every matrix factors into a rotation, a pure scaling, and another rotation — the most useful decomposition in applied linear algebra.
Singular Value Decomposition
The singular value decomposition says something almost too clean to believe: every Matrix A, of any shape, can be written as
where U and V are rotations (orthogonal matrices) and \Sigma is a diagonal scaling. In words, any Linear Transformation — however much it seems to shear and skew — is really just rotate, stretch along perpendicular axes, rotate again. The stretch factors on \Sigma's diagonal are the singular values \sigma_1 \ge \sigma_2 \ge \cdots \ge 0, and they measure how much the map amplifies space along each of its principal directions.
Rotate, scale, rotate
Geometrically, V^{\top} first turns the input so that the transformation's natural axes line up with the coordinate axes; \Sigma stretches each of those axes by its singular value; and U rotates the result into its final orientation. The image of the unit circle under any matrix is an ellipse, and the SVD hands you that ellipse directly: its semi-axis lengths are the singular values, and their directions are the columns of U.
Low-rank approximation
Relation to eigenvalues
The SVD is the well-behaved cousin of eigendecomposition. Eigenvectors need not be perpendicular and exist cleanly only for square matrices, but singular values are always real, non-negative, and come with orthogonal axes — for any matrix at all. In fact the singular values of A are the square roots of the eigenvalues of A^{\top}A, tying the two ideas together.