SVD decomposes a matrix into

where and , both orthonormal, contain left and right singular vectors respectively and diagonal contains singular values; note that has the same shape as and may not be square, but singular values still lie on the diagonal.

1. The singular values are roots of the nonzero eigenvalues of both and . This also means that we have nonzero singular values.
2. The left singular vectors in are eigenvectors of .
3. The right singular vectors in are eigenvectors of .

The singular values in satisfy

for left and right singular vector and , respectively.

Computation#

To compute the SVD, we start with . Its 🤝 Schur Decomposition is

where is diagonal containing eigenvalues . Then, the singular values of are , and the SVD is

where is computed as

and is chosen to form an orthonormal basis for .

Forms#

Observe that if isn't a square diagonal with all nonzero singular values, some parts of and are zeroed out. To make this more concrete, partition , , and based on the square singular value diagonal in :

such that SVD is equivalently

Additionally, note that if are the columns of and are the columsn of , then we can write as a sum,

Each term is a rank-one matrix with range spanned by , and their sum reconstructs rank .

Interpretation#

Geometrically, we can interpret SVD as a rotation, scaling, and another rotation.

From this, we can see that for any vector such that , multiplication with will change the norm such that

with equality when , which is scaled by the largest singular value .

Additionally, and contain subspaces of :

1. Columns of and form orthonormal bases for and .
2. Columns of and form orthonormal bases for and .

Moore-Penrose Pseudoinverse#

The Moore-Penrose pseudoinverse computes the minimum-norm least squares solution to , when , using SVD. Intuitively, the motivation behind this is that can be decomposed via

These terms represent orthogonal projections of onto and respectively, and so if , we can instead solve

Using the definition of SVD and simplifying for the minimum-norm solution, we have

and thus the Moore-Penrose pseudoinverse is

Lastly, if is full rank, we can alternatively find

Or, if its range fully spans , we have

Matrix Approximation#

We can approximate by keeping the largest singular values and their associated vectors. Specifically,

This is the best approximation of , minimizing distortion .

Furthermore,

where the subscript denotes the spectral 📌 Norm.