A matrix is a -by- tuple of elements for and . The matrix has rows and columns, represented as

Note that by this definition, a 🏹 Vector can be viewed as a matrix where one dimension is ; a column vector is a matrix, and a row vector is a matrix.

Generalizing to more than two-dimensions, we get tensors Operations on tensors are usually less defined, but in many cases, we perform matrix operations on two dimensions of the tensor while ignoring the rest.

Operations#

Addition#

Matrices are added element-by-element. For ,

Multiplication#

In matrix multiplication, , is the sum of products between the th row of and th column of . Specifically,

Note that this also means that . Additionally, multiplication with a vector can be interpreted as a weighted sum of the columns of ,

The Hadamard product, often denoted as , performs multiplication element-wise, similar to addition. That is,

where and are required to have the same shape.

Inverse#

The inverse of square matrix is the matrix satisfying . Not all matrices have an inverse, but if they do, the inverse is unique, and the matrix has full rank . For matrix products, and .

We can compute the inverse using row operations from ⚙️ System of Linear Equations > Gaussian Elimination. Simplifying

gives us the inverse. This immediately gives us the solution to the system, .

Transpose#

The transpose of a matrix is the matrix that's flipped across the diagonal. Specifically if , we have . Also, if , we call a symmetric matrix.

Adjoint#

The adjoint of a matrix is the matrix , where we first take the complex conjugate and then transpose the matrix; note that this means transpose and adjoint are the same for real matrices. Two key properties of adjoints is that

Analogously, the adjoint operator is defined as follows: for linear operator , the adjoint operator is and defined by

As a simple example, if , then .

Special Matrices#

Identity#

The identity matrix is a square matrix that has along the diagonals and everywhere else. It preserves the equation for some .

Orthogonal#

Square matrix is orthogonal if and only if its columns are orthonormal, as defined by the inner product between columns. For orthogonal matrix ,

which implies .

Moreover, transformations by orthogonal matrices preserve lengths,

and angles,

Unitary#

More generally for , unitary matrices satisfy

Orthogonality is a special case where the unitary matrix is real.

Symmetric#

Square matrix is symmetric if and only if it equals its transpose, that is

Hermitian#

More generally for , hermitian matrices satisfy

Real symmetric matrices are Hermitian. However, note that complex symmetric matrices are not Hermitian, as the adjoint also takes the complex conjugate.

A key property of Hermitian (and real symmetric) matrices is that they can be diagonalized with unitary matrix and diagonal such that eigenvalues are real, and eigenvectors are orthogonal. This is similar to 🤝 Schur Decomposition.

Square#

We can perform special functions on square matrices . First, for a polynomial , we can define the matrix polynomial

The same applies to arbitrary functions as well,

represented as a Taylor series that converges on .

An easier way to compute such functions is via 💐 Eigenvalue > Diagonalization,

Positive Definite#

A Hermitian matrix is positive definite if

for all (where ). Such a matrix has all eigenvalues .

Positive Semidefinite#

Similarly, a Hermitian matrix is positive definite if

for all (where ). Such a matrix has all eigenvalues . Also, this quality has a close connection with the 🎳 Inner Product, and every matrix can form a positive semidefinite matrix .

Exponentials#

A common function on square matrices is the exponential (eg, with time in 🇺🇸 State Space Models), . For matrix , this is

To compute the exponential, we can't directly use the Taylor series. Instead, we can use 💐 Eigenvalue > Diagonalization,

so