Search for probability and statistics terms on Statlect
StatLect
Index > Matrix algebra

Orthogonal projection

by , PhD

The orthogonal projection of a vector $s$ onto a given subspace $R$ is the vector $rin R$ that is closest to $s$.

Table of Contents

Essential concepts

Before explaining orthogonal projections, we are going to revise some important concepts.

Let $S$ be a vector space. Remember that two vectors $s$ and $r$ belonging to $S$ are orthogonal when their inner product is zero:[eq1]

Let $R$ be a subspace of $S$. The orthogonal complement of $R$, denoted by $R^{ot }$, is the unique subspace satisfying [eq2]

The two subspaces $R$ and $R^{ot }$ are complementary subspaces, which means that[eq3]where $oplus $ denotes a direct sum. By the properties of direct sums, any vector $sin S$ can be uniquely written as[eq4]where $rin R$ and $tin R^{ot }$.

Definition

We can now define orthogonal projections.

Definition Let $S$ be a linear space. Let $R$ be a subspace of $S$ and $R^{ot }$ its orthogonal complement. Let $sin S$ with its unique decomposition[eq5]in which $rin R$ and $tin R^{ot }$. Then, the vector $r$ is called the orthogonal projection of $s$ onto $R$ and it is denoted by [eq6].

Thus, the orthogonal projection is a special case of the so-called oblique projection, which is defined as above, but without the requirement that the complementary subspace of $R$ be an orthogonal complement.

Example Let $S$ be the space of $3 imes 1$ column vectors. Define[eq7]Its orthogonal complement is[eq8]as we can easily verify by checking that the vector spanning $R^{ot }$ is orthogonal to the two vectors spanning $R$. Now, consider the vector[eq9]Then,[eq10]

The orthogonal projection minimizes the distance

The distance between two vectors is measured by the norm of their difference.

It turns out that [eq11] is the vector of $R$ that is closest to $s$.

Proposition Let $S$ be a finite-dimensional vector space. Let $R$ be a subspace of $S$. Then, [eq12]for any $rin R$.

Proof

Since[eq13]where $tin R^{ot }$, the vector [eq14] belongs to $R^{ot }$ and, as a consequence, is orthogonal to any vector belonging to $R $, including the vector [eq15]. Therefore, [eq16]where in step $ rame{A}$ we have used Pythagoras' theorem. By taking the square root of both sides, we obtain the stated result.

Projection matrix

Suppose that $S$ is the space of Kx1 complex vectors and $R$ is a subspace of $S$.

By the results demonstrated in the lecture on projection matrices (that are valid for oblique projections and, hence, for the special case of orthogonal projections), there exists a projection matrix $P_{R}$ such that[eq17]for any $sin S$.

The projection matrix is[eq18]where:

In the case of orthogonal projections, the formula above becomes simpler.

Proposition Let $S$ be the space of complex Kx1 vectors. Let $R$ be a subspace of $S$. Let $B_{R}$ be a matrix whose columns form a basis for $R$. Denote by $B_{R}^{st }$ the conjugate transpose of $B_{R}$. Then, the matrix [eq19] is the projection matrix such that [eq20]for any $sin S$.

Proof

We choose the columns of $B_{R^{ot }}$ in such a way that they form an orthonormal basis for $R^{ot }$. As a consequence, as explained in the lecture on unitary matrices (see the section on non-square matrices with orthonormal columns), we have[eq21]where [eq22] denotes the conjugate transpose of $B_{R^{ot }}$. Moreover, since the columns of $B_{R}$ are orthogonal to the columns of $B_{R^{ot }}$, we have[eq23]and[eq24]The columns of $B_{R}$ are linearly independent since they form a basis. Hence, $B_{R}x eq 0$ for any $x eq 0$, which implies that [eq25] for any $x eq 0$. Thus, $B_{R}^{st }B_{R}$ is full-rank (hence invertible). We use these results to derive the following equality: [eq26]which implies, by the definition of inverse matrix, that[eq27]Thus,[eq28]

When we confine our attention to real vectors, conjugate transposition becomes simple transposition and the formula for the projection matrix becomes[eq29]which might be familiar to those of us that have previously dealt with linear regressions and the OLS estimator.

Orthonormal projection

When the columns of the matrix $B_{R}$ are orthonormal, we have a further simplification: [eq30] and [eq31]

Denote by [eq32] the columns of $B_{R}$.

Then, for any $sin S$, we have[eq33]which is the formula for projections on orthonormal sets that we have already encountered in the lectures on the Gram-Schmidt process and on the QR decomposition.

How to cite

Please cite as:

Taboga, Marco (2021). "Orthogonal projection", Lectures on matrix algebra. https://www.statlect.com/matrix-algebra/orthogonal-projection.

The books

Most of the learning materials found on this website are now available in a traditional textbook format.

Featured pages
Explore
Main sections
About
Glossary entries
Share
  • To enhance your privacy,
  • we removed the social buttons,
  • but don't forget to share.