Search for probability and statistics terms on Statlect
StatLect
Index > Matrix algebra

Inner product

by , PhD

The inner product between two vectors is an abstract concept used to derive some of the most useful results in linear algebra, as well as nice solutions to several difficult practical problems. It is unfortunately a pretty unintuitive concept, although in certain cases we can interpret it as a measure of the similarity between two vectors.

Table of Contents

Vector space over a field

Before giving a definition of inner product, we need to remember a couple of important facts about vector spaces.

When we use the term "vector" we often refer to an array of numbers, and when we say "vector space" we refer to a set of such arrays. However, if you revise the lecture on vector spaces, you will see that we also gave an abstract axiomatic definition: a vector space is a set equipped with two operations, called vector addition and scalar multiplication, that satisfy a number of axioms; the elements of the vector space are called vectors. In that abstract definition, a vector space has an associated field, which in most cases is the set of real numbers R or the set of complex numbers $U{2102} $. The elements of the field are the so-called "scalars", which are used in the multiplication of vectors by scalars (e.g., to build linear combinations of vectors).

When we develop the concept of inner product, we will need to specify the field over which the vector space is defined. Moreover, we will always restrict our attention to the two fields R and $U{2102} $.

Definition of inner product

We are now ready to provide a definition.

Definition Let $S$ be a vector space over $U{2102} $. An inner product on $S$ is a function [eq1] that associates to each ordered pair of vectors $r,sin S$ a complex number, denoted by [eq2], which has the following properties.

  1. Positivity:[eq3]where [eq4] means that [eq5] is real (i.e., its complex part is zero) and positive.

  2. Definiteness:[eq6]

  3. Additivity in first argument:[eq7]

  4. Homogeneity in first argument:[eq8]

  5. Conjugate symmetry:[eq9]where [eq10] denotes the complex conjugate of [eq11].

Although this definition concerns only vector spaces over the complex field $U{2102} $, we will use it to develop a theory that applies also to vector spaces defined over the field of real numbers. In fact, when $S$ is a vector space over R, we just need to replace $U{2102} $ with R in the definition above and pretend that complex conjugation is an operation that leaves the elements of R unchanged, so that property 5) becomes[eq12]

Orthogonal vectors

When the inner product between two vectors is equal to zero, that is,[eq13]then the two vectors are said to be orthogonal.

Example: the dot product of two real arrays

One of the most important examples of inner product is the dot product between two Kx1 column vectors having real entries.

Let $S$ be the space of all Kx1 real vectors (on the real field R).

The dot product between two real Kx1 vectors $r,sin S$ (which has already been introduced in the lecture on matrix multiplication) is[eq14]where $s^{intercal }$ is the transpose of $s$, [eq15] are the K entries of $r$ and [eq16] are the K entries of $s$.

Example Define[eq17]Then[eq18]

We need to verify that the dot product thus defined satisfies the five properties of an inner product.

Positivity and definiteness are satisfied because [eq19]where the equality holds if and only if $r=0$.

Additivity is satisfied because[eq20]

The dot product is homogeneous in the first argument because[eq21]

Finally, (conjugate) symmetry holds because[eq22]

Example: the inner product of two complex arrays

Another important example of inner product is that between two Kx1 column vectors having complex entries.

Let $S$ be the space of all Kx1 complex vectors (on the complex field $U{2102} $).

The inner product between two Kx1 vectors $r,sin S$ is defined to be[eq23]where $s^{st }$ is the conjugate transpose of $s$, [eq24] are the K entries of $r$ and [eq25] are the complex conjugates of the K entries of $s$.

Example Define[eq26]Then[eq27]

Let us check that the five properties of an inner product are satisfied.

Positivity and definiteness are satisfied because [eq28]where [eq29] is the modulus of $r_{k}$ and the equality holds if and only if $r=0$.

Additivity is satisfied because[eq30]

The dot product is homogeneous in the first argument because[eq31]

Finally, conjugate symmetry holds because[eq32]

Further properties of the inner product

We now present further properties of the inner product that can be derived from its five defining properties introduced above.

Additivity in the second argument

We have that the inner product is additive in the second argument:[eq33]

Proof

This is proved as follows:[eq34]where: in steps $ rame{A}$ and $ rame{C}$ we have used the conjugate symmetry of the inner product; in step $ rame{B}$ we have used the additivity in the first argument.

Conjugate homogeneity in the second argument

While the inner product is homogenous in the first argument, it is conjugate homogeneous in the second one:[eq35]

Proof

Here is a demonstration:[eq36]where: in steps $ rame{A}$ and $ rame{C}$ we have used the conjugate symmetry of the inner product; in step $ rame{B}$ we have used the homogeneity in the first argument.

Solved exercises

Below you can find some exercises with explained solutions.

Exercise 1

Let $S$ be a vector space, $r,sin S$ and [eq37] an inner product on $S$. Suppose that[eq38]Compute[eq39]under the assumption that $r$ and $s$ are orthogonal.

Solution

We can compute the given inner product as follows:[eq40]where: in step $ rame{A}$ we have used the linearity in the first argument; in step $ rame{B}$ we have used the orthogonality of $s$ and $r$, which implies that[eq13]

Exercise 2

Let[eq42]

Compute[eq43]using the inner product of complex arrays defined above.

Solution

We have that:[eq44]

How to cite

Please cite as:

Taboga, Marco (2021). "Inner product", Lectures on matrix algebra. https://www.statlect.com/matrix-algebra/inner-product.

The books

Most of the learning materials found on this website are now available in a traditional textbook format.

Featured pages
Explore
Main sections
About
Glossary entries
Share
  • To enhance your privacy,
  • we removed the social buttons,
  • but don't forget to share.