Search for probability and statistics terms on Statlect
StatLect
Index > Fundamentals of probability

Covariance matrix

by , PhD

The covariance matrix of a random vector is a square matrix that contains all the covariances between the entries of the vector.

Table of Contents

Definition

Here is a definition.

Definition Let X be a Kx1 random vector. The covariance matrix of X, or variance-covariance matrix of X, denoted by [eq1], is defined as follows:[eq2]provided the above expected values exist and are well-defined.

It is a multivariate generalization of the definition of variance for a scalar random variable Y:[eq3]

Structure

Let X_1, ..., $X_{K}$ denote the K components of the vector X.

From the definition of [eq4], it can easily be seen that [eq5] is a $K imes K$ matrix with the following structure:[eq6]

Therefore, the covariance matrix of X is a square $K imes K$ matrix whose generic $left( i,j ight) $-th entry is equal to the covariance between X_i and $X_{j}$.

Since [eq7] when $i=j$, the diagonal entries of the covariance matrix are equal to the variances of the individual entries of X.

Example

Here is an example.

Suppose that X is a $2 imes 1$ random vector with components X_1 and X_2.

Let[eq8]

By the symmetry of covariance, it must also be [eq9]

Therefore, the covariance matrix of X is[eq10]

Formula for computing the covariance matrix

The covariance matrix of a Kx1 random vector X can be computed using the formula[eq11]

Proof

The above formula can be derived as follows:[eq12]

This formula also makes clear that the covariance matrix exists and is well-defined only as long as the vector of expected values [eq13] and the matrix of second cross-moments [eq14] exist and are well-defined.

More details

The following subsections contain more details about the covariance matrix.

Addition to constant vectors

Let a be a constant Kx1 vector and let X be a Kx1 random vector. Then,[eq15]

Proof

This is a consequence of the fact that [eq16] (by linearity of the expected value):[eq17]

Multiplication by constant matrices

Let $b$ be a constant $M imes K$ matrix and let X be a Kx1 random vector. Then,[eq18]

Proof

This is easily proved using the fact that [eq19] (by linearity of the expected value):[eq20]

Linear transformations

Let a be a constant $M imes 1$ vector, $b$ be a constant $M imes K$ matrix and X a Kx1 random vector. Then, by combining the two properties above, we obtain[eq21]

Symmetry

The covariance matrix [eq5] is a symmetric matrix, that is, it is equal to its transpose:[eq23]

Proof

The proof is as follows:[eq24]

Semi-positive definiteness

The covariance matrix [eq5] is a positive-semidefinite matrix, that is, for any $1 imes K$ vector a:[eq26]

Proof

This is easily proved by using the Multiplication by constant matrices property above:[eq27]where the last inequality follows from the fact that variance is always positive.

Covariance between linear transformations

Let a and $b$ be two constant $1 imes K$ vectors and X a Kx1 random vector. Then, the covariance between the two linear transformations $aX$ and $bX$ can be expressed as a function of the covariance matrix:[eq28]

Proof

This can be proved as follows:[eq29]

Cross-covariance

The term covariance matrix is sometimes also used to refer to the matrix of covariances between the elements of two vectors.

Let X be a Kx1 random vector and Y be a $L imes 1$ random vector.

The covariance matrix between X and Y, or cross-covariance between X and Y is denoted by [eq30].

It is defined as follows:[eq31]provided the above expected values exist and are well-defined.

It is a multivariate generalization of the definition of covariance between two scalar random variables.

Let X_1, ..., $X_{K}$ denote the K components of the vector X and $Y_{1}$, ..., $Y_{L}$ denote the $L$ components of the vector Y .

From the definition of [eq32], it can easily be seen that [eq33] is a $K imes L$ matrix with the following structure:[eq34]

Note that [eq33] is not the same as [eq36]. In fact, [eq37] is a $L imes K$ matrix equal to the transpose of [eq33]:[eq39]

Solved exercises

Below you can find some exercises with explained solutions.

Exercise 1

Let X be a $2 imes 1$ random vector and denote its components by X_1 and X_2.

The covariance matrix of X is[eq40]

Compute the variance of the random variable Y defined as[eq41]

Solution

By using a matrix notation, Y can be written as[eq42]where we have defined[eq43]Therefore, the variance of Y can be computed by using the formula for the covariance matrix of a linear transformation:[eq44]

Exercise 2

Let X be a $3 imes 1$ random vector and denote its components by X_1, X_2 and $X_{3}$.

The covariance matrix of X is[eq45]

Compute the following covariance:[eq46]

Solution

Using the bilinearity of the covariance operator, we obtain[eq47]The same result can be obtained by using the formula for the covariance between two linear transformations. Defining[eq48]we have[eq49]

Exercise 3

Let X be a Kx1 random vector whose covariance matrix is equal to the identity matrix:[eq50]

Define a new random vector Y as follows:[eq51]where A is a $K imes K$ matrix of constants such that[eq52]

Derive the covariance matrix of Y.

Solution

By the formula for the covariance matrix of a linear transformation, we have[eq53]

How to cite

Please cite as:

Taboga, Marco (2021). "Covariance matrix", Lectures on probability theory and mathematical statistics. Kindle Direct Publishing. Online appendix. https://www.statlect.com/fundamentals-of-probability/covariance-matrix.

The books

Most of the learning materials found on this website are now available in a traditional textbook format.

Featured pages
Explore
Main sections
About
Glossary entries
Share
  • To enhance your privacy,
  • we removed the social buttons,
  • but don't forget to share.