Search for probability and statistics terms on Statlect
StatLect
Index > Probability distributions > Normal distribution

Normal distribution - Quadratic forms

by , PhD

This lecture presents some important results about quadratic forms involving normal random vectors, that is, about forms of the kind [eq1]where X is a Kx1 multivariate normal random vector, A is a $K imes K$ matrix and $ op $ denotes transposition.

Table of Contents

Review of relevant results from matrix algebra

Before discussing quadratic forms involving normal random vectors, we review some results from matrix algebra that are used throughout the lecture.

Orthogonal matrices

A $K imes K$ real matrix A is orthogonal if[eq2]which also implies[eq3]where I is the identity matrix. Of course, if A is orthogonal also $A^{ op }$ is orthogonal.

An important property of orthogonal matrices is the following.

Proposition Let X be a Kx1 standard multivariate normal random vector, i.e., [eq4]. Let A be an orthogonal $K imes K$ real matrix. Define [eq5]Then also Y has a standard multivariate normal distribution, i.e., [eq6].

Proof

The random vector Y has a multivariate normal distribution because it is a linear transformation of another multivariate normal random vector (see the lecture entitled Linear combinations of normal random variables). Y is standard normal because its expected value is[eq7]and its covariance matrix is[eq8]where the last equality is an immediate consequence of the definition of orthogonal matrix.

Symmetric matrices

A $K imes K$ real matrix A is symmetric if[eq9]i.e., A equals its transpose.

Real symmetric matrices have the property that they can be decomposed as[eq10]where $P$ is an orthogonal matrix and $D$ is a diagonal matrix (i.e., a matrix whose off-diagonal entries are zero). The diagonal elements of $D$, which are all real, are the eigenvalues of A and the columns of $P$ are the eigenvectors of A.

Idempotent matrices

A $K imes K$ real matrix A is idempotent if[eq11]which implies[eq12]for any $nin U{2115} $.

Symmetric idempotent matrices

If a matrix A is both symmetric and idempotent then its eigenvalues are either zero or one. In other words, the diagonal entries of the diagonal matrix $D$ in the decomposition[eq13] are either zero or one.

Proof

This can be easily seen as follows:[eq14]which implies[eq15]But this is possible only if the diagonal entries of $D$ are either zero or one.

Trace of a matrix

Let A be a $K imes K$ real matrix and denote by $A_{ij}$ the $left( i,j ight) $-th entry of A (i.e., the entry at the intersection of the i-th row and the $j$-th column). The trace of A, denoted by [eq16], is[eq17]

In other words, the trace is equal to the sum of all the diagonal entries of A.

The trace of A enjoys the following important property: [eq18]where [eq19] are the K eigenvalues of A.

Quadratic forms in standard multivariate normal random vectors

The following proposition shows that certain quadratic forms in standard normal random vectors have a Chi-square distribution.

Proposition Let X be a Kx1 standard multivariate normal random vector, i.e. [eq20]. Let A be a symmetric and idempotent matrix. Let [eq21] be the trace of A. Define[eq22]Then $Q$ has a Chi-square distribution with [eq23] degrees of freedom.

Proof

Since A is symmetric, it can be decomposed as[eq24]where $P$ is orthogonal and $D$ is diagonal. The quadratic form can be written as[eq25]where we have defined[eq26]By the above theorem on orthogonal transformations of standard multivariate normal random vectors, the orthogonality of $P^{ op }$ implies that [eq27]. Since $D$ is diagonal, we can write the quadratic form as[eq28]where $Y_{j}$ is the $j$-th component of Y and $D_{jj}$ is the $j$-th diagonal entry of $D$. Since A is symmetric and idempotent, the diagonal entries of $D$ are either zero or one. Denote by $J$ the set[eq29]and by $r$ its cardinality, i.e. the number of diagonal entries of $D$ that are equal to 1. Since [eq30], we can write[eq31]But the components of a standard normal random vector are mutually independent standard normal random variables. Therefore, $Q$ is the sum of the squares of $r$ independent standard normal random variables. Hence, it has a Chi-square distribution with $r$ degrees of freedom (see the lecture entitled Chi-square distribution for details). Finally, by the properties of idempotent matrices and of the trace of a matrix (see above), $r$ is not only the sum of the number of diagonal entries of $D$ that are equal to 1, but it is also the sum of the eigenvalues of A. Since the trace of a matrix is equal to the sum of its eigenvalues, then [eq32].

The proposition above can be used to derive the following extremely useful proposition.

Proposition Let X be a Kx1 multivariate normal random vector with mean mu and invertible covariance matrix V. Define[eq33]Then $Q$ has a Chi-square distribution with K degrees of freedom.

Proof

Since V is invertible, there exists an invertible matrix Sigma such that[eq34]Therefore, we have[eq35]where we have defined[eq36]Being a linear transformation of a multivariate normal random vector, the vector Z has a multivariate normal distribution. Its mean is[eq37]and its covariance matrix is[eq38]Thus, Z has a standard multivariate normal distribution (mean 0 and variance I) and[eq39]is a quadratic form in a standard normal random vector. As a consequence, it has a Chi-square distribution with [eq40] degrees of freedom.

Independence of quadratic forms in standard multivariate normal random vectors

We start this section with a proposition on the independence between linear transformations.

Proposition Let X be a Kx1 standard multivariate normal random vector, i.e., [eq41]. Let A be a $L_{A} imes K$ matrix and $B$ be a $L_{B} imes K$ matrix. Define[eq42]Then $T_{1}$ and $T_{2}$ are two independent random vectors if and only if $AB^{intercal }=0$.

Proof

First of all, note that $T_{1}$ and $T_{2}$ are linear transformations of the same multivariate normal random vector X. Therefore, they are jointly normal (see the lecture entitled Linear combinations of normal random variables). Their cross-covariance is[eq43]But, as we explained in the lecture entitled Multivariate normal distribution - Partitioned vectors, two jointly normal random vectors are independent if and only if their cross-covariance is equal to 0. In our case, the cross-covariance is equal to zero if and only if $AB^{intercal }=0$, which proves the proposition.

The following proposition gives a necessary and sufficient condition for the independence of two quadratic forms in the same standard multivariate normal random vector.

Proposition Let X be a Kx1 standard multivariate normal random vector, i.e., [eq44]. Let A and $B$ be two $K imes K$ symmetric and idempotent matrices. Define[eq45]Then $Q_{1}$ and $Q_{2}$ are two independent random variables if and only if $AB=0$.

Proof

Since A and $B$ are symmetric and idempotent, we can write[eq46]from which it is apparent that $Q_{1}$ and $Q_{2}$ can be independent only as long as $AX$ and $BX$ are independent. But, by the above proposition on the independence between linear transformations of jointly normal random vectors, $AX$ and $BX$ are independent if and only if $AB^{intercal }=0$. Since $B$ is symmetric, this is the same as $AB=0$.

The following proposition gives a necessary and sufficient condition for the independence between a quadratic form and a linear transformation involving the same standard multivariate normal random vector.

Proposition Let X be a Kx1 standard multivariate normal random vector, i.e., [eq47]. Let A be a $L imes K$ vector and $B$ a $K imes K$ symmetric and idempotent matrix. Define[eq48]Then $T$ and $Q$ are independent if and only if $AB=0$.

Proof

Since $B$ is symmetric and idempotent, we can write[eq49]from which it is apparent that $T$ and $Q$ can be independent only as long as $AX$ and $BX$ are independent. But, by the above proposition on the independence between linear transformations of jointly normal random vectors, $AX$ and $AB$ are independent if and only if $AB^{intercal }=0$. Since $B$ is symmetric, this is the same as $AB=0$.

Examples

We discuss here some quadratic forms that are commonly found in statistics.

Sample variance as a quadratic form

Let X_1, ..., X_n be n independent random variables, all having a normal distribution with mean mu and variance $sigma ^{2} $. Let their sample mean Xbar_n be defined as[eq50]

and their adjusted sample variance be defined as[eq51]

Define the following matrix:[eq52]where I is the n-dimensional identity matrix and $imath $ is a $n imes 1$ vector of ones. In other words, $M$ has the following structure:[eq53]

$M$ is a symmetric matrix. By computing the product $Mcdot M$, it can also be easily verified that $M$ is idempotent.

Denote by X the $n imes 1$ random vector whose i-th entry is equal to X_i and note that X has a multivariate normal distribution with mean $mu imath $ and covariance matrix $sigma ^{2}I$ (see the lecture entitled Multivariate normal distribution).

The matrix $M$ can be used to write the sample variance as[eq54]

Now define a new random vector[eq55]and note that Z has a standard (mean zero and covariance I) multivariate normal distribution (see the lecture entitled Linear combinations of normal random variables).

The sample variance can be written as[eq56]

The last three terms in the sum are equal to zero because[eq57]which can be verified by directly performing the multiplication of $M$ and $imath $.

Therefore, the sample variance[eq58]is proportional to a quadratic form in a standard normal random vector ($Z^{ op }MZ$) and the quadratic form is obtained from a symmetric and idempotent matrix ($M$). Thanks to the propositions above, we know that the quadratic form $Z^{ op }MZ$ has a Chi-square distribution with [eq59] degrees of freedom, where [eq60] is the trace of $M$. But the trace of $M$ is[eq61]

So, the quadratic form $Z^{ op }MZ$ has a Chi-square distribution with $n-1$ degrees of freedom. Multiplying a Chi-square random variable with $n-1$ degrees of freedom by [eq62] one obtains a Gamma random variable with parameters $n-1$ and sigma^2 (see the lecture entitled Gamma distribution for more details).

So, summing up, the adjusted sample variance $s^{2}$ has a Gamma distribution with parameters $n-1$ and sigma^2.

Furthermore, the adjusted sample variance $s^{2}$ is independent of the sample mean Xbar_n, which is proved as follows. The sample mean can be written as[eq63]and the sample variance can be written as[eq64]If we use the above proposition (independence between a linear transformation and a quadratic form), verifying the independence of Xbar_n and $s^{2}$ boils down to verifying that [eq65]which can be easily checked by directly performing the multiplication of $imath ^{ op }$ and $M$.

How to cite

Please cite as:

Taboga, Marco (2021). "Normal distribution - Quadratic forms", Lectures on probability theory and mathematical statistics. Kindle Direct Publishing. Online appendix. https://www.statlect.com/probability-distributions/normal-distribution-quadratic-forms.

The books

Most of the learning materials found on this website are now available in a traditional textbook format.

Featured pages
Explore
Main sections
About
Glossary entries
Share
  • To enhance your privacy,
  • we removed the social buttons,
  • but don't forget to share.