Search for probability and statistics terms on Statlect
StatLect

Gauss Markov theorem

by , PhD

The Gauss Markov theorem says that, under certain conditions, the ordinary least squares (OLS) estimator of the coefficients of a linear regression model is the best linear unbiased estimator (BLUE), that is, the estimator that has the smallest variance among those that are unbiased and linear in the observed output variables.

Table of Contents

Assumptions

The regression model is[eq1]where:

The OLS estimator of $eta $ is [eq2]

We assume that:

  1. X has full-rank (as a consequence, $X^{	op }X$ is invertible, and $widehat{eta }$ is well-defined);

  2. [eq3];

  3. [eq4], where I is the $N	imes N$ identity matrix and sigma^2 is a positive constant.

OLS is linear and unbiased

First of all, note that $widehat{eta }$ is linear in $y$. In fact, $widehat{eta }$ is the product between the $K	imes N$ matrix [eq5] and $y$, and matrix multiplication is a linear operation.

It can easily be proved that $widehat{eta }$ is unbiased, both conditional on X, and unconditionally, that is,[eq6]

Proof

We can use the definition of $y$ to re-write the OLS estimator as follows:[eq7]When we condition on X, we can treat X as a constant matrix. Therefore, the conditional expectation of $widehat{eta }$ is [eq8]The Law of Iterated Expectations implies that[eq9]

What it means to be best

Now that we have shown that the OLS estimator is linear and unbiased, we need to prove that it is also the best linear unbiased estimator.

What exactly do we mean by best?

When $widehat{eta }$ is a scalar (i.e., there is only one regressor), we consider $widehat{eta }$ to be the best among those we are considering (i.e., among all the linear unbiased estimators) if and only if it has the smallest possible variance, that is, if its deviations from the true value $eta $ tend to be the smallest on average. Thus, $widehat{eta }$ is the best linear unbiased estimator (BLUE) if and only if [eq10]for any other linear unbiased estimator $widetilde{eta }$.

Since we often deal with more than one regressor, we have to extend this definition to a multivariate context. We do this by requiring that[eq11]for any $1	imes K$ constant vector a, any other linear unbiased estimator $widetilde{eta }$.

In other words, OLS is BLUE if and only if any linear combination of the regression coefficients is estimated more precisely by OLS than by any other linear unbiased estimator.

Condition (1) is satisfied if and only if [eq12]is a positive semi-definite matrix.

Proof

We can write condition (1) as[eq13]or [eq14]But the latter inequality is true if and only if [eq15] is positive-semidefinite (by the very definition of positive-semidefinite matrix).

In the next two sections we will derive [eq16] (the covariance matrix of the OLS estimator), and then we will prove that (2) is positive-semidefinite, so that OLS is BLUE.

The covariance matrix of the OLS estimator

The conditional covariance matrix of the OLS estimator is[eq17]

Proof

We have already proved (see above) that the OLS estimator can be written as[eq18]Therefore, its conditional variance is[eq19]

OLS is BLUE

Since we are considering the set of linear estimators, we can write any estimator in this set as[eq20]where $C$ is a $K	imes N$ matrix.

Furthermore, if we define[eq21]then we can write[eq22]

It is possible to prove that $DX=0$ if $widetilde{eta }$ is unbiased.

Proof

We have that[eq23]As a consequence, [eq24] is always equal to $eta $ only if $DX=0$.

By using this result, we can also prove that[eq25]

Proof

The proof is as follows:[eq26]where in steps $rame{A}$, $rame{B}$ and $rame{C}$ we have used the fact that $DX=0$.

As a consequence,[eq27]is positive semi-definite because [eq28] is positive semi-definite. This is true for any unbiased linear estimator $widetilde{eta }$. Therefore, the OLS estimator is BLUE.

How to cite

Please cite as:

Taboga, Marco (2021). "Gauss Markov theorem", Lectures on probability theory and mathematical statistics. Kindle Direct Publishing. Online appendix. https://www.statlect.com/fundamentals-of-statistics/Gauss-Markov-theorem.

The books

Most of the learning materials found on this website are now available in a traditional textbook format.