Index > Fundamentals of statistics > Maximum likelihood

Maximum likelihood - Hypothesis testing

This lecture discusses how to perform tests of hypotheses on parameters that have been estimated by Maximum likelihood.

Table of contents

Testing framework
Three popular tests
Assumptions
Examples of restrictions being tested
Wald test
Score test
Likelihood ratio test

Testing framework

We are going to assume that an unknown parameter $heta _{0}$ has been estimated by maximum likelihood methods, that the parameter belongs to a parameter space , and that we want to test the null hypothesiswhere $Theta _{R}$ is a proper subset of , i.e.,

Three popular tests

There are three popular methods to carry out tests of this kind of restriction:

Wald test. An estimator of $heta _{0}$ is obtained by maximizing the log-likelihood over the whole parameter space :where is the likelihood function and $xi _{n}$ is the sample. Subsequently, a test statistic is constructed by measuring how far is from satisfying the null hypothesis.
Score test. An estimator of $heta _{0}$ is obtained by maximizing the log-likelihood over the restricted parameter space $Theta _{R}$ :Subsequently, a test statistic is constructed by comparing the vector of derivatives of the log-likelihood at (the so called score) with its expected value under the null hypothesis.
Likelihood ratio test. This test is based on two different estimators of $heta _{0}$ . One, denoted by , is obtained by maximizing the log-likelihood over the whole parameter space , as in the Wald test. The other, denoted by , is obtained by maximizing the log-likelihood over the restricted parameter space $Theta _{R}$ , as in the score test. Finally, a test statistic is constructed by comparing the log-likelihood of to that of .

Thus, the parameter estimate used to carry out the test is the main aspect in which these tests differ among each other: an unrestricted estimate for the Wald test, a restricted estimate for the score test, and both estimates for the likelihood ratio test.

The following sections will cover the main aspects of these tests and will refer the reader to other sections that contain more detailed explanations.

Assumptions

In the remainder of this lecture it will be assumed that the sample $xi _{n}$ is a sample of observations from an IID sequence and that the log-likelihood function satisfies all the conditions used in previous lectures (see Maximum likelihood) to derive the asymptotic distribution of the maximum likelihood estimator.

It will also be assumed that the restriction being tested can be written aswhere is a vector valued function, , all the entries of are continuously differentiable with respect to its arguments, and the Jacobian , i.e., the matrix of partial derivatives of the entries of with respect to the entries of , has rank .

Examples of restrictions being tested

As the above definition of a restriction may seem a little bit abstract, we provide some examples below.

Example Let the parameter space be the set of all -dimensional vectors, i.e., . Suppose we want to test the restriction $heta _{0,2}=1$ , where $heta _{0,2}$ denotes the second component of $heta _{0}$ . Then, the function is a function defined byIn this case, . The Jacobian of is [eq22] which has obviously rank .

Example Let the parameter space be the set of all -dimensional vectors, i.e., . Suppose we want to test the restrictions [eq24] where $heta _{0,j}$ denotes the -th component of $heta _{0}$ . Then, the function is a function defined by [eq27] In this case, . The Jacobian of is [eq28] which has rank because its two rows are linearly independent.

Wald test

Let be the estimate of a parameter $heta _{0}$ , obtained by maximizing the log-likelihood over the whole parameter space :The Wald test is based on the following test statistic: [eq31] where is the sample size and $widehat{V}_{n}$ is a consistent estimate of the asymptotic covariance matrix of (see the lecture entitled Maximum likelihood - Covariance matrix estimation).

Under the null hypothesis, that is, under the hypothesis that , the Wald statistic $W_{n}$ converges in distribution to a Chi-square distribution with degrees of freedom.

The test is performed by fixing a critical value and by rejecting the null hypothesis if

The size of the test can be approximated by its asymptotic value

where is the distribution function of a Chi-square random variable with degrees of freedom. We can choose so as to achieve a pre-determined size, as follows:

More details about the Wald test, including a detailed derivation of its asymptotic distribution, can be found in the lecture entitled Wald test.

Score test

Let be the estimate of a parameter $heta _{0}$ , obtained by maximizing the log-likelihood over the restricted parameter space $Theta _{R}$ :The score test (also called Lagrange multiplier test) is based on the following test statistic:where is the sample size, $widehat{V}_{n}$ is a consistent estimate of the asymptotic covariance matrix of , and is the score, that is, the gradient of the log-likelihood function.

Under the null hypothesis that , the score statistic $LM_{n}$ converges in distribution to a Chi-square distribution with degrees of freedom.

Once the test statistic has been computed, the test is carried out following the same procedure described above for the Wald test.

More details about the score test, including a detailed derivation of its asymptotic distribution, can be found in the lecture entitled score test.

Likelihood ratio test

Let be the unrestricted estimate of a parameter $heta _{0}$ obtained by solvingand the restricted estimate obtained by solvingThe likelihood ratio test is based on the following test statistic:In other words, the test statistic is equal to two times the difference between the log-likelihood corresponding to the unrestricted estimate and the log-likelihood corresponding to the restricted estimate .

Under the null hypothesis that , the score statistic $LR_{n}$ converges in distribution to a Chi-square distribution with degrees of freedom.

Once the test statistic has been computed, the test is carried out following the same procedure described above for the Wald test.

More details about the likelihood ratio test, including a detailed derivation of its asymptotic distribution, can be found in the lecture entitled likelihood ratio test.

How to cite

Please cite as:

Taboga, Marco (2021). "Maximum likelihood - Hypothesis testing", Lectures on probability theory and mathematical statistics. Kindle Direct Publishing. Online appendix. https://www.statlect.com/fundamentals-of-statistics/maximum-likelihood-hypothesis-testing.