This lecture discusses how to perform tests of hypotheses on parameters that have been estimated by Maximum likelihood.
We are going to assume that an unknown parameter has been estimated by maximum likelihood methods, that the parameter belongs to a parameter space , and that we want to test the null hypothesiswhere is a proper subset of , i.e.,
There are three popular methods to carry out tests of this kind of restriction:
Wald test. An estimator of is obtained by maximizing the log-likelihood over the whole parameter space :where is the likelihood function and is the sample. Subsequently, a test statistic is constructed by measuring how far is from satisfying the null hypothesis.
Score test. An estimator of is obtained by maximizing the log-likelihood over the restricted parameter space :Subsequently, a test statistic is constructed by comparing the vector of derivatives of the log-likelihood at (the so called score) with its expected value under the null hypothesis.
Likelihood ratio test. This test is based on two different estimators of . One, denoted by , is obtained by maximizing the log-likelihood over the whole parameter space , as in the Wald test. The other, denoted by , is obtained by maximizing the log-likelihood over the restricted parameter space , as in the score test. Finally, a test statistic is constructed by comparing the log-likelihood of to that of .
Thus, the parameter estimate used to carry out the test is the main aspect in which these tests differ among each other: an unrestricted estimate for the Wald test, a restricted estimate for the score test, and both estimates for the likelihood ratio test.
The following sections will cover the main aspects of these tests and will refer the reader to other sections that contain more detailed explanations.
In the remainder of this lecture it will be assumed that the sample is a sample of observations from an IID sequence and that the log-likelihood function satisfies all the conditions used in previous lectures (see Maximum likelihood) to derive the asymptotic distribution of the maximum likelihood estimator.
It will also be assumed that the restriction being tested can be written aswhere is a vector valued function, , all the entries of are continuously differentiable with respect to its arguments, and the Jacobian , i.e., the matrix of partial derivatives of the entries of with respect to the entries of , has rank .
As the above definition of a restriction may seem a little bit abstract, we provide some examples below.
Example Let the parameter space be the set of all -dimensional vectors, i.e., . Suppose we want to test the restriction , where denotes the second component of . Then, the function is a function defined byIn this case, . The Jacobian of iswhich has obviously rank .
Example Let the parameter space be the set of all -dimensional vectors, i.e., . Suppose we want to test the restrictions where denotes the -th component of . Then, the function is a function defined byIn this case, . The Jacobian of iswhich has rank because its two rows are linearly independent.
Let be the estimate of a parameter , obtained by maximizing the log-likelihood over the whole parameter space :The Wald test is based on the following test statistic:where is the sample size and is a consistent estimate of the asymptotic covariance matrix of (see the lecture entitled Maximum likelihood - Covariance matrix estimation).
Under the null hypothesis, that is, under the hypothesis that , the Wald statistic converges in distribution to a Chi-square distribution with degrees of freedom.
The test is performed by fixing a critical value and by rejecting the null hypothesis if
The size of the test can be approximated by its asymptotic value
where is the distribution function of a Chi-square random variable with degrees of freedom. We can choose so as to achieve a pre-determined size, as follows:
More details about the Wald test, including a detailed derivation of its asymptotic distribution, can be found in the lecture entitled Wald test.
Let be the estimate of a parameter , obtained by maximizing the log-likelihood over the restricted parameter space :The score test (also called Lagrange multiplier test) is based on the following test statistic:where is the sample size, is a consistent estimate of the asymptotic covariance matrix of , and is the score, that is, the gradient of the log-likelihood function.
Under the null hypothesis that , the score statistic converges in distribution to a Chi-square distribution with degrees of freedom.
Once the test statistic has been computed, the test is carried out following the same procedure described above for the Wald test.
More details about the score test, including a detailed derivation of its asymptotic distribution, can be found in the lecture entitled score test.
Let be the unrestricted estimate of a parameter obtained by solvingand the restricted estimate obtained by solvingThe likelihood ratio test is based on the following test statistic:In other words, the test statistic is equal to two times the difference between the log-likelihood corresponding to the unrestricted estimate and the log-likelihood corresponding to the restricted estimate .
Under the null hypothesis that , the score statistic converges in distribution to a Chi-square distribution with degrees of freedom.
Once the test statistic has been computed, the test is carried out following the same procedure described above for the Wald test.
More details about the likelihood ratio test, including a detailed derivation of its asymptotic distribution, can be found in the lecture entitled likelihood ratio test.
Most of the learning materials found on this website are now available in a traditional textbook format.