The score test, also known as Lagrange Multiplier (LM) test, is a hypothesis test used to check whether some parameter restrictions are violated.
A score test can be performed after estimating the parameters by maximum likelihood (ML).
Table of contents
The score test is used to deal with null hypotheses of the following kind:where:
is an unknown parameter belonging to a parameter space ;
is a vector-valued function ();
is the number of parameters;
is the number of restrictions being tested.
As explained in this introductory lecture, all the most common null hypotheses and parameter restrictions can be written in the form .
Example If has two entries and , and the null hypothesis is , thenThere are parameters and tested restrictions.
The score test is based on the solution of the constrained maximum likelihood problemwhere:
the set contains all the parameters that satisfy the tested restriction;
is the sample of observed data;
is the sample size;
is the likelihood function.
Thus, the parameter estimate satisfies all the tested restrictions.
The test statistic, called score statistic (or Lagrange Multiplier statistic), iswhere:
the column vector is the gradient of the log-likelihood function (called score); in other words, is the vector of partial derivatives of the log-likelihood function with respect to the entries of the parameter vector ;
the matrix is a consistent estimate of the asymptotic covariance matrix of the estimator (see Maximum likelihood - Covariance matrix estimation).
A popular estimator of the asymptotic covariance matrix is the so-called Hessian estimator:whereis the Hessian (i.e., the matrix of second partial derivatives of the log-likelihood with respect to the parameters).
If we plug this estimator in the above formula for the score statistic, we obtain:
Many sources report this formula, but bear in mind that it is only a particular implementation of the LM test. If we use different estimators of the asymptotic covariance matrix, we obtain different formulae.
In order to derive the asymptotic properties of the statistic , the following assumptions will be maintained:
the sample and the likelihood function satisfy some set of conditions that are sufficient to guarantee the consistency and asymptotic normality of (see the lecture on maximum likelihood estimation for a set of such conditions);
for each , the entries of are continuously differentiable with respect to all the entries of ;
the matrix of the partial derivatives of the entries of with respect to the entries of , called the Jacobian of and denoted by , has rank .
The Lagrange Multiplier statistic converges to a Chi-square distribution.
Proposition Provided that some technical conditions are satisfied (see above), and provided that the null hypothesis is true, the statistic converges in distribution to a Chi-square distribution with degrees of freedom.
Denote by the unconstrained maximum likelihood estimate:By the Mean Value Theorem, we have thatwhere is an intermediate point (a vector whose components are strictly comprised between the components of and those of ). Since , we have thatTherefore,Again by the Mean Value Theorem, we have thatwhere is the Hessian matrix (a matrix of second partial derivatives) and is an intermediate point (actually, to be precise, there is a different intermediate point for each row of the Hessian). Because the gradient is zero at an unconstrained maximum, we have thatand, as a consequence,and It descends thatNow, where is a vector of Lagrange multipliers. Thus, we have thatSolving for , we obtainNow, the score statistic can be written asPlugging in the previously derived expression for , the statistic becomeswhereGiven that under the null hypothesis both and converge in probability to , also and converge in probability to , because the entries of and are strictly comprised between the entries of and . Moreover,where is the asymptotic covariance matrix of . We had previously assumed that also converges in probability to . Therefore, by the continuous mapping theorem, we have the following resultsBy putting together everything we have derived so far, we can write the score statistic as a sequence of quadratic forms whereand But in the lecture on the Wald test, we have proved that such a sequence converges in distribution to a Chi-square random variable with a number of degrees of freedom equal to .
In the score test, the null hypothesis is rejected if the score statistic exceeds a pre-determined critical value , that is, if
The size of the test can be approximated by its asymptotic valuewhere is the distribution function of a Chi-square random variable with degrees of freedom.
We can choose so as to achieve a pre-determined size, as follows:
Here is an example of how to perform a Lagrange Multiplier test.
Let the parameter space be the set of all -dimensional vectors, that is, .
Denote the entries of the true parameter by and .
Suppose that we want to test the restriction
In this case, the function is a function defined by
We have that and the Jacobian of iswhose rank is equal to .
Note that the Jacobian does not depend on .
We then maximize the log-likelihood function with respect to (keeping fixed at ).
Suppose that we obtain the following estimates of the parameter and of the asymptotic covariance matrix:where is the sample size.
Suppose also that the value of the score is
Then, the score statistic is
The statistic has a Chi-square distribution with degrees of freedom.
Suppose that we want the size of our test to be .
Then, the critical value iswhere is the cumulative distribution function of a Chi-square random variable with degree of freedom.
The value of
can be calculated with any statistical software (we did it in MATLAB, using
the command chi2inv(0.99,1)
).
Thus, the test statistic exceeds the critical valueand we reject the null hypothesis.
Please cite as:
Taboga, Marco (2021). "Score test", Lectures on probability theory and mathematical statistics. Kindle Direct Publishing. Online appendix. https://www.statlect.com/fundamentals-of-statistics/score-test.
Most of the learning materials found on this website are now available in a traditional textbook format.