Search for probability and statistics terms on Statlect

Normal equations

by , PhD

In linear regression analysis, the normal equations are a system of equations whose solution is the Ordinary Least Squares (OLS) estimator of the regression coefficients.

The normal equations are derived from the first-order condition of the Least Squares minimization problem.

Table of Contents

Simple linear regression

Let us start from the simple linear regression model[eq1]where:

There are $N,$ observations in the sample: $i=1,ldots ,N$.

Normal equations in the simple regression model

The normal equations for the simple regression model are:[eq2]where $widehat{lpha }$ and $widehat{eta }$ (the two unknowns) are the estimators of $lpha $ and $eta $.


The OLS estimators of $lpha $ and $eta $, denoted by $widehat{lpha }$ and $widehat{eta }$, are derived by minimizing the sum of squared residuals:[eq3]We carry out the minimization by computing the first-order conditions for a minimum. In other words, we calculate the derivatives of $SSR$ with respect to $widehat{lpha }$ and $widehat{eta }$, and we set them equal to zero:[eq4]We divide the two equations by $2$ and obtain the equivalent system[eq5]Since[eq6]we can write[eq7]which are the two normal equations displayed above.

Thus, in the case of a simple linear regression, the normal equations are a system of two equations in two unknowns ($widehat{lpha }$ and $widehat{eta }$).

If the system has a unique solution, then the two values of $widehat{lpha }$ and $widehat{eta }$ that solve the system are the OLS estimators of the intercept $lpha $ and the slope $eta $ respectively.

Multiple linear regression

In a multiple linear regression, in which there is more than one regressor, the regression equation can be written in matrix form:[eq8]where:

Normal equations in the multiple regression model

The normal equations for the multiple regression model are expressed in matrix form as[eq9]where the unknown $widehat{eta }$ is a Kx1 vector (the estimator of $eta $).


The OLS estimator of the vector $eta $, denoted by $widehat{eta }$, is derived by minimizing the sum of squared residuals, which can be written in matrix form as follows:[eq10]In order to find a minimizer, we compute the first-order condition for a minimum. We calculate the gradient of $SSR$ (the vector of partial derivatives with respect to the entries of $widehat{eta }$) and we set it equal to zero:[eq11]We divide the equations by $2$ and obtain[eq12]which is a system of normal equations expressed in matrix form.

Thus, in the case of the multiple regression model, the normal equations, expressed above in matrix form, are a system of K equations in K unknowns (the K entries of the coefficient vector $eta $).

If the system has a unique solution, the value of $widehat{eta }$ that solves the system is the OLS estimator of the vector $eta $.

How to solve the normal equations

As stated above, the normal equations are just a system of K linear equations in K unknowns.

Therefore, we can employ the standard methods for solving linear systems.

For example, if the equations are expressed in matrix form and the matrix $X^{	op }X$ is invertible, we can write the solution as[eq13]

More details

More mathematical details about the normal equations and the OLS estimator can be found in these lectures:


If you want to double check the formulae and the derivations shown above, you can check these references:

Keep reading the glossary

Previous entry: Multinomial coefficient

Next entry: Null hypothesis

How to cite

Please cite as:

Taboga, Marco (2021). "Normal equations", Lectures on probability theory and mathematical statistics. Kindle Direct Publishing. Online appendix.

The books

Most of the learning materials found on this website are now available in a traditional textbook format.