Search for probability and statistics terms on Statlect
StatLect

Partitioned regression

by , PhD

Partitioned linear regression is a technique used to subdivide the independent variables in two groups and estimate their coefficients in two separate steps.

Partitioned regression is often used to solve problems in which estimating all the regression coefficients together would be too computationally intensive.

Table of Contents

The regression model

Consider the linear regression model in matrix form:[eq1]where:

Two groups of regressors

We divide the regressors in two groups:

  1. group 1 contains the first $K_{1}$ regressors;

  2. group $2$ contains the remaining $K_{2}$;

Obviously $K_{1}+K_{2}=K$.

Partitioned matrices

We use the subdivision into two groups to partition the vectors and matrices that appear in the regression equation:[eq2]

The dimensions of the blocks are as follows:

The OLS estimator

Remember that, when X is full-rank, the OLS estimator of the vector $eta $ can be written as[eq3]

Also the OLS estimator can be partitioned as[eq4]

Normal equations

If we multiply both sides of the OLS formula by [eq5], we obtain the so-called normal equations:[eq6]

In partitioned form, the normal equations become[eq7]

By using the multiplication rule for partitioned matrices, we obtain[eq8]which can be written as two separate equations:[eq9]

These two equations are used to derive most of the results about partitioned regressions.

The main result

Here is the main result about partitioned regressions, proved in this section and explained in the next one:[eq10]

Proof

The first normal equation, derived previously, is[eq11]We write it as[eq12]or[eq13]The second normal equation is[eq14]We substitute the expression for [eq15]:[eq16]which is equivalent to[eq17]or[eq18]We define[eq19]so that[eq20]or[eq21]The matrix $M_{1}$ is idempotent:[eq22]It is also symmetric, as can be easily verified:[eq23]Therefore, we have[eq24]where we have defined[eq25]

The calculations need to be performed in reverse order, starting from the last equation.

Interpretation of the result

Working out the formulae above is equivalent to deriving the OLS estimators in three steps:

  1. we regress $y$ and the columns of X_2 on X_1; the residuals from these regressions are $y^{st }$ and $X_{2}^{st }$;

  2. we find [eq26] by regressing $y^{st }$ on $X_{2}^{st }$;

  3. we calculate [eq15] by regressing the residuals [eq28] on X_1.

Proof

Let us start from the first step. When we regress $y$ on X_1, the OLS estimator of the regression coefficients is[eq29]and the residuals are[eq30]Similarly, when we regress the columns of X_2 on X_1, the OLS estimators of the regression coefficients are the columns of the matrix[eq31]and the vectors of residuals are the columns of the matrix [eq32]In the second step, we regress $y^{st }$ on $X_{2}^{st }$. The OLS estimator of the regression coefficients is [eq33]But we have proved above that this is also the OLS estimator of $eta _{2}$ in our partitioned regression. In the third step, we regress [eq34] on X_1. The OLS estimator of the regression coefficients is[eq13]But we have proved above that this is also the OLS estimator of $eta _{1}$ in our partitioned regression.

Frisch-Waugh-Lovell theorem

The fact that [eq36] can be calculated by regressing $y^{st }$ on $X_{2}^{st }$ is often called Frisch-Waugh-Lovell theorem.

Example: demeaned regression

As an example, we discuss the so-called demeaned regression.

Suppose that the first column of X is a vector of ones (corresponding to the so-called intercept).

We partition the design matrix as[eq37]where $1_{N}$ is the vector of ones and X_2 contains all the other regressors.

Let us see what happens in the 3 steps explained above.

First step

In the first step, we regress $y$ on $1_{N}$.

The OLS estimate of the regression coefficient is[eq38]where $overline{y}$ is the sample mean of $y$.

Therefore,[eq39]

In other words, $y^{st }$ is the demeaned version of $y$.

Similarly, $X_{2}^{st }$ is the demeaned version of X_2:[eq40]where $overline{x_{2}}$ is a $1	imes K$ row vector that contains the sample means of the columns of X_2.

Second step

In the second step, we regress $y^{st }$ on $X_{2}^{st }$ and we obtain as a result [eq41].

The vector [eq42] is equal to the OLS estimator of the regression coefficients of X_2 in the original regression of $y$ on $1_{N}$ and X_2.

Thus, we have an important rule here: running a regression with an intercept is equivalent to demeaning all the variables and running the same regression without the intercept.

Third step

In the third step, we calculate [eq15] (the intercept) by regressing the residuals [eq44] on $1_{N}$.

We already know that regressing a variable on $1_{N}$ is the same as calculating its sample mean.

Therefore, the intercept [eq15] is equal to the sample mean of the residuals [eq44].

How to cite

Please cite as:

Taboga, Marco (2021). "Partitioned regression", Lectures on probability theory and mathematical statistics. Kindle Direct Publishing. Online appendix. https://www.statlect.com/fundamentals-of-statistics/partitioned-regression.

The books

Most of the learning materials found on this website are now available in a traditional textbook format.