Search for probability and statistics terms on Statlect
StatLect
Index > Matrix algebra

Positive definite matrix

by , PhD

A square matrix is positive definite if pre-multiplying and post-multiplying it by the same vector always gives a positive number as a result, independently of how we choose the vector.

Positive definite symmetric matrices have the property that all their eigenvalues are positive.

Table of Contents

Real quadratic forms

We begin by defining quadratic forms. For the time being, we confine our attention to real matrices and real vectors. At the end of this lecture, we discuss the more general complex case.

Definition Let A be a $K imes K$ real matrix. A quadratic form in A is a transformation[eq1] $ $where x is a Kx1 vector and $x^{ op }$ is its transpose.

The transformation $x^{ op }Ax$ is a scalar because $x^{ op }A$ is a $1 imes K$ row vector and its product with the Kx1 column vector x gives a scalar as a result.

Example Define[eq2]Given a $2 imes 1$ vector x, the quadratic form defined by the matrix A is[eq3]

Restricting attention to symmetric matrices

When we study quadratic forms, we can confine our attention to symmetric matrices without loss of generality.

Remember that a matrix $B$ is symmetric if and only if[eq4]

Any quadratic form can be written as[eq5]where in step $ rame{A}$ we have used the fact that $x^{ op }Ax$ is a scalar and the transpose of a scalar is equal to the scalar itself.

The matrix[eq6]is symmetric because[eq7]

Thus, we have proved that we can always write a quadratic form as[eq8]where $B$ is symmetric.

Definiteness

Square matrices can be classified based on the sign of the quadratic forms that they define.

In what follows iff stands for "if and only if".

Definition Let $S$ be the space of all Kx1 vectors having real entries. A $K imes K$ real symmetric matrix A is said to be:

  1. positive definite iff $x^{ op }Ax>0$ for any non-zero $xin S$;

  2. positive semi-definite iff $x^{ op }Axgeq 0$ for any $xin S$;

  3. negative definite iff $x^{ op }Ax<0$ for any non-zero $xin S$;

  4. negative semi-definite iff $x^{ op }Axleq 0$ for any $xin S$;

  5. indefinite iff there exist $x,yin S$ such that $x^{ op }Ax>0$ and $y^{ op }Ay<0$.

Let us make an example.

Example Define[eq9]Given a $2 imes 1$ vector x, the quadratic form defined by the matrix A is[eq10]Since the sum[eq11]whenever $x_{1} eq 0$ or $x_{2} eq 0$ (hence $x eq 0$), the matrix A is positive definite.

Focus on positivity

From now on, we will mostly focus on positive definite and semi-definite matrices. The results obtained for these matrices can be promptly adapted to negative definite and semi-definite matrices. As a matter of fact, if A is negative (semi-)definite, then $-A$ is positive (semi-)definite. Thus, results can often be adapted by simply switching a sign.

A positive definite matrix is full-rank

An important fact follows.

Proposition Let A be a $K imes K$ matrix. If A is positive definite, then it is full-rank.

Proof

The proof is by contradiction. Suppose that A is not full-rank. Then its columns are not linearly independent. As a consequence, there is a Kx1 vector $x eq 0$ such that[eq12]We can pre-multiply both sides of the equation by $x^{ op }$ and obtain[eq13]Since A is positive definite, this is possible only if $x=0$, a contradiction. Thus A must be full-rank.

Eigenvalues of a positive definite matrix

The following proposition provides a criterion for definiteness.

Proposition A real symmetric $K imes K$ matrix A is positive definite if and only if all its eigenvalues are strictly positive real numbers.

Proof

Let us prove the "only if" part, starting from the hypothesis that A is positive definite. Let $lambda $ be an eigenvalue of A and x one of its associated eigenvectors. The symmetry of A implies that $lambda $ is real (see the lecture on the properties of eigenvalues and eigenvectors). Moreover, x can be chosen to be real since a real solution $x eq 0$ to the equation[eq14]is guaranteed to exist (because $A-lambda I$ is rank-deficient by the definition of eigenvalue). Then, we have[eq15]where [eq16] is the norm of x. Since x is an eigenvector, $x eq 0$. Moreover, by the definiteness property of the norm, [eq17]. Thus, we have[eq18]because $x^{ op }Ax>0$ by the hypothesis that A is positive definite (we have demonstrated above that the quadratic form $x^{ op }Ax$ involves a real vector x, which is required in our definition of positive definiteness). We have proved that any eigenvalue of A is strictly positive, as desired. Let us now prove the "if" part, starting from the hypothesis that all the eigenvalues of A are strictly positive real numbers. Since A is real and symmetric, it can be diagonalized as follows:[eq19]where $P$ is orthogonal and $D$ is a diagonal matrix having the eigenvalues of A on the main diagonal (as proved in the lecture on normal matrices). The eigenvalues are strictly positive, so we can write[eq20]where $D^{1/2}$ is a diagonal matrix such that its $left( k,k ight) $-th entry satisfies[eq21]for $k=1,ldots ,K$. Therefore, [eq22]and, for any vector $x eq 0$, we have[eq23]The matrix $P$, being orthogonal, is invertible (hence full-rank). The matrix $D^{1/2}$ is diagonal (hence triangular) and its diagonal entries are strictly positive, which implies that $D^{1/2}$ is invertible (hence full-rank) by the properties of triangular matrices. The product of two full-rank matrices is full-rank. Therefore, $PD^{1/2}$ is full-rank. Thus,[eq24]because $x eq 0$. By the positive definiteness of the norm, this implies that [eq25]and, as a consequence,[eq26]Thus, A is positive definite.

Eigenvalues of a positive semi-definite matrix

A very similar proposition holds for positive semi-definite matrices.

In what follows positive real number means a real number that is greater than or equal to zero.

Proposition A real symmetric $K imes K$ matrix A is positive semi-definite if and only if all its eigenvalues are positive real numbers.

Proof

We do not repeat all the details of the proof and we just highlight where the previous proof (for the positive definite case) needs to be changed. The first change is in the "only if" part, where we now have[eq27]because $x^{ op }Axgeq 0$ by the hypothesis that A is positive semi-definite. The second change is in the "if part", where we have [eq28]because the entries of $D$ are no longer guaranteed to be strictly positive and, as a consequence, $D^{1/2}$ is not guaranteed to be full-rank. It follows that [eq29]

The complex case

When the matrix A and the vectors x are allowed to be complex, the quadratic form becomes[eq30]where $x^{st }$ denotes the conjugate transpose of x.

Let $S$ be the space of all Kx1 vectors having complex entries. A $K imes K$ complex matrix A is said to be:

The negative definite and semi-definite cases are defined analogously.

Note that conjugate transposition leaves a real scalar unaffected. As a consequence, if a complex matrix is positive definite (or semi-definite), then[eq31]for any x, which implies that $A=A^{st }$. In other words, if a complex matrix is positive definite, then it is Hermitian.

Also in the complex case, a positive definite matrix A is full-rank (the proof above remains virtually unchanged).

Moreover, since A is Hermitian, it is normal and its eigenvalues are real. We still have that A is positive semi-definite (definite) if and only if its eigenvalues are positive (resp. strictly positive) real numbers. The proofs are almost identical to those we have seen for the real case. When adapting those proofs, we just need to remember that in the complex case[eq32]

Solved exercises

Below you can find some exercises with explained solutions.

Exercise 1

Let A be a complex matrix and x one of its eigenvectors. Can you write the quadratic form $x^{st }Ax$ in terms of [eq16]?

Solution

Let $lambda $ be the eigenvalue associated to x. Then,[eq34]

Exercise 2

Can you tell whether the matrix [eq35]is positive definite?

Solution

Let x be a $2 imes 1$ vector. Denote its entries by $x_{1}$ and $x_{2}$. Then,[eq36]Then, $x^{ op }Axgeq 0$ if $x eq 0$ and A is positive semi-definite. However, it is not positive definite because there exist non-zero vectors, for example the vector[eq37]such that $x^{ op }Ax=0$.

Exercise 3

Suppose that A is a complex negative definite matrix. What can you say about the sign of its eigenvalues?

Solution

If A is negative definite, then[eq38]for any $x eq 0$. As a consequence,[eq39]In other words, the matrix $-A$ is positive definite. It follows that the eigenvalues of $-A$ are strictly positive. If $lambda $ is an eigenvalue of $-A$ associated to an eigenvector x, then[eq40]The latter equation is equivalent to[eq41]So, if $lambda $ is an eigenvalue of $-A$, then $-lambda $ is an eigenvalue of A. Thus, the eigenvalues of A are strictly negative.

How to cite

Please cite as:

Taboga, Marco (2021). "Positive definite matrix", Lectures on matrix algebra. https://www.statlect.com/matrix-algebra/positive-definite-matrix.

The books

Most of the learning materials found on this website are now available in a traditional textbook format.

Featured pages
Explore
Main sections
About
Glossary entries
Share
  • To enhance your privacy,
  • we removed the social buttons,
  • but don't forget to share.