StatlectThe Digital Textbook
Index > Fundamentals of probability

Probability

Probability is used to quantify the likelihood of things that can happen, when it is not yet known whether they will happen. Sometimes it is also used to quantify the likelihood of things that could have happened in the past, when it is not yet known whether they actually happened.

Since we usually speak of the "probability of an event", the next section introduces a formal definition of the concept of event. We then discuss the properties that probability needs to satisfy. Finally, we mention some possible interpretations of the concept of probability.

Sample space, sample points and events

Let Omega be a set of things that can happen. We say that Omega is a sample space, or space of all possible outcomes, if it satisfies the following two properties:

  1. Mutually exclusive outcomes. Only one of the things in Omega will happen. That is, when we learn that [eq1] has happened, then we also know that none of the things in the set [eq2] has happened.

  2. Exhaustive outcomes. At least one of the things in Omega will happen.

An element omega in Omega is called a sample point, or a possible outcome.

When, and if, we learn that [eq3] has happened, $overline{omega }$ is called the realized outcome.

A subset $Esubseteq Omega $ is called an event. In the section below, you will see that not every subset of the sample space is, strictly speaking, an event; however, on a first reading you can be happy with this definition.

Note that Omega itself is an event, because every set is a subset of itself, and the empty set $emptyset $ is also an event, because it can be considered a subset of Omega.

Example Suppose that we toss a die. Six numbers, from 1 to 6, can appear face up, but we do not yet know which one of them will appear. The sample space is[eq4]Each of the six numbers is a sample point. The outcomes are mutually exclusive, because only one number at a time can appear face up. The outcomes are also exhaustive, because at least one of the six numbers will appear face up, after we toss the die. Define[eq5]E is an event (a subset of Omega). It can be described as "an odd number appears face up". Now define[eq6]Also F is an event and it can be described as "the number 6 appears face up".

Probability and its properties

The probability of an event is a real number, attached to the event, that tells us how likely that event is. We denote the probability of an event E by [eq7].

Probability needs to satisfy the following properties:

  1. Range. For any event E, [eq8].

  2. Sure thing. [eq9].

  3. Sigma-additivity (or countable additivity). Let [eq10] be a sequence of events. Let all the events in the sequence be mutually exclusive, i.e., [eq11] if $i
eq j$. Then,[eq12]

Property (1) is self-explanatory. It just means that the probability of an event is a real number between 0 and 1.

Property (2) says that at least one of all the things that can possibly happen will happen with probability 1.

Property (3) is a bit more cumbersome. It can be proved (see below) that if sigma-additivity holds, then also the following holds:[eq13]

This property, called finite additivity, while very similar to sigma-additivity, is easier to interpret. It says that if two events are disjoint, then the probability that either one or the other happens is equal to the sum of their individual probabilities.

Example Suppose that we flip a coin. The possible outcomes are either tail ($T$) or head (H), i.e.,[eq14]There are a total of four subsets of Omega (events): Omega itself, the empty set $emptyset $, the event $left{ T
ight} $ and the event $left{ H
ight} $. The following assignment of probabilities satisfies the properties enumerated above:[eq15]All these probabilities are between 0 and 1, so the range property is satisfied. [eq9], so the sure thing property is satisfied. Also sigma-additivity is satisfied because[eq17]and the four couples $left( T,H
ight) $, [eq18], [eq19], [eq20] are the only four possible couples of disjoint sets.

Before ending this section, two remarks are in order. First, we have not discussed the interpretations of probability, but below you can find a brief discussion of the interpretations of probability. Second, we have been somewhat sloppy in defining events and probability, but you can find a more rigorous definition of probability below.

Other properties of probability

The following subsections discuss other properties enjoyed by probability.

The probability of the empty set is 0

Here we prove that [eq21].

Proof

Define a sequence of event as follows: [eq22]The sequence is a sequence of disjoint events, because the empty set is disjoint from any other set. Then, [eq23]which implies[eq24]and [eq21].

A sigma-additive function is additive

A sigma-additive function is also additive:[eq13]

Proof

Define a sequence of events as follows: [eq27]The sequence is a sequence of disjoint events, because the empty set is disjoint from any other set. Then,[eq28]

Probability of the complement

Let E be an event and $E^{c}$ its complement (i.e. the set of all elements of Omega that do not belong to E). Then,[eq29]

Proof

Note that[eq30]and that E and $E^{c}$ are disjoint sets. Then, by using the sure thing property and finite additivity, we obtain[eq31]which implies[eq32]

In other words, the probability that an event does not occur is equal to one minus the probability that it occurs.

Probability of a union

We have already seen how to compute [eq33] in the special case in which E and F are two disjoint events. In the more general case, in which they are not necessarily disjoint, the formula is[eq34]

Proof

This is proved as follows. First note that[eq35]so that[eq36]Furthermore the event $Ecup F$ can be written as follows:[eq37]and the three events on the right hand side are disjoint. Thus,[eq38]

Monotonicity of probability

If two events E and F are such that $Esubseteq F$, then[eq39]

Proof

This is easily proved by using additivity:[eq40]where the latter inequality is a consequence of the fact that [eq41] (by the range property of probability).

In other words, if E occurs less often than F, because the latter contemplates more occurrences, then the probability of E must be less than the probability of F.

Interpretations of probability

This subsection briefly discusses some common interpretations of probability. Although none of these interpretations is sufficient per se to clarify the meaning of probability, they all touch upon important aspects of probability.

Classical interpretation of probability

According to the classical definition of probability, when all the possible outcomes of an experiment are equally likely, the probability of an event is the ratio between the number of outcomes that are favorable to the event and the total number of possible outcomes. While intuitive, this definition has two main drawbacks:

  1. it is circular, because it uses the concept of probability to define probability: it is based on the assumption of 'equally likely' outcomes, where equally likely means "having the same probability";

  2. it is limited in scope, because it does not allow to define probability when the possible outcomes are not all equally likely.

Frequentist interpretation of probability

According to the frequentist definition of probability, the probability of an event is the relative frequency of the event itself, observed over a large number of repetitions of the same experiment. In other words, it is the limit to which the ratio:[eq42]converges when the number of repetitions of the experiment tends to infinity. Despite its intuitive appeal, also this definition of probability has some important drawbacks:

  1. it assumes that all probabilistic experiments can be repeated many times, which is false;

  2. it is also somewhat circular, because it implicitly relies on a Law of Large Numbers, which can be derived only after having defined probability.

Subjectivist interpretation of probability

According to the subjectivist definition of probability, the probability of an event is related to the willingness of an individual to accept bets on that event. Suppose a lottery ticket pays off 1 dollar in case the event occurs and 0 in case the event does not occur. An individual is asked to set a price for this lottery ticket, at which she must be indifferent between being a buyer or a seller of the ticket. The subjective probability of the event is defined to be equal to the price thus set by the individual. Also this definition of probability has some drawbacks:

  1. different individuals can set different prices, therefore preventing an objective assessment of probabilities;

  2. the price an individual is willing to pay to participate in a lottery can be influenced by other factors that have nothing to do with probability; for example, an individual's betting behavior can be influenced by her preferences.

Rigorous definitions

A more rigorous definition of event

The definition of event given above is not entirely rigorous. Often, statisticians work with probability models where some subsets of the sample space are not considered events. This happens mainly for the following two reasons:

  1. sometimes, the sample space is a really complicated set; to make things simpler, attention is restricted to only some subsets of the sample space;

  2. sometimes, it is possible to assign probabilities only to some subsets of the sample space; in these cases, only the subsets to which probabilities can be assigned are considered events.

Denote by $	ciFourier $ the space of events, i.e., the set of subsets of Omega that are considered events. In rigorous probability theory, it is required to be a sigma-algebra.

Definition $	ciFourier $ is a sigma-algebra on Omega if it is a set of subsets of Omega satisfying the following three properties:

  1. Whole set. [eq43].

  2. Closure under complementation. If $Ein 	ciFourier $ then also [eq44] (the complement $E^{c}$ is the set of all elements of Omega that do not belong to E).

  3. Closure under countable unions. If [eq45] is a sequence of subsets of Omega belonging to $	ciFourier $, then[eq46]

Why is a space of events required to satisfy these properties? Besides a number of mathematical reasons, it seems pretty intuitive that they must be satisfied. Property 1) means that the space of events must include the event "something will happen", quite a trivial requirement! Property 2) means that if "one of the things in the set E will happen" is considered an event, then also "none of the things in the set E will happen" is considered an event. This is quite natural: if you are considering the possibility that an event will happen, then, by necessity, you must also be simultaneously considering the possibility that the same event will not happen. Property 3) is a bit more complex. However, the following property, implied by 3), is probably easier to interpret:[eq47]It means that if "one of the things in E will happen" and "one of the things in F will happen" are considered two events, then also "either one of the things in E or one of the things in F will happen" must be considered an event. This simply means that if you are able to separately assess the possibility of two events happening, then, of course, you must be able to assess the possibility of either one or the other happening. Property 3) simply extends this intuitive property to countable collection of events: the extension is needed for mathematical reasons, to derive certain continuity properties of probability measures.

A more rigorous definition of probability

The definition of probability given above was not entirely rigorous. Now that we have defined sigma-algebras and spaces of events, we can make it completely rigorous.

Definition Let $	ciFourier $ be a sigma-algebra on the sample space Omega. A function [eq48] is a probability measure if and only if it satisfies the following two properties:

  1. Sure thing. [eq9].

  2. Sigma-additivity. Let [eq50] be any sequence of elements of $	ciFourier $ such that $i
eq j$ implies [eq51]. Then,[eq52]

Nothing new has been added to the definition given above. This definition just clarifies that a probability measure is a function defined on a sigma-algebra of events. Hence, it is not possible to properly speak of probability for subsets of the sample space that do not belong to the sigma-algebra.

A triple [eq53] is called a probability space.

Solved exercises

Below you can find some exercises with explained solutions.

Exercise 1

A ball is drawn at random from an urn containing colored balls. The balls can be either red or blue (no other colors are possible). The probability of drawing a blue ball is $1/3$. What is the probability of drawing a red ball?

Solution

The sample space Omega can be represented as the union of two disjoint events E and F:[eq54]where the event E can be described as 'a red ball is drawn' and the event F can be described as 'a blue ball is drawn'. Note that E is the complement of F:[eq55]

We know [eq56], the probability of drawing a a blue ball:[eq57]

We need to find [eq58], the probability of drawing a a red ball. Using the formula for the probability of a complement:[eq59]

Exercise 2

Consider a sample space Omega comprising three possible outcomes:[eq60]

Suppose the probabilities assigned to the three possible outcomes are[eq61]

Can you find an event whose probability is $3/4$?

Solution

There are two events whose probability is $3/4$.

The first one is[eq62]

By using the formula for the probability of a union of disjoint events, we get[eq63]

The second one is[eq64]

By using again the formula for the probability of a union of disjoint events, we obtain[eq65]

Exercise 3

Consider a sample space Omega comprising four possible outcomes:[eq66]

Consider the three events E, F and $G$ defined as follows:[eq67]

Suppose their probabilities are[eq68]

Now, consider a fourth event H defined as follows:[eq69]

Find [eq70].

Solution

First note that, by additivity:[eq71]

Therefore, in order to compute [eq70], we need to compute [eq73] and [eq74].

[eq73] is found by using additivity on F:[eq76]so that[eq77]

[eq78] is found using the fact that one minus the probability of an event is equal to the probability of its complement and the fact that [eq79]:[eq80]

As a consequence,[eq81]

The book

Most learning materials found on this website are now available in a traditional textbook format.