19 Probability spaces
A probability space is a very general formal-mathematical model of a random process. The central importance of this model for probability theory and probabilistic inference follows from the fact that the probability space model provides guidance for translating arbitrary random processes about which one wants to reason quantitatively into the formal-mathematical system of probability theory. At the same time, the probability space model and the concepts built on it guarantee that the mechanics of probability calculus lead to logically meaningful quantitative conclusions about random processes. In this chapter, we introduce the concept of a probability space (Section 19.1) and then use probability functions (Section 19.2) to give first examples of modeling random processes (Section 19.3).
19.1 Definition and first properties
We begin with the definition of the probability space model after Kolmogoroff (1933), which we then explain in its individual parts from a frequentist perspective.
Definition 19.1 (Probability space) A probability space is a triple \((\Omega, \mathcal{A}, \mathbb{P})\), where
\(\Omega\) is an arbitrary nonempty set of outcomes \(\omega\) and is called the outcome set,
\(\mathcal{A}\) is a set of subsets of \(\Omega\) with the properties
- \(\Omega \in \mathcal{A}\),
- for all \(A\in \mathcal{A}\), also \(A^c := \Omega \setminus A \in \mathcal{A}\),
- from \(A_1,A_2,... \in \mathcal{A}\) it follows that also \(\cup_{i=1}^\infty A_i \in \mathcal{A}\)
holds, is called a \(\sigma\)-algebra on \(\Omega\), and is called the event system,
\(\mathbb{P}\) is a mapping of the form \(\mathbb{P}:\mathcal{A} \to [0,1]\) with the properties
- \(\mathbb{P}(A) \ge 0\) for all \(A \in \mathcal{A}\) (non-negativity),
- \(\mathbb{P}(\Omega) = 1\) (normalization),
- \(\mathbb{P}(\cup_{i=1}^\infty A_i) = \sum\limits_{i=1}^\infty \mathbb{P}(A_i)\) for pairwise disjoint \(A_i \in \mathcal{A}\) (\(\sigma\)-additivity)
holds and is called a probability measure.
Outcome set and mechanics
We begin with explanations of the concept of the outcome set \(\Omega\) and the implicit mechanics of the probability space model. To make the introduction easier, in what follows we first primarily consider finite probability spaces, in which the cardinality of \(\Omega\) is not infinitely large. Thus let \(|\Omega|<\infty\), so that \(\Omega\) has only finitely many elements. For modeling the throw of a die, for example, one could define \(\Omega := \{1,2,3,4,5,6\}\).
Behind the formal definition of the probability space model are the following frequentist-influenced assumptions about its mechanics as a model of a random process. We first imagine sequential runs of a random process, for example the repeated throwing of a die. According to the assumptions of the probability space model, in each of these runs exactly one \(\omega\) from \(\Omega\) is realized, that is, selected as actually occurring. For example, if one throws a die and a two appears, one says that a two was realized. The probability with which an \(\omega\) from \(\Omega\) is realized in a run is described by the value \(\mathbb{P}(\{\omega\}) \in [0,1]\). If, for example, \(\mathbb{P}(\{\omega\}) = 1\), then this \(\omega\) is realized in every run of the random process; if \(\mathbb{P}(\{\omega\}) = 0\), then this \(\omega\) is realized in no run of the random process; and if \(\mathbb{P}(\{\omega\}) = 1/2\), then \(\omega\) is realized in about half of the runs of the random process. In the model of throwing a fair die, one usually assumes \(\mathbb{P}(\{\omega\}) = 1/6\) for all \(\omega \in \Omega\). Here, for example, a four could be realized in the first run, a one in the second run, a five in the third run, then perhaps a four again, and so on.
Events and event system
The concept of an event \(A \in \mathcal{A}\) is best imagined as a conceptual summary of one or more outcomes. When throwing a die, possible events are, for example, “An even number of pips appears”, that is, \(\omega \in \{2,4,6\}\), “A number of pips greater than two appears”, that is, \(\omega \in \{3,4,5,6\}\), or “A one or a five appears”, that is, \(\omega \in \{1,5\}\). Note that, against the background of the mechanics of the probability space, the event “An even number of pips appears” occurs exactly when, in a run of the random process throwing a die, the realized \(\omega\) is an element of the set \(\{2,4,6\}\), for example when a four appears. The occurrence of the event “A number of pips greater than two appears” may also be read as “In a run of the random process throwing a die, an element of \(\{3,4,5,6\}\) is realized”, that is, concretely, either a three, a four, a five, or a six appears. Of course, the outcomes \(\omega \in \Omega\) themselves are also possible events, so that, for example, the following interpretations hold: the event “A one appears” corresponds to the realization \(\omega \in \{1\}\), and the event “A six appears” corresponds to the realization \(\omega \in \{6\}\). If one considers an outcome \(\omega \in \Omega\) as an event in this context, it is called an elementary event and is written as the one-element set \(\{\omega\}\).
Overall, this procedure for describing random events corresponds to the inherent goal of the definition of the probability space. Kolmogoroff (1933) writes that the actual objects of the subsequent considerations, the random events, have been defined as sets. This has the advantage that sets are mathematical objects with which one can work mathematically, so that an aspect of reality, a “random event”, has been translated into the model domain of mathematics.
The sole purpose of the event system \(\mathcal{A}\) is now to mathematically represent all events that can arise, based on a given outcome set, when an \(\omega \in \Omega\) is selected. Thus there should be no events in reality that were not considered in advance in the probability space model of the random process. If this were the case, the model would be deficient, since it could not assign probabilities to certain events occurring in reality. The event system \(\mathcal{A}\) should therefore be the complete set of all possible events for a given \(\Omega\). The requirement that \(\mathcal{A}\) should satisfy the so-called \(\sigma\)-algebra criteria for this purpose is intuitively motivated as follows.
- It should first be ensured that \(\omega \in \Omega\) for an arbitrary \(\omega\), that is, that some outcome is realized, is one of the possible events. This corresponds to the property \(\Omega \in \mathcal{A}\).
- For every event it should also be possible that this event does not occur. This corresponds to the property that from \(A \in \mathcal{A}\) it should follow that \(A^c := \Omega \setminus A\) is also in \(\mathcal{A}\). This also implies in particular that \(\emptyset = \Omega \setminus \Omega \in \mathcal{A}\). Thus one event is that no elementary event occurs, although this happens only with probability zero, \(\mathbb{P}(\emptyset) = 0\). Thus at least one elementary event always occurs with certainty.
- Finally, the combination of events should always also be an event. In modeling the throw of a die, for example, in addition to the events “An even number appears” and “A number greater than two appears”, the event “An even number appears and/or this number is greater than two” should also be an event. In its most general form, this corresponds to the requirement that from \(A_1,A_2,... \in \mathcal{A}\) it should follow that also \(\cup_{i=1}^\infty A_i \in \mathcal{A}\).
Even if the concepts of the event system and the \(\sigma\)-algebra may seem somewhat abstract, their definition in the practical modeling of random processes is usually not a major challenge, since suitable event systems have long been known both for finite outcome sets and for infinite countable and uncountable outcome sets. Thus, for outcome sets with finite cardinality, the power set of the outcome set always satisfies the requirements of an event system and can always be used to formulate a model of a random process with a finite outcome set. This is the statement of the following theorem.
Theorem 19.1 (Event system for finite outcome set) Let \(\Omega := \{\omega_1,\omega_2,...,\omega_n\}\) with \(n \in \mathbb{N}\) be a finite set. Then the power set \(\mathcal{P}(\Omega)\) of \(\Omega\) is a \(\sigma\)-algebra on \(\Omega\) and thus a suitable event system in the probability space model.
Proof. The power set of \(\Omega\) is the set of all subsets of \(\Omega\). We check the \(\sigma\)-algebra properties. First, \(\Omega\) itself is one of the subsets of \(\Omega\), so the first \(\sigma\)-algebra property is satisfied. Now let \(A\) be a subset of \(\Omega\). Then \(A^c = \Omega \setminus A\) is also a subset of \(\Omega\), and hence the second \(\sigma\)-algebra property is also satisfied. Finally, consider the union of \(k\) subsets \(A_1, A_2, ...,A_k \subseteq \Omega\). Then \(\cup_{i=1}^k A_i\) is the set of \(\omega \in \Omega\) for which \(\omega \in A_1\) and/or \(\omega \in A_2\) … and/or \(\omega \in A_k\). Since for all these \(\omega\) it holds that \(\omega \in \Omega\), also \(\cup_{i=1}^k A_i\) is a subset of \(\Omega\), and therefore the third \(\sigma\)-algebra property is also satisfied. The power set thus satisfies the required properties of an event system.
For uncountable outcome sets such as the real numbers \(\mathbb{R}\) or the \(n\)-dimensional real space \(\mathbb{R}^n\), the construction of a suitable event system is more complex, so in this respect we refer to the advanced literature for formal developments (cf. Meintrup & Schäffler (2005), Schmidt (2009)). However, with the so-called Borel \(\sigma\)-algebra, which goes back to Borel (1898), a set system is known that satisfies the requirements of a \(\sigma\)-algebra on uncountable outcome sets. We denote the Borel \(\sigma\)-algebra on \(\mathbb{R}\) by \(\mathcal{B}(\mathbb{R})\) and the Borel \(\sigma\)-algebra on \(\mathbb{R}^n\) by \(\mathcal{B}(\mathbb{R}^n)\). As sets of subsets of \(\mathbb{R}\) and \(\mathbb{R}^n\), respectively, \(\mathcal{B}(\mathbb{R})\) and \(\mathcal{B}(\mathbb{R}^n)\) contain all sets whose probability assigned by \(\mathbb{P}\) one may be interested in. Intuitively, one may therefore think of the Borel \(\sigma\)-algebras \(\mathcal{B}(\mathbb{R})\) and \(\mathcal{B}(\mathbb{R}^n)\) as the power sets of \(\mathbb{R}\) and \(\mathbb{R}^n\), respectively, even though this is formally incorrect. In fact, the Borel \(\sigma\)-algebra contains only subsets generated by countable set operations, not by uncountable ones.
Overall, this leads to the following procedure for selecting event systems depending on the outcome set \(\Omega\). If \(\Omega\) is finite, one chooses as event system \(\mathcal{A}\) the power set \(\mathcal{P}(\Omega)\) of \(\Omega\). If \(\Omega\) is given by \(\mathbb{R}\), one chooses as event system \(\mathcal{A}\) the Borel \(\sigma\)-algebra \(\mathcal{B}(\mathbb{R})\). Finally, if \(\Omega\) is given by \(\mathbb{R}^n\), one chooses for \(\mathcal{A}\) the Borel \(\sigma\)-algebra \(\mathcal{B}(\mathbb{R}^n)\). In special cases and for very special outcome sets \(\Omega\), one may want to deviate from this procedure, but in general the three cases considered cover most applications.
Probability measure \(\mathbb{P}\)
With \(\Omega\) and \(\mathcal{A}\), which in tuple form \((\Omega,\mathcal{A})\) are also called a measurable space, we have so far examined more closely the structural basis of a probability space model. Many probability spaces, for example for random processes concerning real numbers, are identical with respect to their measurable space. The probability measure \(\mathbb{P}\) now represents the probabilistic characteristics of a probability space model and thus forms the functional basis of such a model. In what follows, especially after introducing the concepts of random variables and random vectors, we will encounter many different probability measures. At this point, we first want to consider only general properties of probability measures.
With the definition \[\begin{equation} \mathbb{P}: \mathcal{A} \to [0,1], A \mapsto \mathbb{P}(A) \end{equation}\] it first follows that a probability measure is defined on a set of sets and assigns probabilities, that is, values in the interval \([0,1]\), to the elements of this set, namely to the sets \(A \in \mathcal{A}\). Of course, with \(\{\omega\} \in \mathcal{A}\) for all \(\omega \in \Omega\), \(\mathbb{P}\) also assigns probabilities to the elementary events, but not only to them. We also emphasize that, by definition, the probability \(\mathbb{P}(A)\) of an event \(A \in \mathcal{A}\) is a number in the interval \([0,1]\) and not, for example, a percentage or a ratio. In what follows, we will examine more closely the defining properties of non-negativity, normalization, and \(\sigma\)-additivity of \(\mathbb{P}\).
The non-negativity \(\mathbb{P}(A) \ge 0\) for all \(A \in \mathcal{A}\) is of course implicit in the definition \([0,1]\) of the target set of \(\mathbb{P}\). In fact, the mapping form of \(\mathbb{P}\) is an addition we have made to the formulation of Kolmogoroff (1933) in the interest of clarity. Formally, the form of the target set of \(\mathbb{P}\) actually follows from the defining properties of non-negativity, normalization, and \(\sigma\)-additivity of \(\mathbb{P}\).
The normalization \(\mathbb{P}(\Omega) = 1\) corresponds to the fact that, in every run of a random process, the realized \(\omega\) is certainly an element of \(\Omega\). Thus in every run of a random process an elementary event occurs and, depending on the nature of the measurable space, many others as well. In the model of throwing a die, for example, the outcome/elementary event realized in a run of the random process is an element of \(\Omega := \{1,2,3,4,5,6\}\) with probability \(1\). If the realized outcome is, for example, a one, then in addition to the event “A one appears”, the events “An odd number appears”, “A number less than three appears”, “An odd number less than three appears”, and many others also occur.
The \(\sigma\)-additivity of the probability measure \(\mathbb{P}\), finally, forms the foundation of probability calculus, that is, the basis for calculating with probabilities. The \(\sigma\)-additivity of \(\mathbb{P}\) makes it possible to calculate the probabilities of other events from already known event probabilities. Based on a definition of \(\Omega, \mathcal{A}\), and \(\mathbb{P}\), one can therefore calculate probabilities for all possible events of a probability space model. Whether these probabilities actually have anything to do with real events concerning a random process in reality depends on whether the modeling is reasonably successful or not. However, the calculated probabilities are at least determined rationally, that is, according to the rules of reason, namely logic and mathematics. Overall, the probability model thus allows inferential thinking about random processes, that is, about phenomena subject to uncertainty.
Finally, we want to illustrate probability calculations based on the \(\sigma\)-additivity of \(\mathbb{P}\) with two basic examples. The first example states that the probability that the \(\omega\) realized in a run of a random process is not an element of the outcome set is equal to zero.
Theorem 19.2 (Probability of the impossible event) Let \((\Omega, \mathcal{A}, \mathbb{P})\) be a probability space. Then \[\begin{equation} \mathbb{P}(\emptyset) = 0. \end{equation}\]
Proof. For \(i = 1,2,...\), let \(A_i := \emptyset\). Then \(A_1,A_2,...\) is a sequence of disjoint events because \(\emptyset \cap \emptyset = \emptyset\), and \(\cup_{i=1}^\infty A_i = \emptyset\). With the \(\sigma\)-additivity of \(\mathbb{P}\) it then follows that \[\begin{equation} \mathbb{P}(\emptyset) = \mathbb{P}\left(\cup_{i=1}^\infty A_i\right) = \sum_{i=1}^\infty \mathbb{P}\left(A_i\right) = \sum_{i=1}^\infty \mathbb{P}\left(\emptyset\right). \end{equation}\] Thus the infinite addition of the number \(\mathbb{P}(\emptyset) \in [0,1]\) should again yield \(\mathbb{P}(\emptyset)\). This is only possible if \(\mathbb{P}(\emptyset) = 0\).
Note that, intuitively, a possible inadequacy of the probability space as a model for random processes in reality can arise here: if, when playing dice, the die falls out of reach under the sofa, then an elementary event \(\omega \notin \Omega\) has occurred, even though its modeled probability is equal to zero.
As a second example, we want to show that \(\sigma\)-additivity, which in the definition of the probability space is defined only for the union of infinitely many disjoint events, implies the \(\sigma\)-additivity of finitely many disjoint events as they often occur in applications.
Theorem 19.3 (\(\sigma\)-additivity for finite sequences of disjoint events) Let \((\Omega, \mathcal{A}, \mathbb{P})\) be a probability space and let \(A_1,...,A_n\) be a finite sequence of pairwise disjoint events. Then \[\begin{equation} \mathbb{P}(\cup_{i=1}^n A_i) = \sum_{i=1}^n \mathbb{P}(A_i). \end{equation}\]
Proof. We consider an infinite sequence of pairwise disjoint events \(A_1, A_2, ...\), where for an \(n\in \mathbb{N}\) it is to hold that \(A_i := \emptyset\) for \(i>n\). Then, with the \(\sigma\)-additivity of \(\mathbb{P}\), it first follows that \[\begin{equation} \mathbb{P}\left(\cup_{i=1}^n A_i\right) = \mathbb{P}\left(\cup_{i=1}^\infty A_i\right) = \sum_{i=1}^\infty \mathbb{P}\left(A_i\right) = \sum_{i=1}^n \mathbb{P}\left(A_i\right) + \sum_{i=n+1}^\infty \mathbb{P}\left(A_i\right). \end{equation}\] With \(\mathbb{P}\left(A_i\right) = \mathbb{P}(\emptyset) = 0\) for \(i = n+1, n+2,...\), it follows directly that \[\begin{equation} \mathbb{P}\left(\cup_{i=1}^n A_i\right) = \sum_{i=1}^n \mathbb{P}\left(A_i\right) + 0 = \sum_{i=1}^n \mathbb{P}\left(A_i\right). \end{equation}\]
To prepare the concept of the joint probability of events, we finally consider one further important property of \(\sigma\)-algebras: in addition to being closed under countable unions, \(\mathcal{A}\) is also closed under countable intersections. This is a direct consequence of the closure of \(\mathcal{A}\) under complements and countable unions and is recorded in the following theorem.
Theorem 19.4 (Closure of \(\sigma\)-algebras under intersections) Let \(\mathcal{A}\) be a \(\sigma\)-algebra on a set \(\Omega\). Then from \(A,B \in \mathcal{A}\) it follows that also \(A \cap B \in \mathcal{A}\). More generally, for \(A_1,A_2,... \in \mathcal{A}\), also \(\cap_{i=1}^\infty A_i \in \mathcal{A}\).
Proof. We show the first statement by way of example. To this end, first note that by the first De Morgan rule and the properties of complements, \[\begin{equation} (A \cap B)^c = A^c \cup B^c \Leftrightarrow \left((A \cap B)^c \right)^c = \left(A^c \cup B^c \right)^c \Leftrightarrow A \cap B = \left(A^c \cup B^c \right)^c. \end{equation}\] Furthermore, with the defining properties of \(\sigma\)-algebras, for \(A,B \in \mathcal{A}\) it holds that also \(\left(A^c \cup B^c\right)^c \in \mathcal{A}\). For from the closure of \(\mathcal{A}\) under complements it first follows that also \(A^c \in \mathcal{A}\) and \(B^c \in \mathcal{A}\). Further, from the closure of \(\mathcal{A}\) under countable unions it also follows that \(A^c \cup B^c \in \mathcal{A}\). Finally, with the closure of \(\mathcal{A}\) under complements, \(\left(A^c \cup B^c\right)^c \in \mathcal{A}\). Overall, then, from \(A,B \in \mathcal{A}\) it follows that \(A \cap B \in \mathcal{A}\). The generalization to countable intersections \(A_1,A_2,... \in \mathcal{A}\) follows analogously.
According to Theorem 19.4, in addition to the probabilities \(\mathbb{P}(A), \mathbb{P}(B)\) and \(\mathbb{P}(A \cup B)\) with \(A,B \in \mathcal{A}\), the probability \(\mathbb{P}(A \cap B)\) is therefore also well-defined.
19.2 Probability functions
In this section, with probability functions, we want to learn about a way of specifying probability measures for probability spaces with finite outcome sets. In the sections on random variables and random vectors, we will encounter probability functions again under the name probability mass functions. We first define the concept of a probability function as follows.
Definition 19.2 (Probability function) Let \(\Omega\) be a finite set. Then a function \(\pi:\Omega \to [0,1]\) is called a probability function if \[\begin{equation} \sum_{\omega \in \Omega} \pi(\omega) = 1. \end{equation}\] Furthermore, let \(\mathbb{P}\) be a probability measure. Then the function defined by \[\begin{equation} \pi: \Omega \to [0,1], \omega \mapsto \pi(\omega) := \mathbb{P}(\{\omega\}) \end{equation}\] is called the probability function of \(\mathbb{P}\) on \(\Omega\).
We note that because \(\mathbb{P}\) is \(\sigma\)-additive by definition, in particular \[\begin{equation} \mathbb{P}(\Omega) = \mathbb{P}(\cup_{\omega \in \Omega}\{\omega\}) = \sum_{\omega \in \Omega}\mathbb{P}(\{\omega\}) = \sum_{\omega \in \Omega}\pi(\omega) = 1. \end{equation}\] The following theorem provides the formal basis for constructing probability measures through probability functions. In particular, it states that for finite \(\Omega\), the probabilities of all possible events can be calculated from the probabilities of the elementary events \(\pi(\omega)\).
Theorem 19.5 Let \((\Omega, \mathcal{A}, \mathbb{P})\) be a probability space with finite outcome set, and let \(\pi: \Omega \to [0,1]\) be a probability function. Then there exists a probability measure \(\mathbb{P}\) on \(\Omega\) with \(\pi\) as the probability function of \(\mathbb{P}\). This probability measure is defined as \[\begin{equation} \mathbb{P} : \mathcal{A} \to [0,1], A \mapsto \mathbb{P}(A) := \sum_{\omega \in A} \pi(\omega). \end{equation}\]
Proof. We first check the probability measure properties of \(\mathbb{P}\). Because \(\pi(\omega) \in [0,1]\) for all \(\omega \in \Omega\), it is always true that \(\sum_{\omega \in A} \pi(\omega) \ge 0\), and hence non-negativity of \(\mathbb{P}\) holds. Furthermore, as seen above, normalization of \(\mathbb{P}\) follows directly from normalization of \(\pi\). Now let \(A_1, A_2,... \in \mathcal{A}\). Then \[\begin{equation} \mathbb{P}\left(\cup_{i=1}^\infty A_i \right) = \sum_{\omega \in \cup_{i=1}^\infty A_i} \pi(\omega) = \sum_{i = 1}^\infty \sum_{\omega \in A_i} \pi(\omega) = \sum_{i = 1}^\infty \mathbb{P}(A_i) \end{equation}\] and hence \(\sigma\)-additivity of \(\mathbb{P}\).
Thus, if for a given outcome set \(\Omega\) one defines a function \(\pi : \Omega \to [0,1]\), ensures that its function values \(\pi(\omega)\) sum to 1 over all \(\omega \in \Omega\), and then interprets the individual function value \(\pi(\omega)\) as the probability \(\mathbb{P}(\{\omega\})\) of the elementary event \(\{\omega\} \in \mathcal{A}\), one has constructed a probability measure. Conversely, if for a finite outcome set \(\Omega\) one defines a probability \(\mathbb{P}\left(\{\omega\}\right) \in [0,1]\) for all elementary events \(\{\omega\} \in \mathcal{A}\) and ensures that \(\mathbb{P}(\Omega) = 1\), then one has of course implicitly also satisfied the requirements of a probability function. In this case, the \(\sigma\)-additivity of \(\mathbb{P}\) can be used directly for the pairwise disjoint elementary events to calculate compound events. We illustrate this and the procedure using a probability function in Section 19.3 by means of examples.
19.3 Examples with finite outcome space
From what has been said so far, the following procedure for modeling a random process using a probability space \((\Omega, \mathcal{A}, \mathbb{P})\) can be summarized:
In a first step, one considers a sensible definition of the outcome set \(\Omega\), that is, of the outcomes or elementary events that are to be realized in each run of the random process.
In a second step, one then chooses a suitable event system. In the case of a finite outcome set, the power set \(\mathcal{P}(\Omega)\) is suitable; in the case of the uncountable outcome set \(\Omega := \mathbb{R}\), the Borel \(\sigma\)-algebra \(\mathcal{B}(\mathbb{R})\) is suitable.
Finally, one defines a probability measure \(\mathbb{P}\) that represents the probabilities of the occurrence of all possible events. In the case of a finite outcome set, this is accomplished in particular by defining the probabilities of the elementary events. In what follows, we illustrate this procedure with examples. For the uncountable outcome set \(\Omega := \mathbb{R}\), defining \(\mathbb{P}\) with the help of probability density functions is suitable, as we will see later.
Throwing one die
We model throwing one die. It is certainly sensible to define the outcome set as \(\Omega := \{1,2,3,4,5,6\}\). However, the definition \[\begin{equation} \Omega := \{ \cdot, \cdot\cdot, \cdot\cdot\cdot, \cdot\cdot\cdot\cdot, \cdot\cdot\cdot\cdot\cdot, \cdot\cdot\cdot\cdot\cdot\cdot \} \end{equation}\] would also be possible in an equivalent way.
Because this is a finite outcome set, we choose the power set \(\mathcal{P}(\Omega)\) as the \(\sigma\)-algebra \(\mathcal{A}\). Then \(\mathcal{A}\) automatically contains all possible events. The cardinality of \(\mathcal{A} := \mathcal{P}(\Omega)\) is \(|\mathcal{P}(\Omega)| = 2^{|\Omega|} = 2^6 = 64\). In Table 19.1 we list six of these 64 events in their verbal description and as a subset \(A\) of \(\Omega\).
| Description | Set form |
|---|---|
| Any number of pips appears | \(\omega \in A = \Omega\) |
| No number of pips appears | \(\omega \in A = \emptyset\) |
| A number of pips greater than 4 appears | \(\omega \in A = \{5,6\}\) |
| An even number of pips appears | \(\omega \in A = \{2,4,6\}\) |
| A six appears | \(\omega \in A = \{6\}\) |
| A one, a three, or a six appears | \(\omega \in A = \{1,3,6\}\) |
This completes the definition of the measurable space \((\Omega, \mathcal{A})\) in the modeling of throwing a die.
As described in Section 19.2, a probability measure \(\mathbb{P}\) can be defined by specifying \(\mathbb{P}(\{\omega\})\) for all \(\omega \in \Omega\). For the model of a fair die, one would choose \[\begin{equation} \mathbb{P}(\{\omega\}) := \frac{1}{|\Omega|} := 1/6 \mbox{ for all } \omega \in \Omega. \end{equation}\] For a model of a biased die that favors throwing a six, one could, for example, define \[\begin{equation} \mathbb{P}(\{\omega\}) := \frac{1}{8} \mbox{ for } \omega \in \{1,2,3,4,5\} \mbox{ and } \mathbb{P}(\{6\}) := \frac{3}{8}. \end{equation}\] In the case of the fair die, for example, the probability of the event “An even number of pips appears” is then obtained with the \(\sigma\)-additivity of \(\mathbb{P}\) as \[\begin{equation} \mathbb{P}(\{2,4,6\}) = \mathbb{P}(\{2\} \cup \{4\} \cup \{6\}) = \mathbb{P}(\{2\}) + \mathbb{P}(\{4\}) + \mathbb{P}(\{6\}) = \frac{1}{6} + \frac{1}{6} + \frac{1}{6} = \frac{3}{6}. \end{equation}\] In the case of the above model of a biased die, by contrast, the probability of the same event is \[\begin{equation} \mathbb{P}(\{2,4,6\}) = \mathbb{P}(\{2\} \cup \{4\} \cup \{6\}) = \mathbb{P}(\{2\}) + \mathbb{P}(\{4\}) + \mathbb{P}(\{6\}) = \frac{1}{8} + \frac{1}{8} + \frac{3}{8} = \frac{5}{8}. \end{equation}\] The event considered has a higher probability in the model of the biased die than in the model of the fair die, which is intuitively sensible because six is an even number.
Simultaneous throwing of a blue and a red die
We now want to model the simultaneous throwing of a blue and a red die. For this purpose it is sensible to define the outcome set as \[\begin{equation} \Omega := \{(r,b)| r \in \{1,2,3,4,5,6\}, b \in \{1,2,3,4,5,6\}\} \end{equation}\] with cardinality \(|\Omega| = 36\), where \(r\) is to represent the number of pips on the red die and \(b\) the number of pips on the blue die.
Again, choosing the power set of \(\Omega\) as the \(\sigma\)-algebra is suitable, so we again define \(\mathcal{A} := \mathcal{P}(\Omega)\). The number of events possible in this model is \(|\mathcal{A}| = 2^{|\Omega|} = 2^{36} = 68,719,476,736\). In Table 19.2 we list six of these events in their verbal description and as a subset \(A\) of \(\Omega\).
| Description | Set form |
|---|---|
| A three appears on the red die | \(\omega \in A = \{(3,1),(3,2),(3,3),(3,4),(3,5),(3,6)\}\) |
| A three appears on the blue die | \(\omega \in A = \{(1,3),(2,3),(3,3),(4,3),(5,3),(6,3)\}\) |
| A three appears on both dice | \(\omega \in A = \{(3,3)\}\) |
| Doubles appear | \(\omega \in A = \{(1,1),(2,2),(3,3),(4,4),(5,5),(6,6)\}\) |
| The sum of the numbers that appear is four | \(\omega \in A = \{(1,3),(3,1),(2,2)\}\) |
The definition of the measurable space \((\Omega, \mathcal{A})\) is therefore complete. A probability measure \(\mathbb{P}\) can again be specified by defining \(\mathbb{P}(\{\omega\})\) for all \(\omega \in \Omega\). For the model of two fair dice, one would choose \[\begin{equation} \mathbb{P}(\{\omega\}) := \frac{1}{|\Omega|} = \frac{1}{36} \mbox{ for all } \omega \in \Omega. \end{equation}\] Under this probability measure, the probability of the event “The sum of the numbers that appear is four”, for example, is obtained with the \(\sigma\)-additivity of \(\mathbb{P}\) as \[\begin{align*} \begin{split} \mathbb{P}\left(\{(1,3),(3,1),(2,2)\}\right) & = \mathbb{P}\left(\{(1,3)\}\cup \{(3,1)\} \cup \{(2,2)\}\right) \\ & = \mathbb{P}\left(\{(1,3)\}\right) + \mathbb{P}\left(\{(3,1)\}\right) + \mathbb{P}\left(\{(2,2)\}\right) \\ & = 1/36 + 1/36 + 1/36 \\ & = 1/12. \end{split} \end{align*}\]
Tossing a coin
We model tossing a coin, one side of which shows heads and the other side tails. It is sensible to define the outcome set as \(\Omega := \{H,T\}\), where \(H\) represents “heads” and \(T\) represents “tails”. However, any other binary definition of \(\Omega\) would also be possible, e.g. \(\Omega := \{0,1\}, \Omega := \{-1,1\}\), or \(\Omega := \{1,2\}\).
The power set \(\mathcal{A} := \mathcal{P}(\Omega)\) contains all possible events. In this case, we can easily list the entire set system \(\mathcal{A}\), as shown in Table 19.3.
| Description | Set form |
|---|---|
| Neither heads nor tails | \(\omega \in A = \emptyset\) |
| Heads appears | \(\omega \in A = \{H\}\) |
| Tails appears | \(\omega \in A = \{T\}\) |
| Heads or tails appears | \(\omega \in A = \{H,T\}\) |
The definition of the measurable space \((\Omega, \mathcal{A})\) is therefore complete.
A probability measure \(\mathbb{P}\) can again be specified by defining \(\mathbb{P}(\{\omega\})\) for all \(\omega \in \Omega\). The normalization of \(\Omega\) here implies in particular that \[\begin{equation} \mathbb{P}(\Omega) = 1 \Leftrightarrow \mathbb{P}(\{H\}) + \mathbb{P}(\{T\}) = 1 \Leftrightarrow \mathbb{P}(\{T\}) = 1 - \mathbb{P}(\{H\}). \end{equation}\] Thus, when the probability of the elementary event \(\{H\}\) is specified, the probability of the elementary event \(\{T\}\) is immediately specified as well, and conversely, of course. For the model of a fair coin, one would choose \(\mathbb{P}(\{H\}) = \mathbb{P}(\{T\}) := 1/2\). In this case, the probabilities of all possible events are \[\begin{equation} \mathbb{P}(\emptyset) = 0, \mathbb{P}(\{H\}) = 1/2, \mathbb{P}(\{T\}) = 1/2 \mbox{ and } \mathbb{P}(\{H,T\}) = 1. \end{equation}\]
Tossing a coin twice
We model tossing a coin twice. Based on the model of a single coin toss, it is sensible to define the outcome set as \(\Omega := \{HH,HT,TH,TT\}\). The power set \(\mathcal{A} := \mathcal{P}(\Omega)\) again contains all \(2^{|\Omega|} = 2^4 = 16\) possible events. In the table below, we list four of them.
| Description | Set form |
|---|---|
| Heads appears on the first toss | \(\omega \in A = \{HH,HT\}\) |
| Heads appears on the second toss | \(\omega \in A = \{HH,TH\}\) |
| Heads does not appear | \(\omega \in A = \{TT\}\) |
| Tails appears at least once | \(\omega \in A = \{HT, TH, TT\}\) |
The definition of the measurable space \((\Omega, \mathcal{A})\) is therefore complete. A probability measure \(\mathbb{P}\) can again be specified by defining \(\mathbb{P}(\{\omega\})\) for all \(\omega \in \Omega\). For the model of tossing a fair coin twice, one would define \[\begin{equation} \mathbb{P}(\{HH\}) = \mathbb{P}(\{HT\}) = \mathbb{P}(\{TH\})= \mathbb{P}(\{TT\}) := \frac{1}{4}. \end{equation}\]
Value of a BDI-II item
We model the value of a patient for the item “Sadness” of the English version of the Beck Depression Inventory II (Hautzinger et al. (2006)),
- I do not feel sad.
- I feel sad much of the time.
- I am sad all the time.
- I am so sad or unhappy that I can’t stand it.
The random process under consideration is therefore a patient’s response to this item. For modeling, the outcome set \(\Omega := \{0,1,2,3\}\) is then suitable. For a first patient, for example, the realization is \(3 \in \Omega\), for a second patient the realization is \(0 \in \Omega\), for a third patient \(1 \in \Omega\), and so on. As the event system \(\mathcal{A}\), the power set is again suitable. In the present case, it has cardinality \[\begin{equation} |\mathcal{P}(\Omega)| = 2^{|\Omega|} = 2^4 = 16 \end{equation}\] and can easily be written down completely as shown in Table 19.5.
| Given response | Set form |
|---|---|
| None | \(\emptyset\) |
| “I do not feel sad” | \(\{0\}\) |
| “I feel sad much of the time” | \(\{1\}\) |
| “I am sad all the time” | \(\{2\}\) |
| “I am so sad or unhappy that I can’t stand it” | \(\{3\}\) |
| “I do not feel sad” or “I feel sad much of the time” | \(\{0,1\}\) |
| “I do not feel sad” or “I am sad all the time” | \(\{0,2\}\) |
| “I do not feel sad” or “I am so sad or unhappy that I can’t stand it” | \(\{0,3\}\) |
| “I feel sad much of the time” or “I am sad all the time” | \(\{1,2\}\) |
| “I feel sad much of the time” or “I am so sad or unhappy that I can’t stand it” | \(\{1,3\}\) |
| “I am sad all the time” or “I am so sad or unhappy that I can’t stand it” | \(\{2,3\}\) |
| Not “I am so sad or unhappy that I can’t stand it” | \(\{0,1,2\}\) |
| Not “I am sad all the time” | \(\{0,1,3\}\) |
| Not “I feel sad much of the time” | \(\{0,2,3\}\) |
| Not “I do not feel sad” | \(\{1,2,3\}\) |
| Any response option | \(\{0,1,2,3\}\) |
Now suppose that the probabilities of the item responses are known as \[\begin{equation} \mathbb{P}(\{0\}) := \frac{2}{10},\quad \mathbb{P}(\{1\}) := \frac{3}{10},\quad \mathbb{P}(\{2\}) := \frac{4}{10},\quad \mathbb{P}(\{3\}) := \frac{1}{10}. \end{equation}\] Then these probabilities define, in the sense of a probability function, \[\begin{equation} \pi: \Omega \to [0,1], \omega \mapsto \pi(\omega) =: \mathbb{P}(\{\omega\}) \end{equation}\] a probability measure, and the definition of the probability space model \((\Omega, \mathcal{A},\mathbb{P})\) for the scenario under consideration is complete. With the help of the \(\sigma\)-additivity of \(\mathbb{P}\), for example, the probability of an item response greater than 1 can now be determined: \[\begin{equation} \mathbb{P}\left(\{2,3\}\right) = \mathbb{P}\left(\{2\} \cup \{3\}\right) = \mathbb{P}\left(\{2\}\right) + \mathbb{P}\left(\{3\}\right) = \frac{4}{10} + \frac{1}{10} = \frac{5}{10} = \frac{1}{2} \end{equation}\] Similarly, for example, the probability that a patient gives one of the two item responses “(0) I do not feel sad” or “(3) I am so sad or unhappy that I can’t stand it” is \[\begin{equation} \mathbb{P}\left(\{0,3\}\right) = \mathbb{P}\left(\{0\} \cup \{3\}\right) = \mathbb{P}\left(\{0\}\right) + \mathbb{P}\left(\{3\}\right) = \frac{2}{10} + \frac{1}{10} = \frac{3}{10}. \end{equation}\]
19.4 Bibliographic remarks
Andrey Kolmogorov’s monograph “Grundbegriffe der Wahrscheinlichkeitsrechnung” (Kolmogoroff (1933)) symbolizes the foundation of modern probability calculus. In addition to the axiomatic introduction of the probability space model discussed here, Kolmogoroff (1933) considers many other aspects of probability calculus and thus offers a readable introduction to the whole field of probability theory. Of course, the approach formulated by Kolmogoroff (1933) is only a provisional end product of the long historical development of probability theory. After all, the development of the mathematical modeling of random processes by no means came to an end with Kolmogoroff (1933). Later works in the 20th century concerned in particular the interpretation of the concept of probability (cf. De Finetti (1975)) or introduced generalized quantitative measures of subjective uncertainty (cf. Walley (1991)). A current overview of the interpretation of the concept of probability and its formal foundations is given by Hájek (2019).
Study questions
- Explain the concept of a random process.
- State the definition of the concept of a \(\sigma\)-algebra.
- State the definition of the concept of a probability measure.
- State the definition of the concept of a probability space.
- Explain the concept of the outcome set \(\Omega\).
- Explain the tacit mechanics of the probability space model.
- Explain the concept of an event \(A \in \mathcal{A}\).
- Explain the concept of the event system \(\mathcal{A}\).
- Which \(\sigma\)-algebra is sensibly chosen for a probability space with finite outcome set?
- Explain the concept of the probability measure \(\mathbb{P}\).
- State the definition of the concept of a probability function.
- Why is the concept of a probability function helpful when modeling a random process by a probability space with finite outcome set?
- Explain the modeling of throwing a die using a probability space.
- Explain the modeling of the simultaneous throwing of a red and a blue die using a probability space.
Study question answers
- A random process is a phenomenon of reality that cannot be predicted with absolute certainty or is associated with uncertainty for humans. See the introduction to probability theory.
- See Definition 19.1 regarding \(\mathcal{A}\).
- See Definition 19.1 regarding \(\mathbb{P}\).
- See Definition 19.1.
- The outcome set contains the elementary events possible in a run of the modeled random process; see the explanations of Outcome set and mechanics.
- See the explanations of Outcome set and mechanics.
- See the explanations of Events and event system.
- See the explanations of Events and event system.
- The power set of the outcome set; it contains all events that are possible in principle; see also Theorem 19.1.
- See the explanations of Probability measure \(\mathbb{P}\).
- See Definition 19.2.
- By defining the function values of a probability function, a probability measure can be specified; see also Section 19.3.
- See the explanations of Throwing one die.
- See the explanations of Simultaneous throwing of a blue and a red die.