20 Elementary probabilities
In this section, we introduce two elementary forms of probability: the joint probability of two events and the conditional probability of an event. Intuitively, joint probability refers to the probability that two events \(A\) and \(B\) occur “simultaneously”, whereas conditional probability refers to the probability that event \(A\) occurs “when we know that another event \(B\) has occurred”. If whether \(B\) has occurred makes no difference to the probability of \(A\), we call \(A\) and \(B\) independent events. Intuitively, independence means that knowing one event has occurred does not change the probability of the other event.
20.1 Joint probabilities
The concept of the joint probability of two events is defined as follows.
Definition 20.1 (Joint probability) Let \((\Omega, \mathcal{A}, \mathbb{P})\) be a probability space and let \(A,B \in \mathcal{A}\). Then \[\begin{equation} \mathbb{P}(A \cap B) \end{equation}\] is called the joint probability of \(A\) and \(B\).
As noted above, \(\mathbb{P}(A \cap B)\) corresponds to the probability that the events \(A\) and \(B\) occur “simultaneously”. This is best understood against the background of the mechanics of the probability space model. According to this mechanics, \(\mathbb{P}(A \cap B)\) is precisely the probability that, in one run of a random process, an \(\omega\) is realized for which both \(\omega \in A\) and \(\omega \in B\) hold.
Example 20.1 (Joint probability for a fair die) As a first example of a joint probability of two events, we again consider the probability space model of throwing a fair die. Let \(A\) be the event “An even number of pips appears”, that is, \(A := \{2,4,6\}\), and let \(B\) be the event “A number of pips greater than three appears”, that is, \(B := \{4,5,6\}\). Set-theoretically, we then have \[\begin{equation} A \cap B = \{2,4,6\} \cap \{4,5,6\} = \{4,6\}. \end{equation}\] The interpretation of \(A \cap B = \{4,6\}\) is exactly “An even number of pips appears and this number of pips is greater than three”. Assuming fairness of the die, that is, for \(\mathbb{P}(\{4\}) = \mathbb{P}(\{6\}) := 1/6\), we can easily calculate the probability of this event using the \(\sigma\)-additivity of \(\mathbb{P}\). We obtain \[\begin{equation} \begin{aligned} \begin{split} \mathbb{P}(A \cap B) & = \mathbb{P}(\{2,4,6\} \cap \{4,5,6\}) \\ & = \mathbb{P}(\{4,6\}) \\ & = \mathbb{P}(\{4\}) + \mathbb{P}(\{6\}) \\ & = \frac{1}{6} + \frac{1}{6} \\ & = \frac{1}{3}. \end{split} \end{aligned} \end{equation}\]
When considering joint probabilities, it is important not to confuse the joint probability \(\mathbb{P}(A\cap B)\) with the probability \(\mathbb{P}(A\cup B)\) of the event \(A\cup B\). Recall that the union of two sets, \(\cup\), corresponds to inclusive or, that is, and/or (see Section 1.3 and Section 2.2). By contrast, the symmetric difference \(A\mathbin{\Delta}B\) corresponds to exclusive or, that is, “either \(A\) or \(B\), but not both”.
When thinking about joint probabilities, it is of course important not to confuse the joint probability \(\mathbb{P}(A \cap B)\) with the probability \(\mathbb{P}(A \cup B)\) of the event \(A \cup B\). Recall that the union of two sets \(\cup\) corresponds to the inclusive or, that is, to an and/or (cf. Section 1.3 and Section 2.2). The event \(A \cup B\) therefore corresponds to the event that event \(A\) and/or event \(B\) occurs. In particular, \(\omega \in A \cup B\) is already satisfied if for the outcome of one run of a random process only \(\omega \in A\) or only \(\omega \in B\) holds. Concretely, for the events \(A := \{2,4,6\}\) and \(B := \{4,5,6\}\) from the die example above, we obtain \[\begin{equation} A \cup B = \{2,4,6\} \cup \{4,5,6\} = \{2,4,5,6\} \end{equation}\] with the interpretation “An even number of pips appears and/or a number of pips greater than three appears”. For the corresponding probability we obtain \[\begin{equation} \mathbb{P}(\{2,4,5,6\}) = \frac{2}{3}, \end{equation}\] so in this case apparently \(\mathbb{P}(A \cap B) \neq \mathbb{P}(A \cup B)\).
The following theorem collects useful rules for calculating probabilities that follow directly from set operations and the \(\sigma\)-additivity of probability measures. The required set-theoretic statements were proved and visualized in Chapter 2.
Theorem 20.1 (Further properties of probabilities) Let \((\Omega, \mathcal{A}, \mathbb{P})\) be a probability space and let \(A,B \in \mathcal{A}\) be events. Then
- \(\mathbb{P}(A^c) = 1 - \mathbb{P}(A)\).
- \(A \subset B \Rightarrow \mathbb{P}(A) \le \mathbb{P}(B)\).
- \(\mathbb{P}(A \cap B^c) = \mathbb{P}(A) - \mathbb{P}(A \cap B)\)
- \(\mathbb{P}(A \cup B) = \mathbb{P}(A) + \mathbb{P}(B) - \mathbb{P}(A \cap B)\).
Proof. We use the statements proved in Theorem 2.3 and Theorem 2.4, together with normalization and \(\sigma\)-additivity of \(\mathbb{P}\). Here, the universal set is \(O=\Omega\), and we substitute events \(A\) and \(B\) for sets \(M\) and \(N\).
For 1: Statements 1 and 2 of Theorem 2.3 give \(A^c\cup A=\Omega\) and \(A^c\cap A=\emptyset\). Normalization and \(\sigma\)-additivity of \(\mathbb{P}\) then imply \[\begin{equation} \mathbb{P}(\Omega) = 1 \Leftrightarrow \mathbb{P}(A^c \cup A) = 1 \Leftrightarrow \mathbb{P}(A^c) + \mathbb{P}(A) = 1 \Leftrightarrow \mathbb{P}(A^c) = 1 - \mathbb{P}(A). \end{equation}\]
For 2: Statement 1 of Theorem 2.4 gives \[\begin{equation} A \subset B \Rightarrow B = A \cup (B \cap A^c) \mbox{ with } A \cap (B \cap A^c) = \emptyset. \end{equation}\] The \(\sigma\)-additivity of \(\mathbb{P}\) then implies \[\begin{equation} \mathbb{P}(B) = \mathbb{P}(A) + \mathbb{P}(B \cap A^c). \end{equation}\] With \(\mathbb{P}(B \cap A^c) \ge 0\) it follows that \(\mathbb{P}(A) \le \mathbb{P}(B)\).
Two events can have the same probability even if one is a proper subset of the other. For example, consider a probability space with \(\Omega=\mathbb{R}\) in which the events \(A=[0,1[\) and \(B=[0,1]\) satisfy \(\mathbb{P}(B)>0\) and \(\mathbb{P}(\{1\})=0\). Then \(A\) is a proper subset of \(B\), but \[\begin{equation} \mathbb{P}(B)=\mathbb{P}(A)+\mathbb{P}(\{1\})=\mathbb{P}(A). \end{equation}\] The standard normal distribution, introduced later, provides a concrete example.
For 3: Statement 2 of Theorem 2.4 gives \[\begin{equation} (A \cap B) \cap (A \cap B^c) = \emptyset \mbox{ and } A = (A \cap B) \cup (A \cap B^c). \end{equation}\] With the \(\sigma\)-additivity of \(\mathbb{P}\) it follows that \[\begin{equation} \begin{split} \mathbb{P}(A) = \mathbb{P}(A \cap B) + \mathbb{P}(A \cap B^c) \Leftrightarrow \mathbb{P}(A \cap B^c) = \mathbb{P}(A) - \mathbb{P}(A \cap B) \end{split} \end{equation}\]
For 4: Statement 3 of Theorem 2.4 gives \[\begin{equation} B \cap (A \cap B^c) = \emptyset \mbox{ and } A \cup B = B \cup (A \cap B^c). \end{equation}\] With the \(\sigma\)-additivity of \(\mathbb{P}\) it follows that \[\begin{equation} \mathbb{P}(A \cup B) = \mathbb{P}(B) + \mathbb{P}(A \cap B^c). \end{equation}\] With 3. it then follows that \[\begin{equation} \mathbb{P}(A \cup B) = \mathbb{P}(B) + \mathbb{P}(A) - \mathbb{P}(A \cap B). \end{equation}\]
20.2 Conditional probabilities
We now turn to the concept of conditional probability.
Definition 20.2 (Conditional probability) Let \((\Omega,\mathcal{A}, \mathbb{P})\) be a probability space and let \(A, B\in \mathcal{A}\) be events with \(\mathbb{P}(B) > 0\). The conditional probability of event \(A\) given event \(B\) is defined as \[\begin{equation} \mathbb{P}(A|B) := \frac{\mathbb{P}(A \cap B)}{\mathbb{P}(B)}. \end{equation}\]
The conditional probability \(\mathbb{P}(A|B)\) of event \(A\) given event \(B\) is the joint probability \(\mathbb{P}(A\cap B)\) scaled by \(1/\mathbb{P}(B)\). Thus, specifying the joint probability \(\mathbb{P}(A\cap B)\) and the probability \(\mathbb{P}(B)>0\) in a probabilistic model also specifies the conditional probability \(\mathbb{P}(A|B)\). For \(\mathbb{P}(A)>0\) and \(\mathbb{P}(B)>0\), the two conditional probabilities have the same numerator but potentially different denominators (see Herzog & Ostwald (2013)): \[\begin{equation} \mathbb{P}(A|B)=\frac{\mathbb{P}(A\cap B)}{\mathbb{P}(B)} \qquad\text{and}\qquad \mathbb{P}(B|A)=\frac{\mathbb{P}(A\cap B)}{\mathbb{P}(A)}. \end{equation}\] If \(\mathbb{P}(A\cap B)>0\), then \(\mathbb{P}(A)\neq\mathbb{P}(B)\) implies \(\mathbb{P}(A|B)\neq\mathbb{P}(B|A)\). If \(\mathbb{P}(A\cap B)=0\), however, both conditional probabilities are zero. Finally, the conditional probability in Definition 20.2 can be generalized to the case \(\mathbb{P}(B)=0\), but this is technically demanding. We refer to the advanced literature, for example Meintrup & Schäffler (2005) and Schmidt (2009).
Examples
Conditional probability for a fair die: formal calculation
We again consider the probability space model \((\Omega,\mathcal A,\mathbb P)\) for throwing a fair die, with \(\Omega=\{1,2,3,4,5,6\}\), \(\mathcal A=\mathcal P(\Omega)\), and \(\mathbb P(\{\omega\})=1/6\) for every \(\omega\in\Omega\). We want to calculate the conditional probability of “An even number appears” given “A number greater than three appears”. Denote the two events by \[\begin{equation} A:=\{2,4,6\} \qquad\text{and}\qquad B:=\{4,5,6\}. \end{equation}\] Since each event contains three outcomes, additivity of the probability measure gives \[\begin{equation} \mathbb P(A)=\frac16+\frac16+\frac16=\frac12 \qquad\text{and}\qquad \mathbb P(B)=\frac16+\frac16+\frac16=\frac12. \end{equation}\]
The intersection \(A\cap B=\{4,6\}\) corresponds to the event “An even number greater than three appears”. Its probability is \[\begin{equation} \mathbb P(A\cap B) =\mathbb P(\{4,6\}) =\mathbb P(\{4\})+\mathbb P(\{6\}) =\frac16+\frac16 =\frac13. \end{equation}\] Since \(\mathbb P(B)=1/2>0\), the conditional probability of \(A\) given \(B\) is defined. Substituting into its defining equation gives \[\begin{equation} \mathbb P(A\mid B) :=\frac{\mathbb P(A\cap B)}{\mathbb P(B)} =\frac{1/3}{1/2} =\frac23. \end{equation}\]
The conditional probability is thus determined by the probability space model and the two events. We now consider two different interpretations of the calculated value.
Bayesian interpretation: updating with information
Someone throws a fair die under a cup and looks at the number. I cannot see it. The person asks how likely I think it is that an even number has appeared. Based on the fair-die model, I assign probability \(1/6\) to each of the six possible outcomes. For the event \(A=\{2,4,6\}\), this gives \[\begin{equation} \mathbb P(A)=\frac12. \end{equation}\] My answer is therefore: “The probability is \(1/2\).” The number on the die has already been determined. Here, probability describes my uncertainty about which outcome was realized.
As agreed, the person then truthfully tells me that the number is greater than three. I now know that the outcome lies in the event \(B=\{4,5,6\}\). Given this information, I describe my assessment that an even number appeared by the conditional probability \(\mathbb P(A\mid B)\). As calculated above, the fair-die model gives \(\mathbb P(A\cap B)=1/3\) and \(\mathbb P(B)=1/2\). Hence \[\begin{equation} \mathbb P(A\mid B) =\frac{\mathbb P(A\cap B)}{\mathbb P(B)} =\frac{1/3}{1/2} =\frac23. \end{equation}\] My updated answer is therefore: “The probability is \(2/3\).”
The information has not changed the number under the cup. What has changed is my state of information and hence my assessment that an even number appeared. In this Bayesian interpretation, probability describes my degree of belief that an event has occurred. The conditional probability \(\mathbb P(A\mid B)\) describes this belief after receiving the information that \(B\) has occurred.
Frequentist interpretation: relative frequency in a subsequence
For the frequentist interpretation, consider repeated throws of a fair die. We are interested in how frequently an even number appears among those throws whose number is greater than three. We simulate 120 independent throws. Our simulation produces the following numbers, read row by row:
\[\begin{equation} {\small\color{gray} \begin{array}{cccccccccccccccccccc} {\color{black}\mathbf{5}} & 2 & {\color{black}\mathbf{5}} & 3 & {\color{blue}\mathbf{4}} & 1 & {\color{blue}\mathbf{6}} & {\color{black}\mathbf{5}} & {\color{black}\mathbf{5}} & 3 & {\color{blue}\mathbf{6}} & 3 & 1 & 3 & 3 & {\color{blue}\mathbf{4}} & 2 & {\color{blue}\mathbf{6}} & 2 & 2 \\ 1 & 1 & 3 & 2 & {\color{blue}\mathbf{6}} & 3 & {\color{blue}\mathbf{6}} & {\color{blue}\mathbf{4}} & {\color{black}\mathbf{5}} & {\color{blue}\mathbf{4}} & 3 & 1 & 1 & {\color{blue}\mathbf{4}} & 3 & {\color{blue}\mathbf{4}} & {\color{blue}\mathbf{4}} & 2 & {\color{black}\mathbf{5}} & 2 \\ 3 & {\color{blue}\mathbf{6}} & {\color{blue}\mathbf{4}} & {\color{black}\mathbf{5}} & {\color{black}\mathbf{5}} & {\color{blue}\mathbf{6}} & 2 & {\color{blue}\mathbf{6}} & 2 & 1 & {\color{blue}\mathbf{4}} & 1 & 2 & {\color{blue}\mathbf{6}} & 3 & {\color{blue}\mathbf{4}} & {\color{black}\mathbf{5}} & 2 & {\color{blue}\mathbf{6}} & 2 \\ {\color{black}\mathbf{5}} & {\color{blue}\mathbf{6}} & 1 & 1 & {\color{blue}\mathbf{4}} & 2 & 1 & {\color{blue}\mathbf{4}} & {\color{blue}\mathbf{4}} & {\color{blue}\mathbf{4}} & {\color{black}\mathbf{5}} & 1 & 2 & 2 & 1 & 1 & {\color{blue}\mathbf{4}} & {\color{blue}\mathbf{6}} & {\color{blue}\mathbf{6}} & 3 \\ 1 & {\color{blue}\mathbf{6}} & {\color{blue}\mathbf{6}} & 1 & 2 & {\color{blue}\mathbf{4}} & {\color{black}\mathbf{5}} & 3 & 2 & 3 & 1 & 3 & {\color{black}\mathbf{5}} & 3 & {\color{black}\mathbf{5}} & 3 & 2 & {\color{black}\mathbf{5}} & {\color{black}\mathbf{5}} & 1 \\ 3 & 3 & {\color{blue}\mathbf{6}} & 2 & {\color{blue}\mathbf{6}} & {\color{blue}\mathbf{6}} & 1 & 1 & {\color{blue}\mathbf{6}} & {\color{blue}\mathbf{6}} & {\color{black}\mathbf{5}} & 3 & {\color{blue}\mathbf{4}} & {\color{blue}\mathbf{4}} & 2 & 3 & 1 & 1 & {\color{blue}\mathbf{4}} & 3 \\ \end{array}} \end{equation}\]
From this sequence, select the throws whose outcomes belong to \(B=\{4,5,6\}\). These are highlighted in bold. The excluded throws showing \(1\), \(2\), or \(3\) are gray. In total, 56 of the 120 throws satisfy condition \(B\).
Among these 56 throws, count those for which \(A=\{2,4,6\}\) has also occurred. These are the blue throws showing a \(4\) or \(6\), that is, an outcome in \(A\cap B=\{4,6\}\). There are 39 such throws in our example. The conditional relative frequency is therefore \[\begin{equation} \widehat{\mathbb P}_{120}(A\mid B) :=\frac{\text{Number of throws with }A\cap B} {\text{Number of throws with }B} =\frac{39}{56} \approx0{,}696. \end{equation}\] The denominator is the number of selected throws. We are thus determining the proportion of even numbers within the subsequence specified by \(B\).
Based on the definition of the conditional probability of an event under the condition of the occurrence of a fixed event \(B\), one can define a probability measure that gives the probabilities of all events of the event system under the condition of the occurrence of a fixed event \(B\). This probability measure is called the conditional probability given event \(B\).
Theorem 20.2 (Conditional probability) For a fixed \(B \in \mathcal{A}\) with \(\mathbb{P}(B) > 0\), let \[\begin{equation} \mathbb{P}(\cdot|B) : \mathcal{A} \to [0,1], A \mapsto \mathbb{P}(A|B) := \frac{\mathbb{P}(A \cap B)}{\mathbb{P}(B)} \end{equation}\] Then \(\mathbb{P}(\cdot|B)\) is a probability measure and is called the conditional probability given event \(B\).
Proof. We verify the defining properties of a probability measure.
. Non-negativity. \(\mathbb{P}(A|B) \ge 0\) for all \(A \in \mathcal{A}\).
By the definition of conditional probability, \(\mathbb{P}(A|B)=\mathbb{P}(A\cap B)/\mathbb{P}(B)\), with \(\mathbb{P}(B)>0\). By Theorem 19.4 and the definition of \(\mathbb{P}\), we have \(\mathbb{P}(A\cap B)\ge0\), which proves the statement.
. Normalization. \(\mathbb{P}(\Omega|B) = 1\).
We have \[\begin{equation} \mathbb{P}(\Omega|B) = \frac{\mathbb{P}(\Omega \cap B)}{\mathbb{P}(B)} = \frac{\mathbb{P}(B)}{\mathbb{P}(B)} = 1. \end{equation}\]
. \(\sigma\)-additivity. \(\mathbb{P}\left(\cup_{i=1}^\infty A_i|B \right) = \sum_{i=1}^\infty \mathbb{P}(A_i|B)\) for pairwise disjoint \(A_1,A_2,... \in \mathcal{A}\).
By Theorem 2.6, we first have \[\begin{equation} \left(\bigcup_{i=1}^{\infty}A_i\right)\cap B =\bigcup_{i=1}^{\infty}(A_i\cap B). \end{equation}\] For \(i\neq j\), we have \(A_i\cap A_j=\emptyset\). By Theorem 2.7, this implies \[\begin{equation} (A_i\cap B)\cap(A_j\cap B)=\emptyset. \end{equation}\] The sets \(A_i\cap B\) are therefore also pairwise disjoint. The \(\sigma\)-additivity of \(\mathbb{P}\) then gives \[\begin{equation} \begin{split} \mathbb{P}\left(\cup_{i=1}^\infty A_i|B \right) & = \frac{\mathbb{P}\left(\left(\cup_{i=1}^\infty A_i\right) \cap B\right)}{\mathbb{P}(B)} \\ & = \frac{\mathbb{P}\left(\cup_{i=1}^\infty \left(A_i \cap B\right) \right)}{\mathbb{P}(B)} \\ & = \frac{\sum_{i=1}^\infty \mathbb{P}\left(A_i \cap B\right)}{\mathbb{P}(B)} \\ & = \sum_{i=1}^\infty \frac{\mathbb{P}\left(A_i \cap B\right)}{\mathbb{P}(B)} \\ & = \sum_{i=1}^\infty \mathbb{P}(A_i|B). \end{split} \end{equation}\]
Note that for the conditional probability \(\mathbb{P}(\cdot|B)\), the calculation rules of probability theory apply to events on the left of the conditioning bar.
For \(\mathbb{P}(B)>0\), the mapping \(\mathbb{P}(\cdot|B)\) is a probability measure. In particular, \[\begin{equation} \mathbb{P}(\Omega|B)=1. \end{equation}\] By contrast, \[\begin{equation} \mathbb{P}(\Omega\cap B)=\mathbb{P}(B). \end{equation}\] For \(0<\mathbb{P}(B)<1\), the mapping \(\mathbb{P}(\cdot\cap B)\) is therefore not normalized and hence is not a probability measure. For \(\mathbb{P}(B)=1\), the two mappings agree.
At the end of this section, we consider three technical consequences of the definition of conditional probability, which we formulate as theorems.
The first theorem concerns the relation between joint and conditional probabilities and reiterates how joint probabilities can be calculated from conditional and total probabilities.
Theorem 20.3 (Joint and conditional probabilities) Let \((\Omega,\mathcal{A}, \mathbb{P})\) be a probability space and let \(A,B\in \mathcal{A}\) with \(\mathbb{P}(A) > 0\) and \(P(B)>0\). Then \[\begin{equation} \mathbb{P}(A \cap B) = \mathbb{P}(A|B)\mathbb{P}(B) = \mathbb{P}(B|A)\mathbb{P}(A). \end{equation}\]
Proof. With the definition of the respective conditional probability, it follows directly that \[\begin{equation} \mathbb{P}(A|B) = \frac{\mathbb{P}(A \cap B)}{\mathbb{P}(B)} \Leftrightarrow \mathbb{P}(A \cap B) =\mathbb{P}(A|B)\mathbb{P}(B) \end{equation}\] and \[\begin{equation} \mathbb{P}(B|A) = \frac{\mathbb{P}(B \cap A)}{\mathbb{P}(A)} \Leftrightarrow \mathbb{P}(A \cap B) =\mathbb{P}(B|A)\mathbb{P}(A). \end{equation}\]
Just as specifying \(\mathbb{P}(A \cap B)\) and \(\mathbb{P}(A)\) specifies the conditional probability \(\mathbb{P}(B|A)\), specifying \(\mathbb{P}(A)\) and \(\mathbb{P}(B|A)\) therefore specifies the joint probability \(\mathbb{P}(A \cap B)\).
The following so-called law of total probability states how unconditional, so-called total probabilities can be calculated based on joint probabilities.
Theorem 20.4 (Law of total probability) Let \((\Omega,\mathcal{A},\mathbb{P})\) be a probability space and let \(A_1,...,A_k\) with \(\mathbb{P}(A_i) > 0\) be a partition of \(\Omega\). Then for every \(B \in \mathcal{A}\), \[\begin{equation} \mathbb{P}(B) = \sum_{i=1}^k \mathbb{P}(B \cap A_i) = \sum_{i=1}^k \mathbb{P}(B|A_i)\mathbb{P}(A_i). \end{equation}\]
Proof. For \(i = 1,...,k\), let \(C_i := B \cap A_i\), so that \(\cup_{i=1}^k C_i = B\) and \(C_i \cap C_j = \emptyset\) for \(1 \le i,j \le k,i \neq j\). We illustrate these definitions in Figure 20.1 using a Venn diagram.
Thus, with the definition of conditional probability for \(\mathbb{P}(A_i) > 0\), we have \[\begin{equation} \mathbb{P}(B) = \sum_{i=1}^k \mathbb{P}(C_i) = \sum_{i=1}^k \mathbb{P}(B \cap A_i) = \sum_{i=1}^k \mathbb{P}(B|A_i)\mathbb{P}(A_i). \end{equation}\]
Intuitively, \(\mathbb{P}(B)\) is the weighted sum of the conditional probabilities \(\mathbb{P}(B|A_i)\), with the unconditional probabilities \(\mathbb{P}(A_i)\) as weights.
Calculating an unconditional probability by summing joint probabilities, \[\begin{equation} \mathbb{P}(B)=\sum_{i=1}^{k}\mathbb{P}(B\cap A_i), \end{equation}\] is also called marginalization in the context of random variables.
The requirement \(\mathbb{P}(A_i)>0\) for all \(i=1,\ldots,k\) is necessary to express the joint probabilities using conditional probabilities (Definition 20.2).
Example 20.2 (The law of total probability for a fair die) As an example of representing the probability of an event using the law of total probability, we consider the probability space model of the fair die. For the outcome space \(\Omega := \{1,2,3,4,5,6\}\), we consider the event \(B := \{2,4,6\}\) (“An even number appears”) with probability \(\mathbb{P}(B) = \frac{1}{2}\) as well as the partition \(A_1 := \{1,2\}\), \(A_2 := \{3\}\), \(A_3 := \{4,5\}\), \(A_4 := \{6\}\) of \(\Omega\). Then apparently
\[\begin{equation} \begin{array}{lllllll} \mathbb{P}\left(B \cap A_1\right) & = & \mathbb{P}\left(\{2,4,6\} \cap \{1,2\}\right) & = & \mathbb{P}\left(\{2\}\right) & = & \frac{1}{6} \\ \mathbb{P}\left(B \cap A_2\right) & = & \mathbb{P}\left(\{2,4,6\} \cap \{3\}\right) & = & \mathbb{P}\left(\emptyset\right) & = & 0 \\ \mathbb{P}\left(B \cap A_3\right) & = & \mathbb{P}\left(\{2,4,6\} \cap \{4,5\}\right) & = & \mathbb{P}\left(\{4\}\right) & = & \frac{1}{6} \\ \mathbb{P}\left(B \cap A_4\right) & = & \mathbb{P}\left(\{2,4,6\} \cap \{6\}\right) & = & \mathbb{P}\left(\{6\}\right) & = & \frac{1}{6} \end{array} \end{equation}\]
Thus we obtain \[\begin{equation} \begin{aligned} \begin{split} \sum_{i=1}^4 \mathbb{P}(B \cap A_i) & = \mathbb{P}(B \cap A_1) + \mathbb{P}(B \cap A_2) + \mathbb{P}(B \cap A_3) + \mathbb{P}(B \cap A_4) \\ & = \frac{1}{6} + 0 + \frac{1}{6} + \frac{1}{6} \\ & = \frac{1}{2} \end{split} \end{aligned} \end{equation}\] and hence \[\begin{equation} \mathbb{P}(B) = \sum_{i=1}^4 \mathbb{P}(B \cap A_i). \end{equation}\] Note that the partition property of \(A_1,A_2,A_3,A_4\) is essential for the validity of the theorem in this example. If, for example, \(A_4 = \{4,6\}\) and thus \(A_3\) and \(A_4\) were not disjoint, then the right-hand side of the law of total probability would yield a value greater than \(\mathbb{P}(B)\). If, by contrast, the union of \(A_1,A_2,A_3,A_4\) did not yield \(\Omega\), for example because \(A_4 = \emptyset\), then conversely a value smaller than \(\mathbb{P}(B)\) would result.
Finally, with Bayes’ theorem, we consider a formula for the alternative calculation of conditional probabilities.
Theorem 20.5 (Bayes’ theorem) Let \((\Omega,\mathcal{A},\mathbb{P})\) be a probability space and let \(A_1, ...,A_k\) be a partition of \(\Omega\) with \(\mathbb{P}(A_i) > 0\) for all \(i = 1,...,k\). If \(\mathbb{P}(B) > 0\), then for every \(i = 1,...,k\), \[\begin{equation} \mathbb{P}(A_i|B) = \frac{\mathbb{P}(B|A_i)\mathbb{P}(A_i)}{\sum_{j=1}^k \mathbb{P}(B|A_j)\mathbb{P}(A_j)}. \end{equation}\]
Proof. With the definition of conditional probability and the law of total probability, \[\begin{equation} \mathbb{P}(A_i|B) = \frac{\mathbb{P}(A_i \cap B)}{\mathbb{P}(B)} = \frac{\mathbb{P}(B|A_i)\mathbb{P}(A_i)}{\mathbb{P}(B)} = \frac{\mathbb{P}(B|A_i)\mathbb{P}(A_i)}{\sum_{j=1}^k \mathbb{P}(B|A_j)\mathbb{P}(A_j)}. \end{equation}\]
Note that Bayes’ theorem is independent of the frequentist or Bayesian interpretation of probability and merely makes a statement about calculating with conditional probabilities. In frequentist inference, however, Bayes’ theorem is used rather rarely. In Bayesian inference, by contrast, Bayes’ theorem is central. In this context, \(\mathbb{P}(A_i)\) is then often called the prior probability of event \(A_i\) and \(\mathbb{P}(A_i|B)\) the posterior probability of event \(A_i\). As explained above, \(\mathbb{P}(A_i|B)\) corresponds to the probability of \(A_i\) when one knows about the occurrence of \(B\).
In applications, the results formulated here as the law of total probability (Theorem 20.4), the theorem on joint and conditional probabilities (Theorem 20.3), and Bayes’ theorem (Theorem 20.5) are often used as the central calculation rules of probability theory. In practice, they are usually called the summation rule, multiplication rule, and Bayes rule (cf. Table 20.1).
| Calculation rule | Probability space form |
|---|---|
| Product rule | \(\mathbb{P}(A \cap B) = \mathbb{P}(B|A)\,\mathbb{P}(A)\), if \(\mathbb{P}(A)>0\) |
| Sum rule | \(\mathbb{P}(B) = \sum_{i=1}^k\mathbb{P}(B\cap A_i)\), if \(\Omega=\cup_{i=1}^k A_i\) and \(A_i\cap A_j=\emptyset\) for \(i\neq j\) |
| Bayes’ rule | \(\mathbb{P}(A|B)=\frac{\mathbb{P}(B|A)\mathbb{P}(A)}{\mathbb{P}(B)}\), if \(\mathbb{P}(A)>0\) and \(\mathbb{P}(B)>0\) |
20.3 Independent events
Intuitively, independence means that knowing one event has occurred does not change the probability of the other event. Its definition states that the joint probability of two events equals the product of their individual probabilities. This is also called factorization of the joint probability. The meaning of this definition becomes clear in the light of conditional probability in Theorem 20.6. We first consider the definition.
Definition 20.3 (Independent events) Two events \(A \in \mathcal{A}\) and \(B \in \mathcal{A}\) are called independent if \[\begin{equation} \mathbb{P}(A \cap B) = \mathbb{P}(A)\mathbb{P}(B). \end{equation}\] A set of events \(\{A_i|i \in I\}\subset \mathcal{A}\) with arbitrary index set \(I\) is called independent if for every finite subset \(J \subseteq I\), \[\begin{equation} \mathbb{P}\left(\cap_{j \in J} A_j \right) = \prod_{j \in J}\mathbb{P}(A_j). \end{equation}\]
Note that the independence of certain events can be assumed in the definition of a probabilistic model or can also follow from the definition of a probabilistic model. If two events are not independent, one also says that these events are dependent. The meaning of the product property under independence becomes clear, as we show below, in the context of conditional probabilities. First, we give an example that illustrates the relationship between independence and pairwise independence of events.
Example 20.3 (Pairwise and joint independence) Independent events are by definition always also pairwise independent. For example, for the set \(\{A_i| i \in \{1,2,3\}\}\) of events and the possible subsets \(J_1 := \{1,2\}\), \(J_2 := \{1,3\}\), \(J_3 := \{2,3\}\), \(J := \{1,2,3\}\) of \(I = \{1,2,3\}\), the defining condition of independence of \(A_1,A_2,A_3\) implies the statements \[\begin{equation} \begin{aligned} \begin{split} \mathbb{P}\left(A_1 \cap A_2\right) & = \mathbb{P}\left(A_1\right)\mathbb{P}\left(A_2\right) \\ \mathbb{P}\left(A_1 \cap A_3\right) & = \mathbb{P}\left(A_1\right)\mathbb{P}\left(A_3\right) \\ \mathbb{P}\left(A_2 \cap A_3\right) & = \mathbb{P}\left(A_2\right)\mathbb{P}\left(A_3\right) \\ \mathbb{P}\left(A_1 \cap A_2 \cap A_3\right) & = \mathbb{P}\left(A_1\right)\mathbb{P}\left(A_2\right)\mathbb{P}\left(A_3\right) \end{split} \end{aligned} \end{equation}\] Conversely, pairwise independence of events does not imply independence of events, as the following example shows (cf. DeGroot & Schervish (2012)). Consider the model of tossing a fair coin twice with outcome space \[\begin{equation} \Omega := \{HH, HT, TH, TT\} \end{equation}\] with elementary event probabilities \[\begin{equation} \mathbb{P}\{HH\} = \mathbb{P}\{HT\} = \mathbb{P}\{TH\} = \mathbb{P}\{TT\} = \frac{1}{4} \end{equation}\] Then the events listed in Table 20.2 are pairwise independent, but not independent.
| Verbal description | Set form |
|---|---|
| Heads appears on the first toss | \(A_1 := \{HH, HT\}\) |
| Heads appears on the second toss | \(A_2 := \{HH, TH\}\) |
| The same outcome appears on both tosses | \(A_3 := \{HH, TT\}\) |
Here, apparently, \[\begin{equation} \mathbb{P}\left(A_1 \right) = \mathbb{P}\left(A_2 \right)= \mathbb{P}\left(A_3 \right) = \frac{1}{2}, \end{equation}\] and \[\begin{equation} \begin{aligned} \begin{split} \mathbb{P}\left(A_1 \cap A_2 \right) & = \mathbb{P}\left(\{HH\}\right) = \frac{1}{4} = \frac{1}{2} \cdot \frac{1}{2} = \mathbb{P}\left(A_1 \right)\mathbb{P}\left(A_2 \right), \\ \mathbb{P}\left(A_1 \cap A_3 \right) & = \mathbb{P}\left(\{HH\}\right) = \frac{1}{4} = \frac{1}{2} \cdot \frac{1}{2} = \mathbb{P}\left(A_1 \right)\mathbb{P}\left(A_3 \right), \\ \mathbb{P}\left(A_2 \cap A_3 \right) & = \mathbb{P}\left(\{HH\}\right) = \frac{1}{4} = \frac{1}{2} \cdot \frac{1}{2} = \mathbb{P}\left(A_2 \right)\mathbb{P}\left(A_3 \right). \\ \end{split} \end{aligned} \end{equation}\] The events are therefore pairwise independent. However, it is also true that \[\begin{equation} \mathbb{P}\left(A_1 \cap A_2 \cap A_3 \right) = \mathbb{P}\left(\{HH\}\right) = \frac{1}{4} \neq \frac{1}{8} = \frac{1}{2} \cdot \frac{1}{2} \cdot \frac{1}{2} = \mathbb{P}\left(A_1\right)\mathbb{P}\left(A_2\right)\mathbb{P}\left(A_3\right) \end{equation}\] and the events \(A_1, A_2, A_3\) are therefore not independent.
Finally, with the relation between independence and conditional probability, we explain the meaning of the product property under independence.
Theorem 20.6 (Conditional probability under independence) Let \((\Omega,\mathcal{A}, \mathbb{P})\) be a probability space and let \(A,B\in \mathcal{A}\) be independent events with \(\mathbb{P}(B) > 0\). Then \[\begin{equation} \mathbb{P}(A|B) = \mathbb{P}(A). \end{equation}\]
Proof. Under the assumptions of the theorem, \[\begin{equation} \mathbb{P}(A|B) = \frac{\mathbb{P}(A \cap B)}{\mathbb{P}(B)} = \frac{\mathbb{P}(A)\mathbb{P}(B)}{\mathbb{P}(B)} = \mathbb{P}(A). \end{equation}\]
Given independence of two events \(A\) and \(B\), it is therefore irrelevant for the probability of event \(A\) whether \(B\) also occurs or not. The probability \(\mathbb{P}(A)\) remains the same. Independence of events is thus modeled precisely as factorization of the joint probability of \(A\) and \(B\) so that \(\mathbb{P}(A|B) = \mathbb{P}(A)\) follows. From the perspective of modeling subjective uncertainty through probabilities, the independence of two events means that knowing about the presence of one of the two events does not change the probability of the presence of the other event. Conversely, dependence of two events means that knowing about the presence of one of the two events changes the probability of the presence of the other event, that is, either increases or decreases it.
Example 20.4 (Conditional probability under independence for a fair die) We want to illustrate the relation between independence, conditional probability, and factorization of the joint probability using the model of the fair die. To this end, we consider the event \(A := \{2,4,6\}\) (“An even number appears”) and the event \(B := \{3,4,5,6\}\) (“A number greater than two appears”). Intuitively, the probability that an even number has appeared does not change if one knows that a number greater than two has appeared: if one has the information that a number greater than two has appeared, then three of six cases correspond to the event of an even number appearing. If one has the information, then two of four cases correspond to the event of an even number appearing. The relative proportion of cases corresponding to the event “An even number appears” therefore remains the same in view of the absence or presence of the information “A number greater than two appears”.
Formally, we first establish the factorization of the joint probability of the two events, that is, of the event “An even number appears and this number is greater than two”, into the product of the probabilities of the two events. Here, with the \(\sigma\)-additivity of \(\mathbb{P}\), \[\begin{equation} \begin{aligned} \begin{split} \mathbb{P}(A \cap B) & = \mathbb{P}\left(\{2,4,6\} \cap \{3,4,5,6\}\right)\\ & = \mathbb{P}\left(\{4,6\}\right)\\ & = \frac{2}{6}\\ & = \frac{1}{2}\cdot\frac{2}{3}\\ & = \mathbb{P}(\{2,4,6\})\cdot\mathbb{P}(\{3,4,5,6\})\\ & = \mathbb{P}(A)\mathbb{P}(B)\\ \end{split} \end{aligned} \end{equation}\] Thus the joint probability of \(A\) and \(B\) can be written as the product of the probabilities of \(A\) and \(B\). For the conditional probability of \(A\) given \(B\), we further have \[\begin{equation} \begin{aligned} \begin{split} \mathbb{P}(A|B) & = \frac{\mathbb{P}(A \cap B)}{\mathbb{P}(B)}\\ & = \frac{\mathbb{P}\left(\{4,6\}\right)}{\mathbb{P}(\{3,4,5,6\})}\\ & = \frac{1}{3} \cdot \frac{3}{2}\\ & = \frac{1}{2}\\ & = \mathbb{P}(\{2,4,6\})\\ & = \mathbb{P}(A) \end{split} \end{aligned} \end{equation}\] Thus in this example both Definition 20.3 and Theorem 20.6 are satisfied.
The property of events of being independent should not be confused with the possibility that events can be disjoint. In fact, the following theorem holds.
Theorem 20.7 (Disjointness and dependence of events) Let \(A \in \mathcal{A}\) and \(B \in \mathcal{A}\) be two disjoint events with \(\mathbb{P}(A)>0\) and \(\mathbb{P}(B)>0\). Then \(A\) and \(B\) are dependent events.
Proof. First, because of the disjointness of \(A\) and \(B\), \[\begin{equation} \mathbb{P}(A \cap B) = \mathbb{P}(\emptyset) = 0 \end{equation}\] Second, \[\begin{equation} \mathbb{P}(A)\mathbb{P}(B) > 0, \end{equation}\] because by assumption both \(\mathbb{P}(A)>0\) and \(\mathbb{P}(B)>0\) hold. Thus \[\begin{equation} \mathbb{P}(A \cap B) \neq \mathbb{P}(A)\mathbb{P}(B) \end{equation}\] and by Definition 20.3, \(A\) and \(B\) are therefore not independent, hence dependent.
Example 20.5 (Disjoint events with positive probabilities are dependent) We want to illustrate Theorem 20.7 using the example of the fair die. Let \(A = \{2,4,6\}\) be the event “An even number appears” and \(B = \{1,3,5\}\) the event “An odd number appears”. Intuitively, the events are dependent, because the information that an even number has appeared reduces the subjective probability that an odd number has appeared to zero. Formally, on the one hand, \[\begin{equation} \mathbb{P}(A)\mathbb{P}(B) = \frac{1}{2}\cdot \frac{1}{2} = \frac{1}{4}, \end{equation}\] but \[\begin{equation} \mathbb{P}(A \cap B) = \mathbb{P}\left(\{2,4,6\} \cap \{1,3,5\}\right) = \mathbb{P}\left(\emptyset\right) = 0. \end{equation}\] On the other hand, it is also true that \[\begin{equation} \mathbb{P}(A|B) = \frac{\mathbb{P}(A \cap B)}{\mathbb{P}(B)} = \frac{\mathbb{P}(\emptyset)}{\mathbb{P}(B)} = \frac{0}{\mathbb{P}(B)} = 0 \neq \frac{1}{2} = \mathbb{P}(\{2,4,6\}) = \mathbb{P}(A). \end{equation}\]
20.4 Bibliographic remarks
Many of the concepts introduced in this section are closely interwoven with the historical genesis of probability theory, so no individual references will be given. An introduction to the history of probability theory of the last two centuries is provided by Hald (1990), and an overview of more modern developments is given by Von Plato (1994). Bayes’ theorem is generally traced back to Bayes (1763), even though it is not the actual main topic of this work.
Study questions
- State the definition of the joint probability of two events.
- Explain the intuitive meaning of the joint probability of two events.
- State the theorem on further properties of probabilities.
- State the definition of the conditional probability of an event.
- State the theorem on conditional probability.
- State the theorem on joint and conditional probabilities.
- State the law of total probability.
- State Bayes’ theorem.
- Reproduce the proof of Bayes’ theorem.
- State the definition of the independence of two events.
- State the theorem on conditional probability under independence.
- Reproduce the proof of the theorem on conditional probability under independence.
- Explain the theorem on conditional probability under independence.
Study question answers
- See Definition 20.1.
- See the remarks on Definition 20.1.
- See Theorem 20.1.
- See Definition 20.2. The conditional probability \(\mathbb{P}(A|B)\) of an event \(A\) given an event \(B\) is a value in \([0,1]\).
- See Theorem 20.2. For a fixed event \(B\) with \(\mathbb{P}(B)>0\), the mapping \[\begin{equation} \mathbb{P}(\cdot|B)\colon\mathcal{A}\to[0,1], \qquad A\mapsto\mathbb{P}(A|B) \end{equation}\] is a probability measure.
- See Theorem 20.3.
- See Theorem 20.4.
- See Theorem 20.5.
- See proof of Theorem 20.5.
- See Definition 20.3.
- See Theorem 20.6.
- See proof of Theorem 20.6.
- See remarks on Theorem 20.6.