22 Random vectors
Random vectors are tuples of random variables. Since every random variable is subject to a probability distribution, a random vector is also subject to a probability distribution. Since a random vector consists of two or more random variables, the distribution of a random vector describes the joint distribution of two or more random variables. In this chapter, we first introduce the concept of a random vector and its associated probability distribution, which is often simply called a multivariate distribution. We then discuss how the concepts of PMFs, PDFs, and CDFs of random variables can be transferred to random vectors (Section 22.1). With marginal distributions and conditional distributions, we then introduce concepts that do not arise in the study of random variables. Finally, with the concept of independent random variables, we introduce the probabilistic standard model for univariate datasets. This is a special case of the joint distribution of the random variables of a random vector.
22.1 Definition and multivariate distributions
The construction and definition of a random vector is analogous to that of a random variable, with the difference that a random variable is a scalar-valued map, whereas a random vector is a vector-valued map on the outcome space of a probability space.
Definition 22.1 (Random vector) Let \((\Omega, \mathcal{A}, \mathbb{P})\) be a probability space and let \((\mathcal{X},\mathcal{S})\) be an \(n\)-dimensional measurable space. An \(n\)-dimensional random vector is defined as a map \[\begin{equation} \xi:\Omega \to \mathcal{X}, \omega \mapsto \xi(\omega) := \begin{pmatrix} \xi_1(\omega) \\ \vdots \\ \xi_n(\omega) \end{pmatrix} \end{equation}\] with the measurability property \[\begin{equation} \{\omega \in \Omega|\xi(\omega) \in S \} \in \mathcal{A} \mbox{ for all } S \in \mathcal{S}. \end{equation}\]
The standard example for the outcome space of a random vector is \(\mathbb{R}^n\), and the standard example for the \(\sigma\)-algebra defined on it is the \(n\)-dimensional Borel \(\sigma\)-algebra \(\mathcal{B}(\mathbb{R}^n)\). For an explicit and formal introduction to the \(n\)-dimensional Borel \(\sigma\)-algebra, we refer to the advanced literature (e.g. Schmidt (2009)). Here we again content ourselves with the still formally incorrect intuition of the \(n\)-dimensional Borel \(\sigma\)-algebra as the set of all subsets of \(\mathbb{R}^n\). The standard example of an \(n\)-dimensional measurable space is therefore \((\mathbb{R}^n, \mathcal{B}(\mathbb{R}^n))\).
As with all vector-valued functions, we call the functions \(\xi_i\) that constitute the random vector the component functions of \(\xi\). If we take the \(n\)-dimensional measurable space \((\mathbb{R}^n, \mathcal{B}(\mathbb{R}^n))\) as the basis of the random vector, then these have the form \[\begin{equation} \xi_i: \Omega \to \mathbb{R}, \omega \mapsto \xi_i(\omega). \end{equation}\] Without proof, we record that the random vector \(\xi\) is measurable if, for all \(i = 1,...,n\), the functions \(\xi_i\) are measurable, and conversely. Thus, the component functions of a random vector are ultimately, by definition, random variables. An \(n\)-dimensional random vector is therefore considered a concatenation of \(n\) random variables; for \(n := 1\), a random vector corresponds to a random variable.
Random vectors are sometimes also called “multivariate random variables”. In fact, when studying random vectors, probability-theoretic aspects are initially primary, rather than aspects of geometric vector space theory. The study of vector space structures is, however, quite common in the context of probabilistic standard models such as the general linear model, so here we prefer the term random vector (cf. Christensen (2011)). Nevertheless, as is common in many texts on probability theory, we will often also write random vectors in row form, for example as \(\xi:= (\xi_1,...,\xi_n)\).
The image measure defined by the construction of a random vector is called the multivariate distribution of the random vector, as set out in the following definition (cf. Figure 22.1).
Definition 22.2 (Multivariate distribution) Let \((\Omega, \mathcal{A}, \mathbb{P})\) be a probability space, let \((\mathcal{X},\mathcal{S})\) be an \(n\)-dimensional measurable space, and let \[\begin{equation} \xi : \Omega \to \mathcal{X}, \omega \mapsto \xi(\omega) \end{equation}\] be a random vector. Then the probability measure \(\mathbb{P}_\xi\), defined by \[\begin{equation} \mathbb{P}_\xi : \mathcal{S} \to [0,1], S \mapsto \mathbb{P}_\xi(S) := \mathbb{P}(\xi^{-1}(S)) = \mathbb{P}\left(\{\omega \in \Omega|\xi(\omega) \in S\}\right) \end{equation}\] is called the multivariate distribution of the random vector \(\xi\).
For simplicity, one often speaks only of the distribution of the random vector \(\xi\) or of a multivariate distribution. The notational conventions for random variables in Definition 21.3 apply analogously to random vectors. For example, we have \[\begin{align} \begin{split} \mathbb{P}_\xi(\xi \in S) & := \mathbb{P}\left(\{\xi \in S\} \right) = \mathbb{P}\left(\{\omega \in \Omega|\xi(\omega) \in S\} \right) \\ \mathbb{P}_\xi(\xi = x) & := \mathbb{P}\left(\{\xi = x\} \right) = \mathbb{P}\left(\{\omega \in \Omega|\xi(\omega) = x\} \right) \\ \mathbb{P}_\xi(\xi \le x) & := \mathbb{P}\left(\{\xi \le x\} \right) = \mathbb{P}\left(\{\omega \in \Omega|\xi(\omega) \le x\} \right) \\ \mathbb{P}_\xi(x_1 \le \xi \le x_2) & := \mathbb{P}\left(\{x_1 \le \xi \le x_2\} \right) = \mathbb{P}\left(\{\omega \in \Omega|x_1 \le \xi(\omega) \le x_2\} \right), \end{split} \end{align}\] where the relational operators \(<, \le, >, \ge\) are understood here componentwise. For example, \(x \le y\) for \(x,y \in \mathbb{R}^n\) means that \(x_i \le y_i\) holds for all components \(x_i,y_i, i = 1,...,n\). This convention is also followed by the definition of the multivariate cumulative distribution function in generalization of Definition 21.13. As with random variables, the qualifying subscripts with respect to a specific random vector are usually omitted in the sense of multiple dispatch. We use this convention already in the following definition of the multivariate cumulative distribution function of a random vector \(\xi\).
Definition 22.3 (Multivariate cumulative distribution functions) Let \(\xi\) be a random vector with outcome space \(\mathcal{X}\). Then a function of the form \[\begin{equation} P: \mathcal{X} \to [0,1],\, x \mapsto P(x) := \mathbb{P}(\xi \le x) \end{equation}\] is called a multivariate cumulative distribution function of \(\xi\).
Like cumulative distribution functions, multivariate cumulative distribution functions can also be used to define multivariate distributions. More commonly, however, as in the univariate case, multivariate distributions are defined by multivariate probability mass functions or probability density functions. We generalize the definitions of discrete and continuous random variables and their associated probability mass and probability density functions as follows (cf. Definition 21.4 and Definition 21.8).
Definition 22.4 (Discrete random vector and multivariate probability mass function) A random vector \(\xi\) is called discrete if its outcome space \(\mathcal{X}\) is finite or countable and if a function \[\begin{equation} p : \mathcal{X} \to [0,1], x \mapsto p(x) \end{equation}\] exists such that
- \(\sum_{x \in \mathcal{X}}p(x) = 1\)
- \(\mathbb{P}(\xi = x) = p(x)\) for all \(x \in \mathcal{X}\).
Such a function \(p\) is called the multivariate probability mass function (PMF) of \(\xi\).
The concept of a multivariate PMF is evidently directly analogous to the concept of a PMF. Like univariate PMFs, multivariate PMFs are non-negative and normalized. For simplicity, one often speaks simply of the PMF of a random vector and, when the relevant random vector is clear from the context, omits the \(\xi\) subscript in its notation, thus often writing simply \(p\) instead of \(p_\xi\).
Example 22.1 (Discrete random vector) To illustrate the concept of the discrete random vector and its PMF, we consider an example. Let \(\xi:= (\xi_1,\xi_2)\) be a random vector that takes values in \(\mathcal{X} := \mathcal{X}_1 \times \mathcal{X}_2\), where \(\mathcal{X}_1 := \{1,2,3\}\) and \(\mathcal{X}_2 = \{1,2,3,4\}\). Then the outcome space of \(\xi\) corresponds to the set of tuples \((x_1,x_2)\) specified in Table 22.1.
| \((x_1,x_2)\) | \(x_2 = 1\) | \(x_2 = 2\) | \(x_2 = 3\) | \(x_2 = 4\) |
|---|---|---|---|---|
| \(x_1 = 1\) | \((1,1)\) | \((1,2)\) | \((1,3)\) | \((1,4)\) |
| \(x_1 = 2\) | \((2,1)\) | \((2,2)\) | \((2,3)\) | \((2,4)\) |
| \(x_1 = 3\) | \((3,1)\) | \((3,2)\) | \((3,3)\) | \((3,4)\) |
An exemplary bivariate PMF of the form \[\begin{equation} p: \{1,2,3\} \times \{1,2,3,4\} \to [0,1], (x_1,x_2) \mapsto p(x_1,x_2) \end{equation}\] is then defined by Table 22.2.
| \(p(x_1,x_2)\) | \(x_2 = 1\) | \(x_2 = 2\) | \(x_2 = 3\) | \(x_2 = 4\) |
|---|---|---|---|---|
| \(x_1 = 1\) | \(0.1\) | \(0.0\) | \(0.2\) | \(0.1\) |
| \(x_1 = 2\) | \(0.1\) | \(0.2\) | \(0.0\) | \(0.0\) |
| \(x_1 = 3\) | \(0.0\) | \(0.1\) | \(0.1\) | \(0.1\) |
Note that the function \(p\) specified in this way satisfies the normalization and non-negativity requirements of a PMF. In particular, here \[\begin{equation} \sum_{x \in \mathcal{X}} p(x) = \sum_{x_1 = 1}^3 \sum_{x_2 = 1}^4 p(x_1,x_2) = 1. \end{equation}\]
We define the concept of the continuous random vector and the multivariate probability density function as follows.
Definition 22.5 (Continuous random vector and multivariate probability density function) A random vector \(\xi\) is called continuous if its outcome space is given by \(\mathbb{R}^n\) and if a function \[\begin{equation} p: \mathbb{R}^n \to \mathbb{R}_{\ge 0}, x \mapsto p(x), \end{equation}\] exists such that
- \(\int_{\mathbb{R}^n} p(x)\,dx = 1\) and
- \(\mathbb{P}(x_1 \le \xi \le x_2) = \int_{x_{1_1}}^{x_{2_1}} \cdots \int_{x_{1_n}}^{x_{2_n}} p(s_1,...,s_n)\,ds_1 \cdots ds_n\).
Such a function \(p\) is called the multivariate probability density function (PDF) of \(\xi\).
Evidently, the concept of the multivariate PDF of a continuous random vector is analogous to the concept of the PDF of a continuous random variable, and like univariate PDFs, multivariate PDFs are non-negative and normalized. For simplicity, one often also speaks simply of multivariate PDFs and omits the subscripts identifying the random vector. As for continuous random variables, for continuous random vectors we have \[\begin{equation} \mathbb{P}(\xi = x) = \mathbb{P}(x \le \xi \le x) = \int_{x_1}^{x_1} \cdots \int_{x_n}^{x_n} p(s_1,...,s_n)\,ds_1 \cdots ds_n = 0. \end{equation}\]
The standard example of a multivariate PDF is the multivariate normal distribution, which is discussed in detail in the chapter on normal distributions.
22.2 Marginal distributions
Once the distribution of a random vector has been specified, one may ask which distributions follow from it for the individual components of the random vector, that is, for the random variables that together form the random vector. In the context of a random vector, these are called the univariate marginal distributions of the random vector. The following definition is fundamental.
Definition 22.6 (Univariate marginal distribution) Let \((\Omega, \mathcal{A}, \mathbb{P})\) be a probability space, let \((\mathcal{X}, \mathcal{S})\) be an \(n\)-dimensional measurable space, let \(\xi:\Omega \to \mathcal{X}\) be a random vector, let \(\mathbb{P}\) be the distribution of \(\xi\), let \(\mathcal{X}_i \subset \mathcal{X}\) be the outcome space of the \(i\)th component \(\xi_i\) of \(\xi\), and let \(\mathcal{S}_i\) be a \(\sigma\)-algebra on \(\xi_i\). Then the distribution defined by \[\begin{equation} \mathbb{P}_{\xi_i} : \mathcal{S}_i \to [0,1], S \mapsto \mathbb{P}\left(\mathcal{X}_1 \times \cdots \times \mathcal{X}_{i-1} \times S \times \mathcal{X}_{i+1} \times \cdots \times \mathcal{X}_n\right) \mbox{ for } S \in \mathcal{S}_i \end{equation}\] is called the \(i\)th univariate marginal distribution of \(\xi\).
Concretely, for both discrete and continuous random vectors, the PMFs or PDFs of their components can be determined directly from the corresponding multivariate PMF or PDF. This is the statement of the following theorem.
Theorem 22.1 (Marginal probability mass and probability density functions) (1) Let \(\xi= (\xi_1,...,\xi_n)\) be an \(n\)-dimensional discrete random vector with PMF \(p\) and component outcome spaces \(\mathcal{X}_1, ..., \mathcal{X}_n\). Then the PMF of the \(i\)th component \(\xi_i\) of \(\xi\) is given by
\[\begin{equation} p : \mathcal{X}_i \to [0,1], x_i \mapsto p(x_i) := \sum_{x_1} \cdots \sum_{x_{i-1}} \sum_{x_{i+1}} \cdots \sum_{x_n} p(x_1,...,x_{i-1},x_i,x_{i+1}, ...,x_n). \end{equation}\]
(2) Let \(\xi= (\xi_1,...,\xi_n)\) be an \(n\)-dimensional continuous random vector with PDF \(p\) and component outcome space \(\mathbb{R}\). Then the PDF of the \(i\)th component \(\xi_i\) of \(\xi\) is given by
\[\begin{equation} p : \mathbb{R} \to \mathbb{R}_{\ge 0}, x_i \mapsto p(x_i) := \int_{x_1} \cdots \int_{x_{i-1}} \int_{x_{i+1}} \cdots \int_{x_n} p(x_1,..., x_{i-1},x_i,x_{i+1}, ...,x_n) \,dx_1...\,dx_{i-1}\,dx_{i+1}...\,dx_n. \end{equation}\]
The PMFs of the univariate marginal distributions of discrete random vectors are thus obtained by summing over all values of the random variables complementary to the random variable currently under consideration, and the PDFs of the univariate marginal distributions of continuous random vectors are obtained analogously by integrating over all values of the random variables complementary to the random variable currently under consideration. For a proof of Theorem 22.1, we refer to the advanced literature.
Example 22.2 (Marginal probability mass functions) Continuing Example 22.1, for the PMF defined there for the random vector \(\xi:= (\xi_1,\xi_2)\), the marginal PMFs are obtained as the PMFs listed on the margins of Table 22.3 using \[\begin{equation} p(x_1) = \sum_{x_2 = 1}^{4} p(x_1,x_2) \mbox{ and } p(x_2) = \sum_{x_1 = 1}^{3} p(x_1,x_2). \end{equation}\]
| \(p(x_1,x_2)\) | \(x_2 = 1\) | \(x_2 = 2\) | \(x_2 = 3\) | \(x_2 = 4\) | \(p(x_1)\) |
|---|---|---|---|---|---|
| \(x_1 = 1\) | \(0.1\) | \(0.0\) | \(0.2\) | \(0.1\) | \(0.4\) |
| \(x_1 = 2\) | \(0.1\) | \(0.2\) | \(0.0\) | \(0.0\) | \(0.3\) |
| \(x_1 = 3\) | \(0.0\) | \(0.1\) | \(0.1\) | \(0.1\) | \(0.3\) |
| \(p(x_2)\) | \(0.2\) | \(0.3\) | \(0.3\) | \(0.2\) |
For the values \(p(x_1)\), the corresponding values of \(p(x_1,x_2)\) are therefore added row by row, and for the values \(p(x_2)\), column by column. Note that the normalization of \(p(x_1)\) and \(p(x_2)\) follows directly from the normalization of \(p(x_1,x_2)\), because the total amount of probability mass does not change: \[\begin{equation} 1 = \sum_{x_1=1}^{3}\sum_{x_2 = 1}^{4} p(x_1,x_2) = \sum_{x_1=1}^{3} p(x_1) = \sum_{x_2=1}^{4} p(x_2). \end{equation}\]
An example of realizations using relative frequencies may make the concept of the marginal PMF intuitively clear. Suppose we had \(n = 100\) independent realizations of \(\xi\). To estimate the probabilities \(p(x_1,x_2)\), we would count the number of realizations of \((x_1,x_2)\) and divide by \(n\). If, for example, we had 12 realizations of \((3,2)\), we would estimate \(p(3,2) \approx 12/100 = 0.12\). The question of the marginal probability of \(x_2 = 2\) would then correspond to the question of how often, among the realizations, there are those for which \(x_2 = 2\), irrespective of the value of \(x_1\). This would be precisely the number of realizations of the form \((1,2), (2,2)\), and \((3,2)\). If there were, for example, \(0\), \(22\), and \(12\) of these, respectively, then one would naturally estimate the probability \(p(2)\) by \[\begin{equation} \frac{0 + 22 + 12}{100} = \frac{0}{100} + \frac{22}{100} + \frac{12}{100} = 0.00 + 0.22 + 0.12 = 0.34. \end{equation}\] Thus, instead of the probabilities \(p(1,2)\), \(p(2,2)\), and \(p(3,2)\), one adds the corresponding relative frequencies here.
Marginal distributions in the case of continuous random vectors are treated using the standard example of the multivariate normal distribution in the chapter on normal distributions.
22.3 Conditional distributions
Once the distribution of a random vector has been specified, one may ask which distribution follows from it for a single component of the random vector if the value of another component is assumed to be known. This leads to the concept of the conditional distribution, which arises naturally from the concept of conditional probability (cf. Section 20.2). We first recall that, for a probability space \((\Omega, \mathcal{A}, \mathbb{P})\) and two events \(A,B \in \mathcal{A}\) with \(\mathbb{P}(B) > 0\), the conditional probability of \(A\) given \(B\) is defined as \[\begin{equation} \mathbb{P}(A|B) = \frac{\mathbb{P}(A \cap B)}{\mathbb{P}(B)}. \end{equation}\] Analogously, for two random variables \(\xi_1,\xi_2\) with outcome spaces \(\mathcal{X}_1,\mathcal{X}_2\) and measurable sets \(S_1 \in \mathcal{X}_1, S_2 \in \mathcal{X}_2\), the conditional distribution of \(\xi_1\) given \(\xi_2\) is defined with the help of the events \[\begin{equation} A := \{\xi_1 \in S_1\} \mbox{ and } B := \{\xi_2 \in S_2\}. \end{equation}\] For example, the conditional probability that \(\xi_1 \in S_1\) given that \(\xi_2 \in S_2\), under the assumption that \(\mathbb{P}(\{\xi_2 \in S_2\}) > 0\), is \[\begin{equation} \mathbb{P}( \{\xi_1 \in S_1\}|\{\xi_2 \in S_2\}) = \frac{\mathbb{P}(\{\xi_1 \in S_1\} \cap \{\xi_2 \in S_2\})}{\mathbb{P}(\{\xi_2 \in S_2\})}. \end{equation}\]
We first consider the definition of conditional distributions of discrete random vectors that consist of only two random variables.
Definition 22.7 (Conditional probability mass function and discrete conditional distribution) Let \(\xi:= (\xi_1,\xi_2)\) be a discrete random vector with outcome space \(\mathcal{X} := \mathcal{X}_1 \times \mathcal{X}_2\), PMF \(p(x_1,x_2)\), and marginal PMFs \(p(x_1)\) and \(p(x_2)\). The conditional PMF of \(\xi_1\) given \(\xi_2 = x_2\) is then defined for \(p(x_2) > 0\) as \[\begin{equation} p: \mathcal{X}_1 \to [0,1], x_1 \mapsto p(x_1|x_2) := \frac{p(x_1,x_2)}{p(x_2)}. \end{equation}\] Analogously, for \(p(x_1) > 0\), the conditional PMF of \(\xi_2\) given \(\xi_1 = x_1\) is defined as \[\begin{equation} p: \mathcal{X}_2 \to [0,1], x_2 \mapsto p(x_2|x_1) := \frac{p(x_1,x_2)}{p(x_1)}. \end{equation}\] The conditional distributions with PMFs \(p(x_1|x_2)\) and \(p(x_2|x_1)\) are then called the discrete conditional distributions of \(\xi_1\) given \(\xi_2 = x_2\) and of \(\xi_2\) given \(\xi_1 = x_1\), respectively.
In analogy to the definition of the conditional probability of events, we thus have \[\begin{equation} p(x_1|x_2) = \frac{p(x_1,x_2)}{p(x_2)} = \frac{\mathbb{P}(\{\xi_1 = x_1\} \cap \{\xi_2 = x_2\})}{\mathbb{P}(\{\xi_2 = x_2\})}. \end{equation}\] It is crucial to recognize here that conditional distributions are merely normalized joint distributions.
Example 22.3 (Conditional probability mass functions) Continuing the example of a two-dimensional random vector \(\xi:= (\xi_1,\xi_2)\) considered in Example 22.1 and its marginal distributions determined in Example 22.2, the conditional PMFs for \(p(x_1|x_2)\) shown in Table 22.4 and the conditional PMFs for \(p(x_2|x_1)\) shown in Table 22.5 are obtained.
| \(p(x_1|x_2)\) | \(x_2 = 1\) | \(x_2 = 2\) | \(x_2 = 3\) | \(x_2 = 4\) |
|---|---|---|---|---|
| \(p(1|x_2)\) | \(\frac{0.1}{0.2} = 0.5\) | \(\frac{0.0}{0.3} = 0.0\) | \(\frac{0.2}{0.3} = 0.6\bar{6}\) | \(\frac{0.1}{0.2} = 0.5\) |
| \(p(2|x_2)\) | \(\frac{0.1}{0.2} = 0.5\) | \(\frac{0.2}{0.3} = 0.6\bar{6}\) | \(\frac{0.0}{0.3} = 0.0\) | \(\frac{0.0}{0.2} = 0.0\) |
| \(p(3|x_2)\) | \(\frac{0.0}{0.2} = 0.0\) | \(\frac{0.1}{0.3} = 0.3\bar{3}\) | \(\frac{0.1}{0.3} = 0.3\bar{3}\) | \(\frac{0.1}{0.2} = 0.5\) |
| \(p(x_2|x_1)\) | \(p(1|x_1)\) | \(p(2|x_1)\) | \(p(3|x_1)\) | \(p(4|x_1)\) |
|---|---|---|---|---|
| \(x_1 = 1\) | \(\frac{0.1}{0.4} = 0.25\) | \(\frac{0.0}{0.4} = 0.00\) | \(\frac{0.2}{0.4} = 0.50\) | \(\frac{0.1}{0.4} = 0.25\) |
| \(x_1 = 2\) | \(\frac{0.1}{0.3} = 0.3\bar{3}\) | \(\frac{0.2}{0.3} = 0.6\bar{6}\) | \(\frac{0.0}{0.3} = 0.00\) | \(\frac{0.0}{0.3} = 0.00\) |
| \(x_1 = 3\) | \(\frac{0.0}{0.3} = 0.00\) | \(\frac{0.1}{0.3} = 0.3\bar{3}\) | \(\frac{0.1}{0.3} = 0.3\bar{3}\) | \(\frac{0.1}{0.3} = 0.3\bar{3}\) |
Note, first, that \[\begin{equation} \sum_{x_1 = 1}^3 p(x_1|x_2) = 1 \mbox{ for all } x_2 \in \mathcal{X}_2 \mbox{ and } \sum_{x_2 = 1}^4 p(x_2|x_1) = 1 \mbox{ for all } x_1 \in \mathcal{X}_1, \end{equation}\] so the conditional PMFs are normalized. Second, note the qualitative similarity of the PMFs \(p(x_1,x_2)\) and \(p(x_1|x_2)\) or \(p(x_2|x_1)\), respectively, which arises simply from the fact that, on the one hand, \(p(x_1,x_2)\) and \(p(x_1|x_2)\) are identical for all \(x_2 \in \mathcal{X}_2\) up to the common scaling factor \(1/p(x_2)\) and, on the other hand, \(p(x_1,x_2)\) and \(p(x_2|x_1)\) are identical for all \(x_1 \in \mathcal{X}_1\) up to the common scaling factor \(1/p(x_1)\).
In the case of a continuous random vector, the analogous conditional PDFs are defined as follows.
Definition 22.8 (Conditional probability density function and continuous conditional distribution) Let \(\xi:= (\xi_1,\xi_2)\) be a continuous random vector with outcome space \(\mathbb{R}^2\), PDF \(p(x_1,x_2)\), and marginal PDFs \(p(x_1)\) and \(p(x_2)\). The conditional PDF of \(\xi_1\) given \(\xi_2 = x_2\) is then defined for \(p(x_2) > 0\) as \[\begin{equation} p: \mathbb{R} \to \mathbb{R}_{\ge 0}, x_1 \mapsto p(x_1|x_2) := \frac{p(x_1,x_2)}{p(x_2)}. \end{equation}\] Analogously, for \(p(x_1) > 0\), the conditional PDF of \(\xi_2\) given \(\xi_1 = x_1\) is defined as \[\begin{equation} p: \mathbb{R} \to \mathbb{R}_{\ge 0}, x_2 \mapsto p(x_2|x_1) := \frac{p(x_1,x_2)}{p(x_1)}. \end{equation}\]
The distributions with PDFs \(p(x_1|x_2)\) and \(p(x_2|x_1)\) are then called the continuous conditional distributions of \(\xi_1\) given \(\xi_2 = x_2\) and \(\xi_2\) given \(\xi_1 = x_1\), respectively.
Note that in the continuous case, although \(\mathbb{P}(\xi = x) = 0\), it does not necessarily follow that \(p(x) = 0\). Conditional distributions of multivariate normal distributions are discussed in a separate chapter.
22.4 Independent random variables
Similar to the conditional probabilities of events, the concept of independent events can also be transferred to random vectors. We first define the concept of independent random variables.
Definition 22.9 (Independent random variables) Let \((\Omega, \mathcal{A}, \mathbb{P})\) be a probability space and let \(\xi: = (\xi_1,\xi_2)\) be a two-dimensional random vector. The random variables \(\xi_1,\xi_2\) with outcome spaces \(\mathcal{X}_1, \mathcal{X}_2\) are called independent if, for all \(S_1 \subseteq \mathcal{X}_1\) and \(S_2 \subseteq \mathcal{X}_2\), \[\begin{equation} \mathbb{P}(\xi_1 \in S_1, \xi_2 \in S_2) = \mathbb{P}(\xi_1 \in S_1)\mathbb{P}(\xi_2 \in S_2). \end{equation}\]
Definition 22.9 states that the events \(\{\xi_1 \in S_1\}\) and \(\{\xi_2 \in S_2\}\) are independent. Thus, it also holds that \[\begin{equation} \mathbb{P}(\{\xi_1 \in S_1\}|\{\xi_2 \in S_2\}) = \mathbb{P}(\{\xi_1 \in S_1\}) \end{equation}\] and knowledge of the occurrence of the event \(\{\xi_2 \in S_2\}\) does not change the probability of the event \(\{\xi_1 \in S_1\}\). The factorization principle for modeling probabilistic independence transfers to PMFs and PDFs of random vectors. This is the statement of the following theorem.
Theorem 22.2 (Independence and factorization)
- Let \(\xi:= (\xi_1,\xi_2)\) be a discrete random vector with outcome space \(\mathcal{X}_1 \times \mathcal{X}_2\), PMF \(p(x_1,x_2)\), and marginal PMFs \(p(x_1), p(x_2)\). Then \[\begin{multline} \xi_1 \mbox{ and } \xi_2 \mbox{ are independent random variables} \Leftrightarrow \\ p(x_1,x_2) = p(x_1)p(x_2) \mbox{ for all } (x_1,x_2) \in \mathcal{X}_1 \times \mathcal{X}_2. \end{multline}\]
- Let \(\xi:= (\xi_1,\xi_2)\) be a continuous random vector with outcome space \(\mathbb{R}^2\), PDF \(p\), and marginal PDFs \(p, p\). Then \[\begin{multline} \xi_1 \mbox{ and } \xi_2 \mbox{ are independent random variables } \Leftrightarrow \\ p(x_1,x_2) = p(x_1)p(x_2) \mbox{ for all } (x_1,x_2) \in \mathbb{R}^2. \end{multline}\]
In general, the independence of two random variables is thus equivalent to the factorization of their joint PMF or PDF. For a proof of Theorem 22.2, we refer to the advanced literature. Theorem 22.2 is fundamental for wide areas of probabilistic modeling.
Example 22.4 (Independent probability mass functions) We again consider the two-dimensional random vector \(\xi:= (\xi_1, \xi_2)\) from Example 22.1, whose joint and marginal PMFs have the forms specified in Table 22.3. We first ask whether \(\xi_1\) and \(\xi_2\) are independent. This is not the case, since here \[\begin{equation} p(1,1) = 0.10 \neq 0.08 = 0.40\cdot 0.20 = p(1)p(1). \end{equation}\] If, based on the marginal distributions of \(\xi\), we want to generate a joint distribution in which \(\xi_1\) and \(\xi_2\) are independent, then each entry of the joint distribution \(p(\xi_1,\xi_2)\) must be obtained from the corresponding product of the marginal probabilities. The joint distribution of \(\xi_1\) and \(\xi_2\) under this assumption of independence of \(\xi_1\) and \(\xi_2\), with the same marginal distributions as in Table 22.3, is given by the joint PMF in Table 22.6.
| \(p(x_1,x_2)\) | \(x_2 = 1\) | \(x_2 = 2\) | \(x_2 = 3\) | \(x_2 = 4\) | \(p(x_1)\) |
|---|---|---|---|---|---|
| \(x_1 = 1\) | \(0.08\) | \(0.12\) | \(0.12\) | \(0.08\) | \(0.40\) |
| \(x_1 = 2\) | \(0.06\) | \(0.09\) | \(0.09\) | \(0.06\) | \(0.30\) |
| \(x_1 = 3\) | \(0.06\) | \(0.09\) | \(0.09\) | \(0.06\) | \(0.30\) |
| \(p(x_2)\) | \(0.20\) | \(0.30\) | \(0.30\) | \(0.20\) |
Furthermore, in the case of independence of \(\xi_1\) and \(\xi_2\), the conditional PMFs \(p(x_2|x_1)\), for example, are obtained as shown in Table 22.7. In the case of independence of \(\xi_1\) and \(\xi_2\), the distribution of \(\xi_2\) given, or with knowledge of, the value of \(\xi_1\) therefore does not change and corresponds in each case to the marginal distribution of \(\xi_2\). This of course corresponds to the intuition of independence of events in the context of elementary probabilities.
| \(p(x_2|x_1)\) | \(p(1|x_1)\) | \(p(2|x_1)\) | \(p(3|x_1)\) | \(p(4|x_1)\) |
|---|---|---|---|---|
| \(x_1 = 1\) | \(\frac{0.08}{0.40} = 0.2\) | \(\frac{0.12}{0.40} = 0.3\) | \(\frac{0.12}{0.40} = 0.3\) | \(\frac{0.08}{0.40} = 0.2\) |
| \(x_1 = 2\) | \(\frac{0.06}{0.30} = 0.2\) | \(\frac{0.09}{0.30} = 0.3\) | \(\frac{0.09}{0.30} = 0.3\) | \(\frac{0.06}{0.30} = 0.2\) |
| \(x_1 = 3\) | \(\frac{0.06}{0.30} = 0.2\) | \(\frac{0.09}{0.30} = 0.3\) | \(\frac{0.09}{0.30} = 0.3\) | \(\frac{0.06}{0.30} = 0.2\) |
Finally, we define the concept of independent random variables for more than two random variables.
Definition 22.10 (\(n\) independent random variables) Let \(\xi:= (\xi_1,...,\xi_n)\) be an \(n\)-dimensional random vector with outcome space \(\mathcal{X} = \times_{i=1}^n \mathcal{X}_i\). The \(n\) random variables \(\xi_1,...,\xi_n\) are called independent if, for all \(S_i \subseteq \mathcal{X}_i, i = 1,...,n\), \[\begin{equation} \mathbb{P}(\xi_1 \in S_1, ...,\xi_n \in S_n) = \prod_{i=1}^n \mathbb{P}(\xi_i \in S_i). \end{equation}\] If the random vector has an \(n\)-dimensional PMF or PDF \(p\) with marginal PMFs or PDFs \(p_{\xi_i}, i = 1,...,n\), then the independence of \(\xi_1,...,\xi_n\) is equivalent to the factorization of the joint PMF or PDF, that is, to \[\begin{equation} p(\xi_1,...,\xi_n) = \prod_{i=1}^n p(x_i). \end{equation}\]
Thus, Definition 22.10 is a direct generalization of the two-dimensional case. If \(n\) random variables are not only independent but also all have the same distribution, they are called independent and identically distributed (i.i.d.).
Definition 22.11 (Independent and identically distributed random variables) \(n\) random variables \(\xi_1,...,\xi_n\) are called independent and identically distributed (i.i.d.) if
- \(\xi_1,...,\xi_n\) are independent random variables, and
- the marginal distributions of the \(\xi_i\) agree, that is, \[\begin{equation} \mathbb{P}(\xi_i) = \mathbb{P}(\xi_j) \mbox{ for all } 1 \le i,j \le n. \end{equation}\]
If the random variables \(\xi_1,...,\xi_n\) are independent and identically distributed and the \(i\)th marginal distribution is \(\mathbb{P}\), one also writes \[\begin{equation} \xi_1,...,\xi_n \sim \mathbb{P}. \end{equation}\]
In short, one says that \(\xi_1,...,\xi_n\) are i.i.d. I.i.d. random variables play an important role in many places in probabilistic modeling. As we will see later, additive error terms in probabilistic models are usually modeled by i.i.d. random variables. Finally, we record that \(n\) i.i.d. normally distributed random variables are written as \[\begin{equation} \xi_1,...,\xi_n \sim N(\mu,\sigma^2). \end{equation}\] In the chapter on normal distributions, we show what the joint distribution of \(n\) i.i.d. normally distributed random variables looks like in terms of a random vector.
Study questions
- State the definition of the concept of a random vector.
- State the definition of the concept of the multivariate distribution of a random vector.
- State the definition of the concept of a multivariate PMF.
- State the definition of the concept of a multivariate PDF.
- State the definition of the concept of the univariate marginal distribution of a random vector.
- How does one compute the PMF of the \(i\)th component of a discrete random vector?
- How does one compute the PDF of the \(i\)th component of a continuous random vector?
- State the definition of the concept of independence of two random variables.
- How can one recognize from the joint PMF or PDF of a two-dimensional random vector \(\xi := (\xi_1,\xi_2)\) whether the components of the random vector are independent or not?
- State the definition of the concept of independence of \(n\) random variables.
- State the definition of the concept of independent and identically distributed random variables.
Study question answers
- See Definition 22.1.
- See Definition 22.2.
- See Definition 22.4.
- See Definition 22.5.
- See Definition 22.6.
- See Theorem 22.1. One obtains the PMF of the \(i\)th component of a discrete random vector by summing the values of the PMF of the random vector over all components of the random vector except the \(i\)th component.
- See Theorem 22.1. One obtains the PDF of the \(i\)th component of a continuous random vector by integrating the values of the PDF of the random vector over all components of the random vector except the \(i\)th component.
- See Definition 22.10.
- One checks whether all values of the joint PMF or PDF \(p(x_1,x_2)\) of the random vector can be obtained by multiplying the corresponding marginal PMF or PDF values \(p(x_1)\) and \(p(x_2)\). If this is the case for all \((x_1,x_2) \in \mathcal{X}_1 \times \mathcal{X}_2\), then \(\xi_1\) and \(\xi_2\) are independent; if this is not the case for all \((x_1,x_2) \in \mathcal{X}_1 \times \mathcal{X}_2\), then \(\xi_1\) and \(\xi_2\) are not independent.
- See Definition 22.9.
- See Definition 22.11.