23 Expectations
In this chapter, we introduce the concept of the expectation of a random variable. The expectation serves as a single scalar characteristic that summarizes a probability distribution and is often used as a central characteristic quantity of random variables. In this sense, the expectation serves as a measure of the “average realization” of a random variable and forms the basis for further concepts such as the variance of a random variable and the covariance of two random variables in what follows. We supplement the concept of expectation with its analogues for random vectors and conditional distributions, as well as with its descriptive-statistical equivalent, the so-called sample mean.
23.1 Definitions
Definition 23.1 (Expectation of a random variable) Let \((\Omega, \mathcal{A},\mathbb{P})\) be a probability space and let \(\xi\) be a random variable. Then the expectation of \(\xi\) is defined as
- \(\mathbb{E}(\xi) := \sum_{x \in \mathcal{X}} x\,p(x)\) if \(\xi : \Omega \to \mathcal{X}\) is discrete with PMF \(p\).
- \(\mathbb{E}(\xi) := \int_{-\infty}^\infty x \,p(x)\,dx\) if \(\xi : \Omega \to \mathbb{R}\) is continuous with PDF \(p\).
The expectation is thus a scalar summary of the distribution of a random variable. A definition of the expectation that does not require distinguishing between continuous and discrete random variables is possible, but, with the introduction of the Lebesgue integral, requires some technical effort. We refer to the advanced literature for this (cf. Schmidt (2009), Meintrup & Schäffler (2005)). It is important to recognize that an expectation is determined solely by specifying the outcome space and the distribution of a random variable by means of a PMF or PDF. Nevertheless, the expectation of a random variable corresponds to the value of the random variable expected on average in the long run. This fact is formalized under the concept of the law of large numbers. We first consider three examples for Definition 23.1.
Example 23.1 (Expectation of a discrete random variable) Let \(\xi\) be a random variable with outcome space \(\mathcal{X} := \{-1,0,1\}\) and PMF \[\begin{equation} p(-1) := \frac{1}{4}, \quad p(0) := \frac{1}{2}, \quad p(1) := \frac{1}{4}. \end{equation}\] Then \[\begin{equation} \mathbb{E}(\xi) = 0. \end{equation}\]
Proof. By Definition 23.1, we have \[\begin{align} \begin{split} \mathbb{E}(\xi) & = \sum_{x \in \mathcal{X}} x p(x) \\ & = -1 \cdot p(-1) + 0 \cdot p(0) + 1 \cdot p(1) \\ & = -1 \cdot \frac{1}{4} + 0 \cdot \frac{1}{2} + 1 \cdot \frac{1}{4} \\ & = 0. \end{split} \end{align}\]
Example 23.2 (Expectation of a Bernoulli random variable) Let \(\xi \sim \mbox{Bern}(\mu)\). Then \(\mathbb{E}(\xi) = \mu\).
Proof. \(\xi\) is discrete with \(\mathcal{X} = \{0,1\}\). Thus, by Definition 23.1, \[\begin{align} \begin{split} \mathbb{E}(\xi) & = \sum_{x \in \{0,1\}} x\,\mbox{Bern}(x;\mu) \\ & = 0\cdot \mu^0 (1 - \mu)^{1-0} + 1\cdot \mu^1 (1 - \mu)^{1-1} \\ & = 1\cdot \mu^1 (1 - \mu)^{0} \\ & = \mu. \end{split} \end{align}\]
Example 23.3 (Expectation of a normally distributed random variable) Let \(\xi \sim N(\mu,\sigma^2)\). Then \(\mathbb{E}(\xi) = \mu\).
Proof. We omit the proof.
Intuitively, Definition 23.1 can be explained as shown in Example 23.4.
Example 23.4 (Expectation and mean) We consider a discrete random variable \(\xi\) with outcome space \(\{1,2,3\}\) and PMF defined by \[\begin{equation} p(1) := \frac{1}{4}, \quad p(2) := \frac{1}{4}, \quad p(3) := \frac{1}{2}. \end{equation}\] Thus, \[\begin{align} \begin{split} \mathbb{E}(\xi) & = \sum_{x \in \{1,2,3\}}xp(x) \\ & = 1 \cdot p(1) + 2 \cdot p(2) + 3 \cdot p(3) \\ & = 1 \cdot \frac{1}{4} + 2 \cdot \frac{1}{4} + 3 \cdot \frac{1}{2} \\ & = \frac{9}{4}. \\ \end{split} \end{align}\] Now imagine \(n = 100\) independent realizations \(x_1,....,x_{100}\) of the random variable. According to its PMF, we would expect roughly 25 ones, 25 twos, and 50 threes. Suppose, concretely, that we had 27 ones, 22 twos, and 51 threes. The mean of these numbers is then (cf. Chapter 3) \[\begin{align} \begin{split} \bar{x} & = \frac{1}{100}\sum_{i=1}^n x_i \\ & = \frac{1}{100}\left(27 \cdot 1 + 22 \cdot 2 + 51 \cdot 3 \right) \\ & = \frac{1}{100}\left(1 \cdot 27 + 2 \cdot 22 + 3 \cdot 51 \right) \\ & = 1 \cdot \frac{27}{100} + 2 \cdot \frac{22}{100} + 3 \cdot \frac{51}{100} \\ & = 1 \cdot 0.27 + 2 \cdot 0.22 + 3 \cdot 0.51 \\ & = 2.24. \end{split} \end{align}\] If, in this calculation, the factors \(0.27 \approx \frac{1}{4}\), \(0.22 \approx \frac{1}{4}\), and \(0.51 \approx \frac{1}{2}\) are understood as estimates \(\hat{p}(1)\), \(\hat{p}(2)\), and \(\hat{p}(3)\) of the PMF values \(p(1)\), \(p(2)\), and \(p(3)\), then \[\begin{equation} \bar{x} = \sum_{x \in \{1,2,3\}} x \hat{p}(x) \approx \sum_{x \in \{1,2,3\}} x p(x) = \mathbb{E}(\xi) \end{equation}\] makes the formal similarity between forming a mean of realizations of a discrete random variable and the definition of the expectation of such a random variable immediately apparent.
In distinction to Definition 23.1, and building on Example 23.4, we define the so-called sample mean.
Definition 23.2 (Sample mean) Let \(\xi_1,...,\xi_n\) be random variables. Then \(\xi_1,...,\xi_n\) are also called a sample. The sample mean of \(\xi_1,...,\xi_n\) is defined as the arithmetic mean \[\begin{equation} \bar{\xi} := \frac{1}{n}\sum_{i=1}^n \xi_i. \end{equation}\]
It is central to recognize that \(\mathbb{E}(\xi)\) is a fixed characteristic of a random variable \(\xi\) and, in particular, is not itself a random variable. By contrast, \(\bar{\xi}\) is a characteristic of a sample \(\xi_1,...,\xi_n\) and, as a function of random variables, is therefore itself a random variable. In Example 23.4, for example, each realization \(x_i\) for \(i = 1,...,n\) is modeled as a realization of the random variable \(\xi_i := \xi\) for \(i = 1,...,n\). Since \(\bar{\xi}\) is a scaled sum of random variables, \(\bar{\xi}\) is also itself a random variable. We denote realizations of the random variable \(\bar{\xi}\) by \(\bar{x}\), as in the following.
Example 23.5 (Sample mean) As an example of the realization of a sample mean, we consider the realizations of the random variables \(\xi_1,...,\xi_{10} \sim N(1,2)\) shown in Table 23.1.
| \(x_1\) | \(x_2\) | \(x_3\) | \(x_4\) | \(x_5\) | \(x_6\) | \(x_7\) | \(x_8\) | \(x_9\) | \(x_{10}\) |
|---|---|---|---|---|---|---|---|---|---|
| 0.54 | 1.01 | -3.28 | 0.35 | 2.75 | -0.51 | 2.32 | 1.49 | 0.96 | 1.25 |
The sample mean realization is \[\begin{equation} \bar{x} = \frac{1}{10}\sum_{i = 1}^{10}x_i = \frac{6.88}{10} = 0.68. \end{equation}\]
Generalizing Definition 23.1, the following definition applies to the expectation of a function of a random variable.
Definition 23.3 (Expectation of a function of a random variable) Let \((\Omega, \mathcal{A},\mathbb{P})\) be a probability space, let \(\xi\) be a random variable with outcome space \(\mathcal{X}\), and let \(f: \mathcal{X} \to \mathcal{Z}\) be a function with target set \(\mathcal{Z}\). Then the expectation of the function \(f\) of the random variable \(\xi\) is defined as
- \(\mathbb{E}(f(\xi)) := \sum_{x \in \mathcal{X}} f(x)\,p(x)\) if \(\xi : \Omega \to \mathcal{X}\) is discrete with PMF \(p\),
- \(\mathbb{E}(f(\xi)) := \int_{-\infty}^\infty f(x) \,p(x)\,dx\) if \(\xi : \Omega \to \mathbb{R}\) is continuous with PDF \(p\).
The expectation of a random variable follows from Definition 23.3 as the special case in which \[\begin{equation} f : \mathcal{X} \to \mathcal{Z}, x \mapsto f(x) := x. \end{equation}\] In the English-language literature, Definition 23.3 is also known as the “law of the unconscious statistician” and is often used directly to define expectation. The following theorem gives a first example of the expectation of a function of a random variable.
Theorem 23.1 (Expectation under linear-affine transformation of a random variable) Let \(\xi\) be a random variable with outcome space \(\mathcal{X}\) and let \[\begin{equation} f: \mathcal{X} \to \mathcal{Z}, x \mapsto f(x) := ax + b \mbox{ for } a,b \in \mathbb{R} \end{equation}\] be a linear-affine function. Then \[\begin{equation} \mathbb{E}(f(\xi)) = \mathbb{E}(a\xi + b) = a\mathbb{E}(\xi) + b. \end{equation}\]
Proof. The statement of the theorem follows from the linearity properties of sums and integrals. We consider the case of a discrete random variable \(\xi\) with finite outcome space \(\mathcal{X}\) and PMF \(p\). We have \[\begin{align} \begin{split} \mathbb{E}(f(\xi)) & = \mathbb{E}(a\xi + b) \\ & = \sum_{x \in \mathcal{X}} (ax + b)p(x) \\ & = \sum_{x \in \mathcal{X}} axp(x) + b p(x) \\ & = a\sum_{x \in \mathcal{X}} xp(x) + b \sum_{x \in \mathcal{X}} p(x) \\ & = a\mathbb{E}(\xi) + b. \end{split} \end{align}\]
If one considers a random vector instead of a random variable, the definition of expectation extends as follows.
Definition 23.4 (Expectation of a random vector) Let \(\xi\) be an \(n\)-dimensional random vector. Then the expectation of \(\xi\) is defined as the \(n\)-dimensional real vector \[\begin{equation} \mathbb{E}(\xi) := \begin{pmatrix} \mathbb{E}(\xi_1) \\ \vdots \\ \mathbb{E}(\xi_n) \end{pmatrix}. \end{equation}\]
The expectation of a random vector is thus the vector of expectations of its components. Computing the expectation of a random vector is therefore reduced to determining the expectations of the random variables constituting the random vector with respect to their marginal distributions. In analogy to Definition 23.3, one defines the expectation of this transformation for a function of a random vector as follows.
Definition 23.5 (Expectation of a function of a random vector) Let \((\Omega, \mathcal{A},\mathbb{P})\) be a probability space, let \(\xi\) be a random vector with outcome space \(\mathcal{X}\), and let \(f: \mathcal{X} \to \mathcal{Z}\) be a function with target set \(\mathcal{Z}\). Then the expectation of the function \(f\) of the random vector \(\xi\) is defined as
- \(\mathbb{E}(f(\xi)) := \sum_{x \in \mathcal{X}} f(x)\,p(x)\) if \(\xi : \Omega \to \mathcal{X}\) is discrete with PMF \(p\),
- \(\mathbb{E}(f(\xi)) := \int_{-\infty}^\infty f(x) \,p(x)\,dx\) if \(\xi : \Omega \to \mathbb{R}\) is continuous with PDF \(p\).
23.2 Properties
The following statements are particularly relevant when calculating with samples.
Theorem 23.2 (Expectation under linear-affine combination of random variables) Let \(\xi\) be an \(n\)-dimensional random vector with components \(\xi_1,...,\xi_n\) and outcome space \(\mathcal{X} := \mathcal{X}_1 \times \cdots \times \mathcal{X}_n\). Furthermore, let \[\begin{equation} f : \mathcal{X} \to \mathcal{Z}, x \mapsto f(x) := a_0 + \sum_{i=1}^n a_ix_i \mbox{ for } a_0,...,a_n \in \mathbb{R}. \end{equation}\] be a linear-affine combination of the components of \(\xi\). Then \[\begin{equation} \mathbb{E}(f(\xi)) = \mathbb{E}\left(a_0 +\sum_{i=1}^n a_i\xi_i \right) = a_0 + \sum_{i = 1}^n a_i \mathbb{E}(\xi_i). \end{equation}\]
Proof. As shown below, the theorem follows from the linearity properties of sums and integrals, as well as Fubini’s theorem for interchanging the order of summation or integration. We restrict ourselves to the discrete case, where, to simplify notation, we write \(\sum_{x_i \in \mathcal{X}_i}\) as \(\sum_{x_i}\). Then \[\begin{align} \begin{split} \mathbb{E}(f(\xi)) & = \mathbb{E}\left(a_0 + \sum_{i=1}^{n} a_i\xi_i\right) \\ & = \mathbb{E}(a_0 + a_1\xi_1 + \cdots + a_{n}\xi_{n}) \\ & = \sum_{x_1}\cdots\sum_{x_n} (a_0 + a_1x_1 + \cdots + a_{n}x_{n})p(x_1,...,x_{n}) \\ & = \sum_{x_1}\cdots\sum_{x_n} a_0p(x_1,...,x_n) + a_1x_1p(x_1,...,x_n) + \cdots + a_nx_np(x_1,...,x_n) \\ & = \sum_{x_1}\cdots\sum_{x_n} a_0p(x_1,...,x_n) + \sum_{x_1}\cdots\sum_{x_n} a_1x_1p(x_1,...,x_n) + \cdots + \sum_{x_1}\cdots\sum_{x_n} a_nx_np(x_1,...,x_n) \\ & = a_0\sum_{x_1}\cdots\sum_{x_n}p(x_1,...,x_n) + a_1\sum_{x_1}\cdots\sum_{x_n} x_1p(x_1,...,x_n) + \cdots + a_n\sum_{x_n}\cdots\sum_{x_1} x_np(x_1,...,x_n) \\ & = a_0 + a_1\sum_{x_1}x_1 \sum_{x_2} \cdots\sum_{x_n} p(x_1,...,x_n) + \cdots + a_n\sum_{x_n}x_n \sum_{x_{n-1}} \cdots\sum_{x_1} p(x_1,...,x_n) \\ & = a_0 + a_1\sum_{x_1}x_1 p(x_1) + \cdots + a_n\sum_{x_n}x_n p(x_n) \\ & = a_0 + a_1\mathbb{E}(\xi_1) + \cdots + a_n\mathbb{E}(\xi_n) \\ & = a_0 + \sum_{i=1}^n a_i\mathbb{E}(\xi_i) \end{split} \end{align}\] The result follows analogously for a continuous random vector.
Theorem 23.3 (Expectation of the product of independent random variables) Let \(\xi\) be an \(n\)-dimensional random vector with independent components \(\xi_1,...,\xi_n\) and outcome space \(\mathcal{X} := \mathcal{X}_1 \times \cdots \times \mathcal{X}_n\). Furthermore, let \[\begin{equation} f : \mathcal{X} \to \mathcal{Z}, x \mapsto f(x) := \prod_{i=1}^n x_i. \end{equation}\] be the product of the components of \(\xi\). Then \[\begin{equation} \mathbb{E}(f(\xi)) = \prod_{i=1}^n \mathbb{E}(\xi_i). \end{equation}\]
Proof. As shown below, the theorem follows from the linearity properties of sums and integrals, as well as Fubini’s theorem for interchanging the order of summation or integration. We restrict ourselves to the discrete case, where, to simplify notation, we write \(\sum_{x_i \in \mathcal{X}_i}\) as \(\sum_{x_i}\). Since the components \(\xi_1,...,\xi_n\) are assumed to be independent, the PMF of the random vector first satisfies \[\begin{equation} p(x_1,...,x_n) = \prod_{i=1}^n p(x_i). \end{equation}\] Furthermore, \[\begin{align} \begin{split} \mathbb{E}\left(\prod_{i=1}^n \xi_i\right) & = \sum_{x_1} \cdots \sum_{x_n} \left(\prod_{i=1}^n x_i\right)p(x_1,...,x_n) \\ & = \sum_{x_1} \cdots \sum_{x_n} \prod_{i=1}^n x_i \prod_{i=1}^n p(x_i) \\ & = \sum_{x_1} \cdots \sum_{x_n} \prod_{i=1}^n x_i p(x_i) \\ & = \prod_{i=1}^n \sum_{x_i }x_i p(x_i) \\ & = \prod_{i=1}^n\mathbb{E}(\xi_i). \end{split} \end{align}\]
23.3 Conditional expectation
If, when forming an expectation, one considers the conditional distribution of a random variable instead of the distribution of a random variable, one is accordingly led to the concept of conditional expectation.
Definition 23.6 (Conditional expectation) Given a random vector \(\xi := (\xi_1,\xi_2)\) with outcome space \(\mathcal{X} := \mathcal{X}_1 \times \mathcal{X}_2\), PMF or PDF \(p(x_1,x_2)\), and conditional PMF or PDF \(p(x_1|x_2)\) for all \(x_2 \in \mathcal{X}_2\). Then the conditional expectation of \(\xi_1\) given \(\xi_2 = x_2\) is defined as
- \(\mathbb{E}(\xi_1|\xi_2 = x_2) := \sum_{x_1 \in \mathcal{X}_1}x_1p(x_1|x_2)\) if \(\xi\) is a discrete random vector.
- \(\mathbb{E}(\xi_1|\xi_2 = x_2) := \int_{\mathcal{X}_1}x_1p(x_1|x_2)\,dx_1\) if \(\xi\) is a continuous random vector.
The conditional expectation of a random variable is thus the expectation of a random variable in a conditional distribution. Note that in Definition 23.6 we defined the conditional expectation for a fixed value \(x_2\) of \(\xi_2\). For a fixed value \(x_2\) of \(\xi_2\), \(\mathbb{E}(\xi_1|\xi_2 = x_2)\) is therefore a fixed value, and by exchanging the subscripts one obtains correspondingly \(\mathbb{E}(\xi_2|\xi_1 = x_1)\).
In general, however, \(\mathbb{E}(\xi_1|\xi_2)\) is a random variable because \(\xi_2\) is a random variable and assumes a value \(\xi_2 = x_2\) only with a certain probability. We will see later that conditional variances, conditional covariances, and conditional correlations are defined analogously to Definition 23.6. In psychology, the concept of conditional expectation is central because, in classical test theory, the true scores of persons in test measurements are defined as conditional expectations. To illustrate Definition 23.6, we consider an example for a discrete random vector.
Example
For a two-dimensional random vector \(\xi:= (\xi_1,\xi_2)\) that takes values in \(\mathcal{X} := \mathcal{X}_1 \times \mathcal{X}_2\) with \(\mathcal{X}_1 := \{1,2,3,4\}\) and \(\mathcal{X}_2 = \{1,2,3\}\), let conditional PMFs for \(p(x_1|x_2)\) for all \(x_1 \in \mathcal{X}_1\) be specified as in Table 23.2.
| \(p(x_1|x_2)\) | \(x_2 = 1\) | \(x_2 = 2\) | \(x_2 = 3\) |
|---|---|---|---|
| \(p(x_1 = 1|x_2)\) | \(\frac{1}{4}\) | \(\frac{1}{3}\) | \(0\) |
| \(p(x_1 = 2|x_2)\) | \(0\) | \(\frac{2}{3}\) | \(\frac{1}{3}\) |
| \(p(x_1 = 3|x_2)\) | \(\frac{1}{2}\) | \(0\) | \(\frac{1}{3}\) |
| \(p(x_1 = 4|x_2)\) | \(\frac{1}{4}\) | \(0\) | \(\frac{1}{3}\) |
Then the conditional expectations are \[\begin{align} \begin{split} \mathbb{E}(\xi_1|\xi_2 = 1) & = \sum_{x_1 \in \mathcal{X}_1}x_1p(x_1|x_2 = 1) = 1 \cdot \frac{1}{4} + 2 \cdot 0 + 3 \cdot \frac{1}{2} + 4 \cdot \frac{1}{4} = \frac{11}{4} \\ \mathbb{E}(\xi_1|\xi_2 = 2) & = \sum_{x_1 \in \mathcal{X}_1}x_1p(x_1|x_2 = 2) = 1 \cdot \frac{1}{3} + 2 \cdot \frac{2}{3} + 3 \cdot 0 + 4 \cdot 0 = \frac{5}{3} \\ \mathbb{E}(\xi_1|\xi_2 = 3) & = \sum_{x_1 \in \mathcal{X}_1}x_1p(x_1|x_2 = 3) = 1 \cdot 0 + 2 \cdot \frac{1}{3} + 3 \cdot \frac{1}{3} + 4 \cdot \frac{1}{3} = \frac{8}{3}. \end{split} \end{align}\]
The following theorem, the so-called law of iterated expectation, establishes a connection between the marginal expectation of a random variable and the conditional expectation of this random variable. The theorem is sometimes also called the law of total expectation.
Theorem 23.4 (Law of iterated expectation) Given a random vector \(\xi := (\xi_1,\xi_2)\), we have \[\begin{equation} \mathbb{E}(\xi_1) = \mathbb{E}\left(\mathbb{E}\left(\xi_1|\xi_2\right)\right). \end{equation}\]
Proof. We restrict ourselves to the proof for a discrete random vector; the proof for a continuous random vector follows analogously. We have \[\begin{align} \begin{split} \mathbb{E}(\xi_1) & = \sum_{x_1\in \mathcal{X}_1}x_1p(x_1) \\ & = \sum_{x_1 \in \mathcal{X}_1}x_1\sum_{x_2 \in \mathcal{X}_2} p(x_1,x_2) \\ & = \sum_{x_1\in \mathcal{X}_1}x_1\sum_{x_2 \in \mathcal{X}_2} p(x_1|x_2)p(x_2) \\ & = \sum_{x_1\in \mathcal{X}_1}\sum_{x_2 \in \mathcal{X}_2} x_1 p(x_1|x_2)p(x_2) \\ & = \sum_{x_2\in \mathcal{X}_2}\sum_{x_1 \in \mathcal{X}_1} x_1 p(x_1|x_2)p(x_2) \\ & = \sum_{x_2\in \mathcal{X}_2}\mathbb{E}(\xi_1|\xi_2 = x_2)p(x_2) \\ & = \mathbb{E}\left(\mathbb{E}(\xi_1|\xi_2 )\right). \\ \end{split} \end{align}\]
Evidently, the different expectations in Theorem 23.4 refer to expectations with respect to different distributions: the expectation on the left-hand side of the equation denotes the expectation with respect to the marginal distribution of \(\xi_1\), the outer expectation on the right-hand side denotes the expectation with respect to the marginal distribution of \(\xi_2\), and the inner expectation on the right-hand side denotes the conditional expectation of \(\xi_1\) given \(\xi_2\).
Study questions
- State the definition of the expectation of a random variable.
- Explain the concept of the expectation of a random variable.
- Compute the expectation of a Bernoulli random variable.
- State the definition of the concept of a sample.
- State the definition of the sample mean.
- Explain the differences between an expectation and a sample mean.
- State the definition of the expectation of a function of a random variable.
- State the theorem on expectation under linear-affine transformation of a random variable.
- State the definition of the expectation of a random vector.
- State the definition of the expectation of a function of a random vector.
- State the theorem on expectation under linear-affine combination of random variables.
- State the theorem on expectation of the product of independent random variables.
Study question answers
- See Definition 23.1.
- See the explanations following Definition 23.1, especially also Example 23.4.
- See Example 23.2.
- See Definition 23.2.
- See Definition 23.2.
- See the explanations of Example 23.4, Definition 23.2, and Example 23.5.
- See Definition 23.3.
- See Theorem 23.1.
- See Definition 23.4.
- See Definition 23.5.
- See Theorem 23.2.
- See Theorem 23.3.