25 Covariances
With the covariance of two random variables and its standardized form, the correlation, we introduce in this chapter scalar quantities for the linear-affine dependence of random variables.
25.1 Definitions
Definition 25.1 (Covariance and correlation of two random variables) The covariance of two random variables \(\xi_1\) and \(\xi_2\) is defined as \[\begin{equation} \mathbb{C}(\xi_1,\xi_2) := \mathbb{E}\left((\xi_1-\mathbb{E}(\xi_1))(\xi_2-\mathbb{E}(\xi_2))\right). \end{equation}\] The correlation of two random variables \(\xi_1\) and \(\xi_2\) with \(\mathbb{S}(\xi_1) > 0\) and \(\mathbb{S}(\xi_2) > 0\) is defined as \[\begin{equation} \rho(\xi_1,\xi_2) := \frac{\mathbb{C}(\xi_1,\xi_2)}{\mathbb{S}(\xi_1)\mathbb{S}(\xi_2)}. \end{equation}\]
The covariance of a random variable with itself is its variance, since \[\begin{equation} \mathbb{C}(\xi_1,\xi_1) = \mathbb{E}\left((\xi_1-\mathbb{E}(\xi_1))(\xi_1-\mathbb{E}(\xi_1))\right) = \mathbb{E}\left((\xi_1-\mathbb{E}(\xi_1))^2\right) = \mathbb{V}(\xi_1). \end{equation}\] The number \(\rho(\xi_1,\xi_2)\) is also called the correlation coefficient of \(\xi_1\) and \(\xi_2\). If \(\rho(\xi_1,\xi_2) = 0\), then \(\xi_1\) and \(\xi_2\) are called uncorrelated. In the chapter on inequalities, the correlation inequality shows that \[\begin{equation} -1 \le \rho(\xi_1,\xi_2) \le 1. \end{equation}\] The definition of covariance in Definition 25.1 implicitly makes clear that the expectations appearing in it are of different kinds. The definition of covariance first assumes that \(\xi_1\) and \(\xi_2\) are the components of a random vector \(\xi := (\xi_1,\xi_2)\) with outcome space \(\mathcal{X}_1 \times \mathcal{X}_2\), joint distribution \(\mathbb{P}(\xi_1,\xi_2)\), and marginal distributions \(\mathbb{P}(\xi_1)\) and \(\mathbb{P}(\xi_2)\). The expectations \(\mathbb{E}(\xi_1)\) and \(\mathbb{E}(\xi_2)\) are the expectations of the components \(\xi_1\) and \(\xi_2\) with respect to these marginal distributions. The covariance itself is then the expectation of the function \[ f : \mathcal{X}_1 \times \mathcal{X}_2 \to \mathcal{Z}, (x_1,x_2) \mapsto f(x_1,x_2) := (x_1 - \mathbb{E}(\xi_1))(x_2 - \mathbb{E}(\xi_2)) \tag{25.1}\] with respect to the joint distribution \(\mathbb{P}(\xi_1,\xi_2)\). We illustrate this with the example of a discrete random vector.
Example 25.1 (Covariance of two discrete random variables) Let \(\xi := (\xi_1,\xi_2)\) be a discrete random vector with outcome space \(\mathcal{X} := \{1,2\} \times \{1,2,3\}\) and the joint PMF \(p(x_1,x_2)\) and marginal PMFs \(p(x_1)\) and \(p(x_2)\) shown in Table 25.1.
| \(p(x_1,x_2)\) | \(x_2 = 1\) | \(x_2 = 2\) | \(x_2 = 3\) | \(p(x_1)\) |
|---|---|---|---|---|
| \(x_1 = 1\) | \(0.10\) | \(0.05\) | \(0.15\) | \(0.30\) |
| \(x_1 = 2\) | \(0.60\) | \(0.05\) | \(0.05\) | \(0.70\) |
| \(p(x_2)\) | \(0.70\) | \(0.10\) | \(0.20\) |
Using the definition of the covariance of \(\xi_1\) and \(\xi_2\) and taking Equation 25.1 into account, we have \[\begin{align} \begin{split} \mathbb{C}(\xi_1,\xi_2) & = \mathbb{E}(f(\xi_1,\xi_2)) \\ & = \sum_{x_1 = 1}^2 \sum_{x_2 = 1}^3 f(x_1, x_2)p(x_1,x_2) \\ & = \sum_{x_1 = 1}^2 \sum_{x_2 = 1}^3 (x_1-\mathbb{E}(\xi_1))(x_2-\mathbb{E}(\xi_2))p(x_1,x_2). \\ \end{split} \end{align}\] Substituting the values from Table 25.1 first gives \[\begin{equation} \mathbb{E}(\xi_1) = \sum_{x_1 = 1}^2 x_1 p(x_1) = 1\cdot 0.3 + 2\cdot 0.7 = 1.7 \end{equation}\] and \[\begin{equation} \mathbb{E}(\xi_2) = \sum_{x_2=1}^3 x_2 p(x_2) = 1\cdot 0.7 + 2\cdot 0.1 + 3\cdot 0.2 = 1.5. \end{equation}\] Thus, finally, \[\begin{align} \begin{split} \mathbb{C}(\xi_1,\xi_2) & = \sum_{x_1 = 1}^2 \sum_{x_2 = 1}^3 \left(x_1-1.7\right)\left(x_2-1.5\right)p(x_1,x_2) \\ & = \sum_{x_1 = 1}^2 (x_1 -1.7)(1 - 1.5)p(x_1,1) + (x_1 -1.7)(2 - 1.5)p(x_1,2) + (x_1 -1.7)(3 - 1.5)p(x_1,3) \\ & = \quad (1 - 1.7)(1 - 1.5)p(1,1) + (1 - 1.7)(2 - 1.5)p(1,2) + (1 - 1.7)(3 - 1.5)p(1,3) \\ & \quad\quad\,\, (2 - 1.7)(1 - 1.5)p(2,1) + (2 - 1.7)(2 - 1.5)p(2,2) + (2 - 1.7)(3 - 1.5)p(2,3) \\ & = (-0.7)\cdot(-0.5)\cdot 0.10+ (-0.7)\cdot 0.5\cdot 0.05 + (-0.7)\cdot 1.5\cdot 0.15 \\ & \quad\,\, + 0.3\cdot(-0.5)\cdot 0.60 \,\, + 0.3\cdot 0.5\cdot 0.05 \quad\,\, + 0.3\cdot 1.5\cdot 0.05 \\ & = 0.035- 0.0175- 0.1575 - 0.09+ 0.0075+ 0.0225 \\ & = - 0.2. \end{split} \end{align}\] The covariance of the random variables \(\xi_1\) and \(\xi_2\) with the distribution specified in the table above is therefore \(\mathbb{C}(\xi_1,\xi_2) = -0.2\). Unlike the variance, the covariance can therefore evidently also take negative values.
25.2 Properties
We next consider some basic properties of covariance.
Theorem 25.1 (Symmetry of covariance and correlation) Let \(\xi_1\) and \(\xi_2\) be two random variables. Then \[\begin{equation} \mathbb{C}(\xi_1,\xi_2) = \mathbb{C}(\xi_2,\xi_1) \mbox{ and } \rho(\xi_1,\xi_2) = \rho(\xi_2,\xi_1). \end{equation}\]
Proof. By the commutativity of multiplication, we first have \[\begin{equation} \mathbb{C}(\xi_1,\xi_2) = \mathbb{E}\left((\xi_1-\mathbb{E}(\xi_1))(\xi_2-\mathbb{E}(\xi_2))\right) = \mathbb{E}\left((\xi_2-\mathbb{E}(\xi_2))(\xi_1-\mathbb{E}(\xi_1))\right) = \mathbb{C}(\xi_2,\xi_1). \end{equation}\] Furthermore, \[\begin{equation} \rho(\xi_1,\xi_2) = \frac{\mathbb{C}(\xi_1,\xi_2)}{\mathbb{S}(\xi_1)\mathbb{S}(\xi_2)} = \frac{\mathbb{C}(\xi_2,\xi_1)}{\mathbb{S}(\xi_2)\mathbb{S}(\xi_1)} = \rho(\xi_2,\xi_1). \end{equation}\]
As with the calculation of variances, calculating covariances is sometimes made easier by the following theorem.
Theorem 25.2 (Covariance shift theorem) Let \(\xi_1\) and \(\xi_2\) be random variables. Then \[\begin{equation} \mathbb{C}(\xi_1,\xi_2) = \mathbb{E}(\xi_1\xi_2) - \mathbb{E}(\xi_1)\mathbb{E}(\xi_2). \end{equation}\]
Proof. By Definition 25.1, \[\begin{align} \begin{split} \mathbb{C}(\xi_1,\xi_2) & = \mathbb{E}\left((\xi_1-\mathbb{E}(\xi_1))(\xi_2-\mathbb{E}(\xi_2))\right) \\ & = \mathbb{E}\left(\xi_1\xi_2-\xi_1\mathbb{E}(\xi_2)-\mathbb{E}(\xi_1)\xi_2+\mathbb{E}(\xi_1)\mathbb{E}(\xi_2)\right) \\ & = \mathbb{E}(\xi_1\xi_2)-\mathbb{E}(\xi_1)\mathbb{E}(\xi_2)-\mathbb{E}(\xi_1)\mathbb{E}(\xi_2)+\mathbb{E}(\xi_1)\mathbb{E}(\xi_2) \\ & = \mathbb{E}(\xi_1\xi_2)-\mathbb{E}(\xi_1)\mathbb{E}(\xi_2). \end{split} \end{align}\]
Of course, Theorem 25.2 is only truly useful if \(\mathbb{E}(\xi_1\xi_2)\) is easy to compute. The variance shift theorem from Theorem 24.2 follows from Theorem 25.2 for the case \(\xi_1 = \xi_2 := \xi\) by \[\begin{equation} \mathbb{V}(\xi) = \mathbb{C}(\xi,\xi) = \mathbb{E}(\xi\xi) - \mathbb{E}(\xi)\mathbb{E}(\xi) = \mathbb{E}(\xi^2) - \mathbb{E}(\xi)\mathbb{E}(\xi). \end{equation}\]
With regard to the covariance \(\mathbb{C}(\xi_1,\xi_2)\), one is also interested in how it behaves under the application of linear-affine transformations to \(\xi_1\) and \(\xi_2\). The following result shows that the covariance of two random variables depends on their scale, whereas the correlation does not. This motivates the frequent use of correlation as a general measure of association that is comparable across different application scenarios.
Theorem 25.3 (Covariance and correlation under linear-affine transformation) Let \(\xi_1\) and \(\xi_2\) be random variables with joint outcome space \(\mathcal{X}_1 \times \mathcal{X}_2\), and let \[\begin{align} \begin{split} f_1 : \mathcal{X}_1 \to \mathcal{Z}_1, x_1 \mapsto f_1(x_1) := ax_1 + b \\ f_2 : \mathcal{X}_2 \to \mathcal{Z}_2, x_2 \mapsto f_2(x_2) := cx_2 + d \end{split} \end{align}\] be two linear-affine functions for \(a,b,c,d\in \mathbb{R}\). Then \[\begin{equation} \mathbb{C}(f_1(\xi_1), f_2(\xi_2)) = \mathbb{C}(a\xi_1 + b, c\xi_2 + d) = ac\mathbb{C}(\xi_1,\xi_2) \end{equation}\] and \[\begin{equation} \rho(f_1(\xi_1), f_2(\xi_2)) = \rho(a\xi_1 + b, c\xi_2 + d) = \rho(\xi_1,\xi_2). \end{equation}\]
Proof. First, \[\begin{align} \begin{split} \mathbb{C}(a\xi_1+b,c\xi_2+d) & = \mathbb{E}((a\xi_1+b-\mathbb{E}(a\xi_1+b))(c\xi_2+d-\mathbb{E}(c\xi_2+d))) \\ & = \mathbb{E}((a\xi_1+b-a\mathbb{E}(\xi_1)-b)(c\xi_2+d-c\mathbb{E}(\xi_2)-d)) \\ & = \mathbb{E}(a(\xi_1-\mathbb{E}(\xi_1))c(\xi_2-\mathbb{E}(\xi_2))) \\ & = \mathbb{E}(ac((\xi_1-\mathbb{E}(\xi_1))(\xi_2-\mathbb{E}(\xi_2)))) \\ & = ac\mathbb{C}(\xi_1,\xi_2). \end{split} \end{align}\] Thus, \[\begin{align} \begin{split} \rho(a\xi_1 + b, c\xi_2 + d) & = \frac{\mathbb{C}(a\xi_1+b,c\xi_2+d)}{\sqrt{\mathbb{V}(a\xi_1+b)}\sqrt{\mathbb{V}(c\xi_2+d)}} \\ & = \frac{ac\mathbb{C}(\xi_1,\xi_2)}{\sqrt{a^2\mathbb{V}(\xi_1)}\sqrt{c^2\mathbb{V}(\xi_2)}} \\ & = \frac{ac\mathbb{C}(\xi_1,\xi_2)}{a\mathbb{S}(\xi_1)c\mathbb{S}(\xi_2)} \\ & = \frac{\mathbb{C}(\xi_1,\xi_2)}{\mathbb{S}(\xi_1)\mathbb{S}(\xi_2)} \\ & = \rho(\xi_1,\xi_2). \end{split} \end{align}\]
With the concept of covariance, it is possible to make more general statements about the variances of sums and differences of random variables than in Chapter 24, where only the variances of sums of independent random variables were considered. With the following theorem, we first investigate the very general case of the covariance of two random variables that arise as linear combinations of \(n\) random variables \(\xi_1,...,\xi_n\) and \(m\) random variables \(\zeta_1,...,\zeta_m\), with real constants added. This yields a number of special cases that are important in applications, especially in classical test theory.
Theorem 25.4 (Covariance of linear combinations of random variables) Let \(n\) random variables \(\xi_1,...,\xi_n\) and \(n+1\) real constants \(a_0,a_1,...,a_n\) be given, and let \(m\) random variables \(\zeta_1,...,\zeta_m\) and \(m+1\) real constants \(b_0,b_1,...,b_m\) be given. Then \[\begin{equation} \mathbb{C}\left(a_0 + \sum_{i=1}^n a_i \xi_i, b_0 + \sum_{j=1}^m b_j \zeta_j\right) = \sum_{i=1}^n \sum_{j=1}^m a_ib_j\mathbb{C}(\xi_i, \zeta_j). \end{equation}\]
Proof. \[\begin{align*} \begin{split} & \mathbb{C}\left(a_0 + \sum_{i=1}^n a_i \xi_i, b_0 + \sum_{j=1}^m b_j \zeta_j \right) \\ & = \mathbb{E}\left( \left(a_0 + \sum_{i=1}^n a_i \xi_i - \mathbb{E}\left(a_0 + \sum_{i=1}^n a_i \xi_i\right)\right) \left(b_0 + \sum_{j=1}^m b_j \zeta_j - \mathbb{E}\left(b_0 + \sum_{j=1}^m b_j \zeta_j \right)\right) \right) \\ & = \mathbb{E}\left( \left(a_0 + \sum_{i=1}^n a_i \xi_i - a_0 - \mathbb{E}\left(\sum_{i=1}^n a_i \xi_i\right)\right) \left(b_0 + \sum_{j=1}^m b_j \zeta_j - b_0 - \mathbb{E}\left(\sum_{j=1}^m b_j \zeta_j \right)\right) \right) \\ & = \mathbb{E}\left( \left(\sum_{i=1}^n a_i \xi_i - \sum_{i=1}^n a_i \mathbb{E}(\xi_i) \right) \left(\sum_{j=1}^m b_j \zeta_j - \sum_{j=1}^m b_j \mathbb{E}(\zeta_j)\right) \right) \\ & = \mathbb{E} \left( \sum_{i=1}^n a_i \xi_i\sum_{j=1}^m b_j \zeta_j - \sum_{i=1}^n a_i \xi_i \sum_{j=1}^m b_j \mathbb{E}(\zeta_j) - \sum_{i=1}^n a_i \mathbb{E}(\xi_i)\sum_{j=1}^m b_j \zeta_j + \sum_{i=1}^n a_i \mathbb{E}(\xi_i) \sum_{j=1}^m b_j \mathbb{E}(\zeta_j) \right) \\ & = \mathbb{E}\left(\sum_{i=1}^n a_i \xi_i\sum_{j=1}^m b_j \zeta_j\right) - \mathbb{E}\left(\sum_{i=1}^n a_i \xi_i \sum_{j=1}^m b_j \mathbb{E}(\zeta_j)\right) - \mathbb{E}\left(\sum_{i=1}^n a_i \mathbb{E}(\xi_i)\sum_{j=1}^m b_j \zeta_j\right) + \mathbb{E}\left(\sum_{i=1}^n a_i \mathbb{E}(\xi_i) \sum_{j=1}^m b_j \mathbb{E}(\zeta_j)\right) \\ & = \sum_{i=1}^n \sum_{j=1}^m a_ib_j \mathbb{E}(\xi_i\zeta_j) - 2\sum_{i=1}^n \sum_{j=1}^m a_i b_j \mathbb{E}(\xi_i)\mathbb{E}(\zeta_j) + \sum_{i=1}^n\sum_{j=1}^m a_ib_j \mathbb{E}(\xi_i) \mathbb{E}(\zeta_j) \\ & = \sum_{i=1}^n \sum_{j=1}^m a_ib_j \left(\mathbb{E}(\xi_i\zeta_j) - 2 \mathbb{E}(\xi_i)\mathbb{E}(\zeta_j) +\mathbb{E}(\xi_i)\mathbb{E}(\zeta_j)\right) \\ & = \sum_{i=1}^n \sum_{j=1}^m a_ib_j \mathbb{E}\left((\xi_i - \mathbb{E}(\xi_i))(\zeta_j - \mathbb{E}(\zeta_j))\right) \\ & = \sum_{i=1}^n \sum_{j=1}^m a_ib_j\mathbb{C}(\xi_i, \zeta_j). \end{split} \end{align*}\]
With the following theorem, we now consider a first special case of Theorem 25.4.
Theorem 25.5 (Covariance under pairwise addition of random variables) Let \(\xi_1,\xi_2,\zeta_1,\zeta_2\) be four random variables. Then \[\begin{equation} \mathbb{C}(\xi_1 + \xi_2, \zeta_1 + \zeta_2) = \mathbb{C}(\xi_1,\zeta_1) + \mathbb{C}(\xi_1,\zeta_2) + \mathbb{C}(\xi_2,\zeta_1) + \mathbb{C}(\xi_2,\zeta_2). \end{equation}\]
Proof. Let \(n := 2, m := 2, a_0 = b_0 := 0\), and \(a_i = b_i := 1\) for \(i = 1,2\). Then, by Theorem 25.4, \[\begin{align} \begin{split} \mathbb{C}\left(\xi_1 +\xi_2, \zeta_1 + \zeta_2 \right) & = \mathbb{C}\left(a_0 + \sum_{i=1}^2 a_i \xi_i, b_0 + \sum_{j=1}^2 b_j \zeta_j \right) \\ & = \sum_{i=1}^2 \sum_{j=1}^2 a_i b_j\mathbb{C}\left(\xi_i, \zeta_j \right) \\ & = \sum_{i=1}^2 \sum_{j=1}^2 \mathbb{C}\left(\xi_i, \zeta_j \right) \\ & = \mathbb{C}\left(\xi_1, \zeta_1\right) + \mathbb{C}\left(\xi_1, \zeta_2 \right) + \mathbb{C}\left(\xi_2, \zeta_1\right) + \mathbb{C}\left(\xi_2, \zeta_2 \right). \end{split} \end{align}\]
The following theorem states how, in general, the variance of a linear combination of random variables with an added constant can be determined.
Theorem 25.6 (Variance of a linear combination of random variables) Let \(n\) random variables \(\xi_1,...,\xi_n\) and \(n+1\) real constants \(a_0,a_1,...,a_n\) be given. Then \[\begin{equation} \mathbb{V}\left(a_0 + \sum_{i=1}^n a_i \xi_i \right) = \sum_{i=1}^n a_i^2 \mathbb{V}(\xi_i) + 2 \sum_{i=1}^{n-1}\sum_{j=i+1}^n a_ia_j\mathbb{C}(\xi_i,\xi_j). \end{equation}\]
Proof. With \(\mathbb{V}(\xi) = \mathbb{C}(\xi,\xi)\) and Theorem 25.4, we first have \[\begin{align} \begin{split} \mathbb{V}\left(a_0 + \sum_{i=1}^n a_i\xi_i\right) & = \mathbb{C}\left(a_0 + \sum_{i=1}^n a_i\xi_i, a_0 + \sum_{i=1}^n a_i\xi_i\right) \\ & = \mathbb{C}\left(a_0 + \sum_{i=1}^n a_i\xi_i, a_0 + \sum_{j=1}^n a_j \xi_j \right) \\ & = \sum_{i=1}^n \sum_{j=1}^n a_ia_j \mathbb{C}(\xi_i,\xi_j) \\ & = \sum_{i=1}^n\sum_{\substack{j=1 \\ j = i}}^n a_ia_j \mathbb{C}(\xi_i,\xi_j) + \sum_{i=1}^n \sum_{\substack{j=1 \\ j \neq i}}^n a_ia_j \mathbb{C}(\xi_i,\xi_j) \\ & = \sum_{i=1}^n a_i a_i\mathbb{C}(\xi_i,\xi_i) + \sum_{i=1}^n \sum_{\substack{j=1 \\ j \neq i}}^n a_ia_j \mathbb{C}(\xi_i,\xi_j) \\ & = \sum_{i=1}^n a_i^2\mathbb{V}(\xi_i) + \sum_{i=1}^n \sum_{\substack{j=1 \\ j \neq i}}^n a_ia_j \mathbb{C}(\xi_i,\xi_j) \\ & = \sum_{i=1}^n a_i^2\mathbb{V}(\xi_i) + 2 \sum_{i=1}^{n-1}\sum_{j=i+1}^na_ia_j \mathbb{C}(\xi_i,\xi_j). \end{split} \end{align}\] In the fourth equality, the double sum was split into those terms for which \(i = j\) and those for which \(i \neq j\). In the seventh equality, we used that \[\begin{equation} a_ia_j\mathbb{C}(\xi_i,\xi_j) = a_ja_i\mathbb{C}(\xi_j,\xi_i). \end{equation}\] We illustrate the resulting identity \[\begin{equation} \sum_{i=1}^n \sum_{\substack{j=1 \\ j \neq i}}^n a_ia_j \mathbb{C}(\xi_i,\xi_j) = 2 \sum_{i=1}^{n-1}\sum_{j=i+1}^na_ia_j \mathbb{C}(\xi_i,\xi_j), \end{equation}\] for the example \(n = 3\) below. Analogously, one may imagine summing over all elements except the diagonal elements of the \(i = 1,...,n\) rows and \(j = 1,...,n\) columns of a symmetric matrix with entries \(a_ia_j\mathbb{C}(\xi_i,\xi_j)\). The left side of the equation above then corresponds to summing row by row over all column entries except the one in the column of the row currently considered, that is, the diagonal entry. The right side corresponds to exploiting the symmetry of the matrix, that is, taking into account that the entries of the matrix above and below the diagonal are identical, so it is enough to add the entries of the upper triangle and double them. For each row \(i = 1,...,n\), only the columns \(j = i+1,...,n\) to the right of the diagonal are considered and their entries are summed. The last row contains only a diagonal element, which is not part of the sum, so the index of the outer sum only runs to \(n-1\). Concretely, for \(n := 3\), \[\begin{align} \begin{split} & \sum_{i=1}^3\sum_{\substack{j=1 \\ j \neq i}}^3 a_ia_j \mathbb{C}(\xi_i,\xi_j) \\ & = \sum_{\substack{j=1 \\ j \neq 1}}^3 a_1a_j\mathbb{C}(\xi_1,\xi_j) + \sum_{\substack{j=1 \\ j \neq 2}}^3 a_2a_j\mathbb{C}(\xi_2,\xi_j) + \sum_{\substack{j=1 \\ j \neq 3}}^3 a_3a_j\mathbb{C}(\xi_3,\xi_j)\\ & = a_1a_2\mathbb{C}(\xi_1,\xi_2) + a_1a_3\mathbb{C}(\xi_1,\xi_3) + a_2a_1\mathbb{C}(\xi_2,\xi_1) + a_2a_3\mathbb{C}(\xi_2,\xi_3) + a_3a_1\mathbb{C}(\xi_3,\xi_1) + a_3a_2\mathbb{C}(\xi_3,\xi_2)\\ & = a_1a_2\mathbb{C}(\xi_1,\xi_2) + a_2a_1\mathbb{C}(\xi_2,\xi_1) + a_1a_3\mathbb{C}(\xi_1,\xi_3) + a_3a_1\mathbb{C}(\xi_3,\xi_1) + a_2a_3\mathbb{C}(\xi_2,\xi_3) + a_3a_2\mathbb{C}(\xi_3,\xi_2)\\ & = 2a_1a_2\mathbb{C}(\xi_1,\xi_2) + 2a_1a_3\mathbb{C}(\xi_1,\xi_3) + 2a_2a_3\mathbb{C}(\xi_2,\xi_3)\\ & = 2\left(a_1a_2\mathbb{C}(\xi_1,\xi_2) + a_1a_3\mathbb{C}(\xi_1,\xi_3) + a_2a_3\mathbb{C}(\xi_2,\xi_3)\right)\\ & = 2\left(\sum_{j=2}^3 a_1a_j\mathbb{C}(\xi_1,\xi_j) + \sum_{j=3}^3 a_2a_j\mathbb{C}(\xi_2,\xi_j)\right)\\ & = 2\left(\sum_{j=1+1}^3 a_1a_j\mathbb{C}(\xi_1,\xi_j) + \sum_{j=2+1}^3 a_2a_j\mathbb{C}(\xi_2,\xi_j)\right)\\ & = 2\sum_{i=1}^{2}\sum_{j=i+1}^3 a_ia_j\mathbb{C}(\xi_i,\xi_j). \end{split} \end{align}\]
Theorem 25.7 (Variances of special linear combinations of random variables)
(1) (Variance under addition of two random variables). Let two random variables \(\xi\) and \(\zeta\) be given. Then \[\begin{equation} \mathbb{V}(\xi + \zeta) = \mathbb{V}(\xi) + \mathbb{V}(\zeta) + 2\mathbb{C}(\xi,\zeta). \end{equation}\] (2) (Variance under subtraction of two random variables). Let two random variables \(\xi\) and \(\zeta\) be given. Then \[\begin{equation} \mathbb{V}(\xi - \zeta) = \mathbb{V}(\xi) + \mathbb{V}(\zeta) - 2\mathbb{C}(\xi,\zeta). \end{equation}\]
Proof. Let \(n := 2\), \(\xi_1 := \xi\), \(\xi_2 := \zeta\), \(a_0 := 0\), \(a_1 := 1\), and \(a_2 := 1\). Then, by Theorem 25.6, \[\begin{align} \begin{split} \mathbb{V}(\xi + \zeta) & = \mathbb{V}(a_0 + a_1\xi_1 + a_2\xi_2) \\ & = \mathbb{V}\left(a_0 + \sum_{i=1}^2 a_i \xi_i \right) \\ & = \sum_{i=1}^2 a_i^2 \mathbb{V}(\xi_i) + 2 \sum_{i=1}^{2-1}\sum_{j = i+ 1}^2 a_i a_j\mathbb{C}(\xi_i,\xi_j) \\ & = \sum_{i=1}^2 a_i^2 \mathbb{V}(\xi_i) + 2 \sum_{i=1}^{1}\sum_{j = i + 1}^2 a_ia_j \mathbb{C}(\xi_i,\xi_j) \\ & = \sum_{i=1}^2 a_i^2 \mathbb{V}(\xi_i) + 2 \sum_{j = 1 + 1}^2 a_ia_j \mathbb{C}(\xi_1,\xi_j) \\ & = \sum_{i=1}^2 a_i^2 \mathbb{V}(\xi_i) + 2 \sum_{j = 2}^2 a_1a_j\mathbb{C}(\xi_1,\xi_j) \\ & = a_1^2\mathbb{V}(\xi_1) + a_2^2\mathbb{V}(\xi_2) + 2 a_1a_2\mathbb{C}(\xi_1,\xi_2) \\ & = 1^2 \cdot \mathbb{V}(\xi) + 1^2 \cdot \mathbb{V}(\zeta) + 2 \cdot 1 \cdot 1 \cdot \mathbb{C}(\xi,\zeta) \\ & = \mathbb{V}(\xi) + \mathbb{V}(\zeta) + 2 \mathbb{C}(\xi,\zeta). \\ \end{split} \end{align}\] Now let \(n := 2\), \(\xi_1 := \xi\), \(\xi_2 := \zeta\), \(a_0 := 0\), \(a_1 := 1\), and \(a_2 := -1\). Then, by Theorem 25.6, \[\begin{align} \begin{split} \mathbb{V}(\xi - \zeta) & = \mathbb{V}(a_0 + a_1\xi_1 + a_2\xi_2) \\ & = \mathbb{V}\left(a_0 + \sum_{i=1}^2 a_i \xi_i \right) \\ & = \sum_{i=1}^2 a_i^2 \mathbb{V}(\xi_i) + 2 \sum_{i=1}^{2-1}\sum_{j = i+ 1}^2 a_i a_j\mathbb{C}(\xi_i,\xi_j) \\ & = \sum_{i=1}^2 a_i^2 \mathbb{V}(\xi_i) + 2 \sum_{i=1}^{1}\sum_{j = i + 1}^2 a_ia_j \mathbb{C}(\xi_i,\xi_j) \\ & = \sum_{i=1}^2 a_i^2 \mathbb{V}(\xi_i) + 2 \sum_{j = 1 + 1}^2 a_ia_j \mathbb{C}(\xi_1,\xi_j) \\ & = \sum_{i=1}^2 a_i^2 \mathbb{V}(\xi_i) + 2 \sum_{j = 2}^2 a_1a_j\mathbb{C}(\xi_1,\xi_j) \\ & = a_1^2\mathbb{V}(\xi_1) + a_2^2\mathbb{V}(\xi_2) + 2 a_1a_2\mathbb{C}(\xi_1,\xi_2) \\ & = 1^2\cdot \mathbb{V}(\xi) + (-1)^2 \cdot \mathbb{V}(\zeta) + 2 \cdot 1 \cdot (-1)\cdot \mathbb{C}(\xi,\zeta) \\ & = \mathbb{V}(\xi) + \mathbb{V}(\zeta) - 2 \mathbb{C}(\xi,\zeta). \\ \end{split} \end{align}\] Alternatively, without using Theorem 25.6 and with Theorem 23.2, we have \[\begin{align} \begin{split} & \mathbb{V}(a\xi + b\zeta + c) \\ & = \mathbb{E}\left((a\xi + b\zeta + c - a\mathbb{E}(\xi) - b\mathbb{E}(\zeta) - c)^2\right) \\ & = \mathbb{E}\left((a(\xi - \mathbb{E}(\xi)) + b(\zeta - \mathbb{E}(\zeta)))^2\right) \\ & = \mathbb{E}\left(a^2(\xi - \mathbb{E}(\xi))^2 + 2ab(\xi - \mathbb{E}(\xi))(\zeta - \mathbb{E}(\zeta))) + b^2(\zeta - \mathbb{E}(\zeta))^2\right) \\ & = a^2\mathbb{E}\left((\xi - \mathbb{E}(\xi))^2\right) + b^2\mathbb{E}\left((\zeta - \mathbb{E}(\zeta))^2\right) + 2ab\mathbb{E}\left((\xi - \mathbb{E}(\xi))(\zeta - \mathbb{E}(\zeta))\right) \\ & = a^2\mathbb{V}(\xi)+ b^2\mathbb{V}(\zeta) + 2ab\mathbb{C}(\xi,\zeta). \end{split} \end{align}\] The special cases then follow directly with \(a := b := 1\) and \(a := 1, b := -1\), respectively.
Finally, the following theorem provides a first impression of the relationship between covariance and correlation and the concept of independence of random variables. It turns out that covariance and correlation are sensitive only to certain forms of dependence between random variables and, in particular, that a covariance of zero does not imply independence of the random variables. On the other hand, independence of two random variables always implies that their covariance is zero and that they are therefore uncorrelated. Dependence and independence of random variables are thus much more general concepts for describing the association between random variables than covariance and correlation.
Theorem 25.8 (Correlation and independence of random variables) Let \(\xi_1\) and \(\xi_2\) be two random variables. If \(\xi_1\) and \(\xi_2\) are independent, then \(\mathbb{C}(\xi_1,\xi_2) = 0\), and \(\xi_1\) and \(\xi_2\) are therefore uncorrelated. Conversely, if \(\mathbb{C}(\xi_1,\xi_2) = 0\) and \(\xi_1\) and \(\xi_2\) are therefore uncorrelated, then \(\xi_1\) and \(\xi_2\) are not necessarily independent.
Proof. In a first step, we show that independence of \(\xi_1\) and \(\xi_2\) implies \(\mathbb{C}(\xi_1,\xi_2) = 0\). We then provide an example showing that the covariance of dependent random variables \(\xi_1\) and \(\xi_2\) can indeed be zero.
(1) We show that independence of \(\xi_1\) and \(\xi_2\) implies \(\mathbb{C}(\xi_1,\xi_2) = 0\). For this purpose, we first note that for independent random variables, \[\begin{equation} \mathbb{E}(\xi_1\xi_2) = \mathbb{E}(\xi_1)\mathbb{E}(\xi_2). \end{equation}\] With the covariance shift theorem, it follows that \[\begin{equation} \mathbb{C}(\xi_1,\xi_2) = \mathbb{E}(\xi_1\xi_2) - \mathbb{E}(\xi_1)\mathbb{E}(\xi_2) = \mathbb{E}(\xi_1)\mathbb{E}(\xi_2) - \mathbb{E}(\xi_1)\mathbb{E}(\xi_2) = 0. \end{equation}\] With the definition of the correlation coefficient, it then follows that \(\rho(\xi_1,\xi_2) = 0\), and thus \(\xi_1\) and \(\xi_2\) are uncorrelated.
(2) We now provide an example showing that the covariance of dependent random variables \(\xi_1\) and \(\xi_2\) can be zero. For this purpose, we consider two discrete random variables \(\xi_1\) and \(\xi_2\) with outcome spaces \(\mathcal{X}_1 = \{-1,0,1\}\) and \(\mathcal{X}_2 = \{0,1\}\), the marginal PMF of \(\xi_1\) given by \(p(x_1) := \frac{1}{3}\) for all \(x_1 \in \mathcal{X}_1\), and the definition \[\begin{equation} \xi_2 := \xi_1^2. \end{equation}\] The definition of \(\xi_2\) then implies the conditional PMF shown in Table 25.2.
| \(p(x_2|x_1)\) | \(x_1 = -1\) | \(x_1 = 0\) | \(x_1 = 1\) |
|---|---|---|---|
| \(x_2 = 0\) | \(0\) | \(1\) | \(0\) |
| \(x_2 = 1\) | \(1\) | \(0\) | \(1\) |
The marginal PMF \(p(x_1)\) and the conditional PMF \(p(x_2|x_1)\) then imply the joint PMF shown in Table 25.3.
| \(p(x_1,x_2)\) | \(x_1 = -1\) | \(x_1 = 0\) | \(x_1 = 1\) | \(p(x_2)\) |
|---|---|---|---|---|
| \(x_2 = 0\) | \(0\) | \(1/3\) | \(0\) | \(1/3\) |
| \(x_2 = 1\) | \(1/3\) | \(0\) | \(1/3\) | \(2/3\) |
| \(p(x_1)\) | \(1/3\) | \(1/3\) | \(1/3\) |
For example, by Table 25.3, \[\begin{equation} p(-1,0) = 0 \neq \frac{1}{9} = \frac{1}{3} \cdot \frac{1}{3} = p(-1)p(0). \end{equation}\] The joint PMF of \(\xi_1\) and \(\xi_2\) therefore does not factorize, and \(\xi_1\) and \(\xi_2\) are thus not independent. However, with \[\begin{equation} \mathbb{E}(\xi_1) = \sum_{x_1 \in \mathcal{X}_1} x_1 p(x_1) = -1 \cdot \frac{1}{3} + 0\cdot \frac{1}{3} + 1\cdot\frac{1}{3} = 0 \end{equation}\] and \[\begin{equation} \mathbb{E}(\xi_1\xi_2) = \mathbb{E}(\xi_1\xi_1^2) = \mathbb{E}(\xi_1^3) = \sum_{x_1 \in \mathcal{X}_1} x_1^3 p(x_1) = -1^3 \cdot \frac{1}{3} + 0^3\cdot \frac{1}{3} + 1^3\cdot\frac{1}{3} = 0, \end{equation}\] as well as the covariance shift theorem, it follows that \[\begin{equation} \mathbb{C}(\xi_1,\xi_2) = \mathbb{E}(\xi_1\xi_2) - \mathbb{E}(\xi_1)\mathbb{E}(\xi_2) = \mathbb{E}(\xi_1^3) - \mathbb{E}(\xi_1)\mathbb{E}(\xi_2) = 0 - 0\cdot \mathbb{E}(\xi_2) = 0. \end{equation}\] The covariance of \(\xi_1\) and \(\xi_2\) is therefore zero, and \(\xi_1\) and \(\xi_2\) are thus uncorrelated, even though \(\xi_1\) and \(\xi_2\) are not independent.
25.3 Conditional covariance
Like the expectation and the variance, the covariance can also be considered in a conditional joint distribution of two random variables. This leads to the concept of conditional covariance.
Definition 25.2 (Conditional covariance) Let a random vector \(\xi := (\xi_1,\xi_2,\xi_3)\) with outcome space \(\mathcal{X} := \mathcal{X}_1 \times \mathcal{X}_2\times \mathcal{X}_3\) and PMF or PDF \(p(x_1,x_2,x_3)\), and conditional PMF or PDF \(p(x_1,x_2|x_3)\) for all \(x_3 \in \mathcal{X}_3\), be given. Then the conditional covariance of \(\xi_1\) and \(\xi_2\) given \(\xi_3 = x_3\) is defined as \[\begin{equation} \mathbb{C}(\xi_1,\xi_2|\xi_3 = x_3) = \mathbb{E} \left( \left(\xi_1 - \mathbb{E}\left(\xi_1|\xi_3 = x_3\right)\right) \left(\xi_2 - \mathbb{E}\left(\xi_2|\xi_3 = x_3\right)\right)|\xi_3 = x_3 \right). \end{equation}\]
Thus, conditional covariance is defined in the sense of the conditional expectation of the random vector \((\xi_1,\xi_2)\) and, like the conditional expectation and the conditional variance, is generally a random variable. Finally, we note that the covariance shift theorem also holds for conditional covariance.
Theorem 25.9 (Shift theorem of conditional covariance) Let a random vector \(\xi := (\xi_1,\xi_2,\xi_3)\) be given. Then \[\begin{equation} \mathbb{C}(\xi_1, \xi_2|\xi_3) = \mathbb{E}\left(\xi_1\xi_2|\xi_3\right) - \mathbb{E}\left(\xi_1|\xi_3\right)\mathbb{E}\left(\xi_2|\xi_3\right). \end{equation}\]
Proof. A proof is obtained by replacing the corresponding expectations in the proof of Theorem 25.2 with the corresponding conditional expectations.
25.4 Covariance matrices
The multivariate analog of the variance of a random variable is the covariance matrix of a random vector. In addition to the variances of the components of the random vector, it also encodes their pairwise covariances and is defined as follows.
Definition 25.3 (Covariance matrix of a random vector) Let \(\xi\) be an \(n\)-dimensional random vector. Then the covariance matrix of \(\xi\) is defined as the \(n \times n\) matrix \[\begin{equation} \mathbb{C}(\xi) := \mathbb{E}\left((\xi - \mathbb{E}(\xi))(\xi - \mathbb{E}(\xi))^T \right). \end{equation}\]
The covariance matrix in Definition 25.3 is formally defined analogously to the covariance of two random variables. The following theorem gives a direct reduction of the concept of the covariance matrix of a random vector to the concept of covariance of two random variables known from the univariate context.
Theorem 25.10 (Properties of the covariance matrix) Let \(\xi\) be an \(m\)-dimensional random vector and let \(\mathbb{C}(\xi)\) be its covariance matrix. Then:
(1) (Elements) The elements of \(\mathbb{C}(\xi)\) are the covariances of the components of \(\xi\), \[\begin{equation} \mathbb{C}(\xi) = \left(\mathbb{C}(\xi_i,\xi_j)\right)_{1 \le i,j \le m}. \end{equation}\]
(2) (Covariance matrix shift theorem) We have \[\begin{equation} \mathbb{C}(\xi) = \mathbb{E}\left(\xi\xi^T\right) - \mathbb{E}(\xi)\mathbb{E}(\xi)^T. \end{equation}\] (3) (Linear-affine transformation) For \(A \in \mathbb{R}^{n \times m}\) and \(b \in \mathbb{R}^n\), \[\begin{equation} \mathbb{C}(A\xi +b) = A\mathbb{C}(\xi)A^T. \end{equation}\] (4) (Matrix properties) \(\mathbb{C}(\xi)\) is symmetric and positive semidefinite.
Proof. (1) We have \[\begin{align} \begin{split} \mathbb{C}(\xi) & := \mathbb{E}\left((\xi - \mathbb{E}(\xi))(\xi - \mathbb{E}(\xi))^T \right) \\ & = \mathbb{E} \left( \left( \begin{pmatrix} \xi_1 \\ \vdots \\ \xi_n \end{pmatrix} - \begin{pmatrix} \mathbb{E}(\xi_1) \\ \vdots \\ \mathbb{E}(\xi_n) \end{pmatrix} \right) \left( \begin{pmatrix} \xi_1 \\ \vdots \\ \xi_n \end{pmatrix} - \begin{pmatrix} \mathbb{E}(\xi_1) \\ \vdots \\ \mathbb{E}(\xi_n) \end{pmatrix} \right)^T \right) \\ & = \mathbb{E} \left( \begin{pmatrix} \xi_1 - \mathbb{E}(\xi_1) \\ \vdots \\ \xi_n - \mathbb{E}(\xi_n) \end{pmatrix} \begin{pmatrix} \xi_1 - \mathbb{E}(\xi_1)\\ \vdots \\ \xi_n - \mathbb{E}(\xi_n) \end{pmatrix}^T \right) \\ & = \mathbb{E} \left( \begin{pmatrix} \xi_1 - \mathbb{E}(\xi_1) \\ \vdots \\ \xi_n - \mathbb{E}(\xi_n) \end{pmatrix} \begin{pmatrix} \xi_1 - \mathbb{E}(\xi_1) & \dots & \xi_n - \mathbb{E}(\xi_n) \end{pmatrix} \right) \\ & = \mathbb{E} \begin{pmatrix} (\xi_1 - \mathbb{E}(\xi_1))(\xi_1 - \mathbb{E}(\xi_1)) & \dots & (\xi_1 - \mathbb{E}(\xi_1))(\xi_n - \mathbb{E}(\xi_n)) \\ \vdots & \ddots & \vdots \\ (\xi_n - \mathbb{E}(\xi_n))(\xi_1 - \mathbb{E}(\xi_1)) & \dots & (\xi_n - \mathbb{E}(\xi_n))(\xi_n - \mathbb{E}(\xi_n)) \\ \end{pmatrix} \\ & = \left(\mathbb{E}\left((\xi_i - \mathbb{E}(\xi_i))(\xi_j - \mathbb{E}(\xi_j)) \right) \right)_{1 \le i,j \le n} \\ & = \left(\mathbb{C}(\xi_i,\xi_j)\right)_{1 \le i,j \le n}. \\ \end{split} \end{align}\]
(2) With the properties of expectations, \[\begin{align} \begin{split} \mathbb{C}(\xi) & = \mathbb{E}\left((\xi - \mathbb{E}(\xi))(\xi - \mathbb{E}(\xi))^T \right) \\ & = \mathbb{E}\left(\xi\xi^T - \xi\mathbb{E}(\xi)^T - \mathbb{E}(\xi)\xi^T + \mathbb{E}(\xi)\mathbb{E}(\xi)^T \right) \\ & = \mathbb{E}\left(\xi\xi^T\right) - \mathbb{E}(\xi)\mathbb{E}(\xi)^T - \mathbb{E}(\xi)\mathbb{E}(\xi)^T + \mathbb{E}(\xi)\mathbb{E}(\xi)^T \\ & = \mathbb{E}\left(\xi\xi^T\right) - \mathbb{E}(\xi)\mathbb{E}(\xi)^T. \\ \end{split} \end{align}\]
(3) With the properties of expectations, \[\begin{align} \begin{split} \mathbb{C}(A\xi+b) & = \mathbb{E}\left((A\xi+b-\mathbb{E}(A\xi+b))(A\xi+b-\mathbb{E}(A\xi+b))^T \right) \\ & = \mathbb{E}\left((A\xi+b-A\mathbb{E}(\xi)-b)(A\xi+b-A\mathbb{E}(\xi)-b)^T \right) \\ & = \mathbb{E}\left((A(\xi-\mathbb{E}(\xi)))(A(\xi-\mathbb{E}(\xi)))^T \right) \\ & = \mathbb{E}\left(A(\xi-\mathbb{E}(\xi))(\xi-\mathbb{E}(\xi))^T A^T\right) \\ & = A\mathbb{E}\left((\xi-\mathbb{E}(\xi))(\xi-\mathbb{E}(\xi))^T\right)A^T \\ & = A\mathbb{C}(\xi)A^T. \\ \end{split} \end{align}\]
(4) The symmetry of \(\mathbb{C}(\xi)\) follows from the symmetry of the covariance of random variables, \[\begin{equation} \mathbb{C}(\xi_i,\xi_j) = \mathbb{C}(\xi_j, \xi_i) \mbox{ for all } i = 1,...,m, j = 1,...,m. \end{equation}\] To show the positive semidefiniteness of \(\mathbb{C}(\xi)\), we need to show that \(a^T\mathbb{C}(\xi)a \ge 0\) for all \(a\in \mathbb{R}^m\) with \(a \neq 0_m\). Let \(a \in \mathbb{R}^m\) with \(a\neq 0\). Then, by statement (3) with \(A := a^T \in \mathbb{R}^{1\times m}\), \[\begin{equation} a^T\mathbb{C}(\xi)a = \mathbb{C}(a^T\xi). \end{equation}\] Furthermore, by the definition of the covariance matrix, \[\begin{equation} \mathbb{C}(a^T\xi) = \mathbb{E}\left(\left(a^T\xi-\mathbb{E}(a^T\xi)\right)^2\right) = \mathbb{V}\left(a^T\xi\right). \end{equation}\] Since, by the properties of variance, the variance of the random variable \(a^T\xi\) is always nonnegative, it follows that \[\begin{equation} a^T\mathbb{C}(\xi)a = \mathbb{V}(a^T\xi)\ge 0, \end{equation}\] and thus that \(\mathbb{C}(\xi)\) is positive semidefinite.
The diagonal elements of \(\mathbb{C}(\xi)\) are the variances of the components of \(\xi\), since \[\begin{equation} \mathbb{V}(\xi_i) = \mathbb{C}(\xi_i,\xi_i) \mbox{ for } i = 1,...,m. \end{equation}\] Properties (2) and (3) are essentially analogous to the properties of variance.
The covariance matrix of a random vector \(\xi\) is therefore the matrix of the covariances of the components of \(\xi\). Thus, the covariance matrix is also directly given in terms of the concept of covariance of random variables. Since the covariance of a random variable with itself is known to be its variance, the covariance matrix contains the variances of the components of \(\xi\) on its diagonal.
The following theorem documents a notation for the covariance matrix of a partitioned random vector in terms of expectations of products of random vectors, which is helpful, for example, in canonical correlation analysis.
Theorem 25.11 (Covariance matrices of random vectors) Let \[\begin{equation} \zeta = \begin{pmatrix} \xi \\ \upsilon \end{pmatrix} \mbox{ with } \mathbb{E}(\zeta) := 0_m \end{equation}\] be an \(m_\xi + m_\upsilon\)-dimensional random vector and its expectation vector, respectively. Then the \(m \times m\) covariance matrix of \(\zeta\) can be written as \[\begin{equation} \mathbb{C}(\zeta) = \begin{pmatrix} \Sigma_{\xi\xi} & \Sigma_{\xi_1\xi_2} \\ \Sigma_{\upsilon\xi} & \Sigma_{\upsilon\upsilon} \\ \end{pmatrix} \in \mathbb{R}^{m \times m} \end{equation}\] where \[\begin{align} \begin{split} \Sigma_{\xi\xi} & := \mathbb{E}\left(\xi\xi^T \right) \in \mathbb{R}^{m_\xi \times m_\xi}\\ \Sigma_{\xi_1\xi_2} & := \mathbb{E}\left(\xi_1\xi_2^T \right) \in \mathbb{R}^{m_\xi \times m_\upsilon}\\ \Sigma_{\upsilon\xi} & := \mathbb{E}\left(\upsilon\xi^T \right) \in \mathbb{R}^{m_\upsilon \times m_\xi}\\ \Sigma_{\upsilon\upsilon} & := \mathbb{E}\left(\upsilon\upsilon^T\right) \in \mathbb{R}^{m_\xi \times m_\upsilon}. \end{split} \end{align}\]
Proof. By the definition of the covariance matrix of a random vector, \[\begin{align} \begin{split} \mathbb{C}(\zeta) & = \mathbb{E}\left((\zeta - \mathbb{E}(\zeta))(\zeta - \mathbb{E}(\zeta))^T \right) \\ & = \mathbb{E}\left((\zeta - 0_m)(\zeta - 0_m)^T \right) \\ & = \mathbb{E}\left(\zeta\zeta^T\right)\\ & = \mathbb{E}\left(\begin{pmatrix} \xi \\ \upsilon \end{pmatrix} \begin{pmatrix} \xi^T & \upsilon^T \end{pmatrix} \right) \\ & = \mathbb{E}\left(\begin{pmatrix} \xi\xi^T & \xi_1\xi_2^T \\ \upsilon\xi^T & \upsilon\upsilon^T \end{pmatrix}\right) \\ & = \begin{pmatrix} \mathbb{E}\left(\xi\xi^T\right) & \mathbb{E}\left(\xi_1\xi_2^T\right) \\ \mathbb{E}\left(\upsilon\xi^T\right) & \mathbb{E}\left(\upsilon\upsilon^T\right) \end{pmatrix} \\ & = \begin{pmatrix} \Sigma_{\xi\xi} & \Sigma_{\xi_1\xi_2} \\ \Sigma_{\upsilon\xi} & \Sigma_{\upsilon\upsilon} \\ \end{pmatrix}. \end{split} \end{align}\]
Finally, in some applications one is interested in a normalized, scale-independent representation of the covariances of a random vector. As in the univariate case, the natural approach is to normalize the covariance of two random variables by means of their respective variances in the sense of a correlation. This consideration leads to the concept of the correlation matrix of a random vector.
Definition 25.4 (Correlation matrix) Let \(\xi\) be an \(n\)-dimensional random vector. Then the correlation matrix of \(\xi\) is defined as the \(n \times n\) matrix \[\begin{equation} \mathbb{R}(\xi) := \left(\rho_{ij} \right)_{1 \le i,j\le n} = \left(\frac{\mathbb{C}(\xi_i,\xi_j)}{\sqrt{\mathbb{V}(\xi_i)}\sqrt{\mathbb{V}(\xi_j)}}\right)_{1 \le i,j\le n}. \end{equation}\]
Since the variances of the components of \(\xi\) are the diagonal elements of the covariance matrix of \(\xi\), the correlation matrix is of course implicit in the covariance matrix. Furthermore, as always for correlations, the entries \(\rho_{ij}\), \(1 \le i,j \le n\), of the correlation matrix satisfy \[\begin{equation} \rho_{ij} \in [-1,1] \mbox{ for } 1 \le i,j \le n \mbox{ and } \rho_{ii} = 1 \mbox{ for } 1 \le i \le n. \end{equation}\]
Study questions
- State the definition of covariance and correlation of two random variables.
- Explain the definition of covariance and correlation of two random variables.
- State the theorem on symmetry of covariance and correlation.
- State the theorem on covariance and correlation under linear-affine transformation.
- Explain the theorem on covariance and correlation under linear-affine transformation.
- State the theorem on variances of special linear combinations of random variables.
- State the theorem on correlation and independence of random variables.
Study question answers
- See Definition 25.1.
- See the explanations of Definition 25.1 and Example 25.1.
- See Theorem 25.1.
- See Theorem 25.3.
- See the explanations of Theorem 25.3.
- See Theorem 25.7.
- See Theorem 25.8.