45 Multivariate descriptive statistics
45.1 Data-analysis scenarios
We first want to classify the data-analysis procedures considered below in terms of the dimensionality of their independent variables and dependent variables. As usual, we denote an independent variable by \(x\) and a dependent variable by \(y\). Furthermore, in \(x_{ij}\) and \(y_{ij}\), the subscripts \(i\) and \(j\) denote the value of the \(j\)th univariate component of the respective variable, for example a test score, for the \(i\)th experimental unit, for example a participant. The total number of experimental units is denoted by \(n\).
Table 45.1 shows the scenario of a univariate independent variable and a univariate dependent variable. Typical inference procedures used in this scenario are the determination of the correlation of \(x_1\) and \(y_1\), the performance of a simple linear regression of \(y_1\) on \(x_1\), and, if \(x_1\) is a categorical factor variable, t-tests and analyses of variance.
| \(x_{1}\) | \(y_1\) |
|---|---|
| \(x_{11}\) | \(y_{11}\) |
| \(\vdots\) | \(\vdots\) |
| \(x_{n1}\) | \(y_{n1}\) |
Table 45.2 shows the scenario of a multivariate independent variable and a univariate dependent variable. Typical inference procedures used in this scenario are the determination of multiple and partial correlations between \(x_1,...,x_m\) and \(y_1\), the performance of multiple regression analyses and analyses of covariance, and, in general, all data-analytic special cases of the general linear model.
| \(x_{1}\) | \(\cdots\) | \(x_{m}\) | \(y_{1}\) |
|---|---|---|---|
| \(x_{11}\) | \(\cdots\) | \(x_{1m}\) | \(y_{11}\) |
| \(\vdots\) | \(\ddots\) | \(\vdots\) | \(\vdots\) |
| \(x_{n1}\) | \(\cdots\) | \(x_{nm}\) | \(y_{n1}\) |
Table 45.3 shows the scenario relevant in the context of one-sample T\(^2\)-tests, one-way multivariate analysis of variance, and many scenarios of predictive modeling. In this case, the independent variable is univariate and categorically codes a factor level or group membership, whereas the dependent variable is multivariate. Especially in predictive modeling, the independent variable in this context is also called the target variable or label, and the components of the dependent variable are called features.
| \(x_{1}\) | \(y_{1}\) | \(\cdots\) | \(y_{m}\) |
|---|---|---|---|
| \(x_{11}\) | \(y_{11}\) | \(\cdots\) | \(y_{1m}\) |
| \(\vdots\) | \(\vdots\) | \(\vdots\) | \(\vdots\) |
| \(x_{n1}\) | \(y_{n1}\) | \(\cdots\) | \(y_{nm}\) |
Finally, Table 45.4 shows the scenario of a multivariate independent variable and a multivariate dependent variable. This is the data-analytic scenario that we will consider in more detail in the context of canonical correlation analysis and that can generally be represented and treated data-analytically by the multivariate general linear model.
| \(x_1\) | \(\cdots\) | \(x_{m_x}\) | \(y_1\) | \(\cdots\) | \(y_{m_y}\) |
|---|---|---|---|---|---|
| \(x_{11}\) | \(\cdots\) | \(x_{1m_x}\) | \(y_{11}\) | \(\cdots\) | \(y_{1m_y}\) |
| \(\vdots\) | \(\ddots\) | \(\vdots\) | \(\vdots\) | \(\ddots\) | \(\vdots\) |
| \(x_{n1}\) | \(\cdots\) | \(x_{nm_x}\) | \(y_{n1}\) | \(\cdots\) | \(y_{nm_y}\) |
45.2 Descriptive statistics
We now want to discuss some standard descriptive statistics for describing multivariate data sets. To this end, we first generalize the concepts of sample mean and sample variance known from univariate descriptive statistics, and then consider Mahalanobis distances as multivariate measures of signal-to-noise relations. As always, these concepts are developed against the background of the assumption that observed data are realizations of corresponding random vectors. In contrast to the organization of data considered in Section 45.1 and familiar from empirical contexts, in which experimental units are organized in rows and their respective dependent-variable components in columns, organizing the variable components belonging to one experimental unit in column form is more useful here and more consistent with the notation of the univariate case.
Sample mean, sample covariance matrix, and sample correlation matrix
Definition 45.1 (Sample mean, sample covariance matrix, and sample correlation matrix) Let \(y_1,...,y_n\) be a set of \(m\)-dimensional random vectors, called a sample.
Without proof, we note that, analogously to the univariate case, for independent and identically distributed random vectors \(y_1,...,y_n\), the sample mean is an unbiased estimator of the sample-variable expectation \(\mathbb{E}(y_i) \in \mathbb{R}^m, i = 1,...,n\). Likewise, in this case the sample covariance matrix is an unbiased estimator of the sample-variable covariance matrix \(\mathbb{C}(y_i) \in \mathbb{R}^{m \times m}, i = 1,...,n\). For the concrete computation of the sample mean, sample covariance matrix, and sample correlation matrix based on a data set, the statements of the following theorem are useful.
Theorem 45.1 (Data matrix and sample statistics)
Let \[\begin{equation} y := \begin{pmatrix} y_1 & \cdots & y_n \end{pmatrix} \end{equation}\] be an \(m \times n\) given by the column-wise concatenation of \(n\) \(m\)-dimensional random vectors \(y_1, ...,y_n\). ThenProof. The representation of the sample mean follows from \[\begin{align} \begin{split} \bar{y} & := \frac{1}{n} \sum_{i=1}^ny_i \\ & = \frac{1}{n}\begin{pmatrix} \sum_{i=1}^ny_{i1} \\ \vdots \\ \sum_{i=1}^ny_{im} \end{pmatrix} \\ & = \frac{1}{n}\left(\begin{pmatrix}y_{11} & \cdots &y_{n1} \\ \vdots & \ddots & \vdots \\ y_{1m} & \cdots &y_{nm} \\ \end{pmatrix} \begin{pmatrix} 1 \\ \vdots \\ 1 \end{pmatrix} \right) \\ & = \frac{1}{n}y 1_{n}. \end{split} \end{align}\] With respect to the representation of the sample covariance matrix, we first note that, by definition, \[\begin{align} \begin{split} C & := \frac{1}{n-1}\sum_{i=1}^n (y_i - \bar{y})(y_i - \bar{y})^T \\ & = \frac{1}{n-1}\sum_{i=1}^n \left(y_iy_i^T-y_i\bar{y}^T - \bar{y}y_i^T+ \bar{y}\bar{y}^T\right) \\ & = \frac{1}{n-1}\left(\sum_{i=1}^ny_iy_i^T- \sum_{i=1}^ny_i\bar{y}^T - \sum_{i=1}^n \bar{y}y_i^T+ \sum_{i=1}^n \bar{y}\bar{y}^T\right) \\ & = \frac{1}{n-1}\left(\sum_{i=1}^ny_iy_i^T- n\bar{y}\bar{y}^T - n\bar{y}\bar{y}^T + n\bar{y}\bar{y}^T\right) \\ & = \frac{1}{n-1}\left(\sum_{i=1}^ny_iy_i^T- n\bar{y}\bar{y}^T\right). \end{split} \end{align}\] With \(1_{n}1_{n}^T = 1_{nn}\), it further follows that \[\begin{align} \begin{split} y\left(I_n - \frac{1}{n}1_{nn}\right)y^T & = \left(y I_n - \frac{1}{n}y 1_{nn}\right)y^T \\ & = yy^T - \frac{1}{n}y 1_{nn}y^T \\ & = \begin{pmatrix} y_1 & \cdots & y_n\end{pmatrix} \begin{pmatrix} y_1^T \\ \vdots \\ y_n^T\end{pmatrix} - \frac{1}{n}y 1_n 1_n^Ty^T \\ & = \sum_{i=1}^ny_iy_i^T- n\left(\frac{1}{n}y 1_n\right)\left(\frac{1}{n}1_n^Ty^T\right) \\ & = \sum_{i=1}^ny_iy_i^T- n\bar{y}\bar{y}^T \\ & = \sum_{i=1}^n (y_i - \bar{y})(y_i - \bar{y})^T \\ & = (n-1)C. \end{split} \end{align}\] With respect to the correlation matrix, for an arbitrary index pair \(i,j\) with \(1 \le i,j \le m\), definition finally gives \[\begin{align} \begin{split} R_{{y}_{ij}} & = \frac{(C)_{ij}}{\sqrt{ (C)_{ii}}\sqrt{ (C)_{jj}}} \\ & = \frac{1}{\sqrt{(C)_{ii}}}(C)_{ij}\frac{1}{\sqrt{(C)_{jj}}} \\ & = (DCD)_{ij}. \end{split} \end{align}\]
The following R code applies the results discussed in Theorem 45.1 to an example data set with data dimensionality \(m = 4\) and number of experimental units \(n := 12\).
[,1] [,2] [,3] [,4] [,5] [,6] [,7]
[1,] 0.380757 3.206102 0.7449730 -0.4244947 0.8556004 1.207538 3.3079784
[2,] 1.681932 1.070638 0.5125397 0.9248077 3.0000288 1.378733 0.6155732
[3,] 1.085641 4.176583 1.3350276 2.5364696 1.8840799 2.249181 5.0871665
[4,] 3.318340 3.675730 4.0601604 3.4111055 4.5314962 2.481606 4.3065579
[,8] [,9] [,10] [,11] [,12]
[1,] 1.105802 1.456999 0.9228471 0.6659992 0.965274
[2,] 3.869291 2.425100 1.7613529 3.0584830 2.886423
[3,] 3.017396 1.713699 1.3593945 3.4501871 2.981440
[4,] 2.463550 3.699024 3.4717201 3.3479052 3.943103# evaluation of descriptive statistics
n = ncol(Y) # number of data-vector realizations
I_n = diag(n) # identity matrix I_n
J_n = matrix(rep(1,n^2), nrow = n) # 1_{nn}
y_bar = (1/n)* Y %*% J_n[,1] # sample mean
C = (1/(n-1))*(Y %*% (I_n-(1/n)*J_n) %*% t(Y)) # sample covariance matrix
D = diag(1/sqrt(diag(C))) # cov-corr transformation matrix
R = D %*% C %*% D # sample correlation matrix [,1]
[1,] 1.199615
[2,] 1.932075
[3,] 2.573022
[4,] 3.559191 [,1] [,2] [,3] [,4]
[1,] 1.1450788 -0.29291622 0.91837375 0.16929650
[2,] -0.2929162 1.20170927 -0.09630563 -0.16923746
[3,] 0.9183737 -0.09630563 1.50573951 0.08718263
[4,] 0.1692965 -0.16923746 0.08718263 0.40266165 [,1] [,2] [,3] [,4]
[1,] 1.0000000 -0.24970431 0.69940198 0.2493218
[2,] -0.2497043 1.00000000 -0.07159407 -0.2432913
[3,] 0.6994020 -0.07159407 1.00000000 0.1119657
[4,] 0.2493218 -0.24329134 0.11196568 1.0000000Mahalanobis distances
Finally, with the so-called Mahalanobis distances, we want to consider multivariate generalizations of signal-to-noise measures known from univariate applications. We define the concept of Mahalanobis distance as follows.
Definition 45.2 (Mahalanobis distances) Let \(x_1\) be a random vector, a realization of a random vector, a multivariate expectation, or a multivariate sample mean; let \(x_2\) be a random vector, a realization of a random vector, a multivariate expectation, or a multivariate sample mean; and let \(X\) be a covariance matrix or a sample covariance matrix. Then \[\begin{equation} D = \left(x_1 - x_2 \right)^TX^{-1}\left(x_1 - x_2\right) \end{equation}\] is called the of \(x_1\) and \(x_2\) with respect to \(X\).
A Mahalanobis distance is thus a squared Euclidean distance normalized by a covariance matrix (cf. Section 8.2). Similar measures of the ratio of a distance to variability are, as is well known, the \(z\)-transformation \(z = (y - \mu)/\sigma\) for \(y \in \mathbb{R}\) and the parameters \(\mu \in \mathbb{R}, \sigma^2>0\) of a univariate normal distribution, as well as Cohen’s \(d = (\bar{y}_1-\bar{y}_2)/s_{12}\) for two sample means \(\bar{y}_1\) and \(\bar{y}_2\) and their corresponding pooled sample standard deviation \(s_{12}\). In contrast to Mahalanobis distance, however, these measures are not squared and therefore have a sign. In analogy to the \(z\)-transformation or Cohen’s \(d\), however, Mahalanobis distances also measure a distance in units of variability. For Cohen’s \(d\), a value of \(d = 1\) means precisely that the distance between \(\bar{y}_1\) and \(\bar{y}_2\) is one pooled standard deviation. The situation is analogous for Mahalanobis distances.
Using Figure 45.1 and Figure 45.2, we want to consider the influence of the variance and covariance of components of \(x_1\) and \(x_2\) on their Mahalanobis distance in somewhat more detail. The titles of the subfigures in Figure 45.1 show the Mahalanobis distances of the vectors \(x_1 := (-1,-1)^T\) and \(x_2 := (1,1)^T\) for covariance matrices \[\begin{equation} \Sigma_1 := \begin{pmatrix} 1.0 & 0.0 \\ 0.0 & 1.0\end{pmatrix}, \Sigma_2 := \begin{pmatrix} 0.5 & 0.0 \\ 0.0 & 0.5\end{pmatrix} \mbox{ and } \Sigma_3 := \begin{pmatrix} 1.5 & 0.0 \\ 0.0 & 1.5\end{pmatrix}, \end{equation}\] which are represented with normal-distribution isocontours. For \(\Sigma_1\), the Mahalanobis distance corresponds to the squared Euclidean distance between \(x_1\) and \(x_2\). The representation for \(\Sigma_2\) shows that, in the case of spherical covariance matrices, a lower component variance of \(x_1\) and \(x_2\) leads to a larger Mahalanobis distance. Conversely, the representation for \(\Sigma_3\) shows that, in the case of spherical covariance matrices, a higher component variance of \(x_1\) and \(x_2\) results in a smaller Mahalanobis distance. Intuitively, the component variance therefore brings the components closer together.
The titles of the subfigures in Figure 45.2 show the Mahalanobis distances of the same vectors for covariance matrices \[\begin{equation} \Sigma_1 := \begin{pmatrix} 1.0 & 0.0 \\ 0.0 & 1.0\end{pmatrix}, \Sigma_2 := \begin{pmatrix} 1.0 & 0.9 \\ 0.9 & 1.0\end{pmatrix}\mbox{ and } \Sigma_3 := \begin{pmatrix*}[r] 1.0 & -0.9 \\ -0.9 & 1.0\end{pmatrix*}, \end{equation}\] For \(\Sigma_1\), the Mahalanobis distance again corresponds to the squared Euclidean distance between \(x_1\) and \(x_2\). The representation for \(\Sigma_2\) shows that a strongly positive covariance of the components of \(x_1\) and \(x_2\) results in a smaller Mahalanobis distance. Conversely, the representation for \(\Sigma_3\) shows that a strongly negative covariance of the components of \(x_1\) and \(x_2\) leads to a larger Mahalanobis distance. Intuitively, the Mahalanobis distance therefore also takes the covariance of the components \(x_1\) and \(x_2\) into account. Alternatively, the magnitude of a Mahalanobis distance can also be understood as a measure of the improbability of realizing two values of a random vector given a particular covariance matrix.
45.3 Bibliographic remarks
The results on the matrix representation of sample mean, sample covariance matrix, and sample correlation matrix follow Rencher & Christensen (2012). The concept of Mahalanobis distance goes back to Mahalanobis (1936).