| Person | BDI-II T0 | BDI-II T1 | PHQ-9 T0 | PHQ-9 T1 | HDRS T0 | HDRS T1 | CES-D T0 | CES-D T1 | MADRS T0 | MADRS T1 |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | 53.2 | 56.8 | 46.9 | 51.8 | 61.4 | 58.6 | 46.3 | 51.3 | 55.6 | 52.6 |
| 2 | 58.2 | 55.4 | 58.3 | 59.0 | 68.6 | 43.5 | 53.8 | 63.8 | 64.3 | 57.6 |
| 3 | 54.6 | 58.8 | 51.3 | 64.2 | 51.1 | 50.3 | 61.0 | 52.8 | 58.7 | 55.7 |
| 4 | 43.0 | 45.6 | 43.9 | 55.6 | 47.1 | 50.6 | 40.2 | 49.0 | 53.0 | 41.5 |
| 5 | 54.0 | 53.0 | 58.5 | 51.8 | 47.5 | 45.9 | 47.2 | 50.0 | 48.2 | 59.7 |
| 6 | 59.3 | 57.5 | 50.8 | 56.9 | 69.8 | 55.0 | 62.9 | 63.8 | 57.0 | 60.5 |
| 7 | 37.8 | 20.7 | 24.7 | 33.2 | 22.3 | 20.2 | 23.1 | 26.3 | 27.9 | 30.3 |
| 8 | 33.8 | 35.7 | 42.5 | 53.3 | 38.5 | 36.6 | 40.0 | 33.7 | 48.6 | 32.4 |
| 9 | 51.3 | 48.2 | 48.8 | 50.0 | 43.5 | 53.2 | 43.6 | 54.4 | 48.5 | 51.9 |
| 10 | 60.1 | 58.2 | 61.6 | 61.4 | 65.5 | 61.2 | 60.8 | 60.6 | 68.8 | 59.1 |
| 11 | 44.4 | 47.9 | 43.8 | 43.6 | 43.2 | 44.2 | 41.9 | 52.8 | 42.9 | 42.0 |
| 12 | 62.1 | 55.3 | 58.0 | 57.0 | 62.5 | 49.8 | 54.1 | 48.5 | 53.5 | 50.0 |
| 13 | 52.3 | 48.3 | 51.7 | 48.1 | 57.7 | 63.0 | 58.4 | 60.8 | 61.1 | 51.9 |
| 14 | 53.9 | 54.4 | 50.9 | 58.0 | 54.2 | 55.6 | 44.5 | 58.6 | 51.8 | 49.5 |
| 15 | 37.5 | 41.0 | 40.6 | 38.3 | 42.1 | 34.0 | 35.7 | 43.8 | 36.9 | 43.1 |
| 16 | 62.4 | 51.8 | 58.7 | 41.6 | 52.0 | 47.1 | 51.2 | 54.6 | 50.8 | 63.6 |
| 17 | 61.8 | 65.5 | 57.0 | 60.6 | 72.9 | 70.0 | 56.6 | 64.3 | 59.3 | 64.6 |
| 18 | 45.1 | 39.9 | 44.5 | 42.4 | 36.8 | 36.9 | 43.4 | 44.1 | 39.1 | 39.6 |
| 19 | 53.7 | 55.5 | 49.4 | 47.6 | 47.5 | 42.5 | 49.4 | 51.7 | 49.2 | 56.8 |
| 20 | 44.6 | 39.9 | 34.7 | 41.5 | 42.1 | 45.6 | 32.0 | 42.8 | 46.5 | 47.2 |
50 Classical test theory
50.1 Multiple test measurement models
Classical test theory takes its starting point from models for observed scores from several test measurements from several people. In order to have a concrete data example in mind for the following model formulation, based on which parameters of the model of classical test theory can then be estimated, we assume that we have a data set as shown in the table below. This data set is intended to represent the T scores of \(n = 20\) people for whom \(m = 10\) tests were carried out to measure depression symptoms. Specifically, we imagine that the T score of the BDI-II, the PHQ-9, the HDRS, the CES-D, and the MADRS was determined for each person at two time points (T0 and T1). In the following we will denote people with the index \(i\), so here we have \(i = 1,...,20\) and test measurements with the index \(j\), so we have here \(j = 1,...,10\).
50.1.1 The multiple test measurement model
The classical test theory in the formalization according to Novick (1966) and Lord & Novick (1968) takes its starting point from the definition of the true score of a test measurement of a person, which implies the definitions of the observed score and the measurement error. The definition uses the concept of a conditional expectation and takes the following form.
Definition 50.1 (Classic definition of true score, observed score and measurement error) For \(i = 1,...,n\) and \(j = 1,...,m\), a person’s true score \(t_{ij}\) \(i\) for a test measurement \(j\) is defined as the conditional expected value of the person’s observed score for that test measurement \[\begin{equation} t_{ij} := \mathbb{E}(y_{ij}|\tau_{ij} = t_{ij}). \end{equation}\] A person’s error score for this test measurement is defined as the random variable \[\begin{equation} \varepsilon_{ij} := y_{ij} - t_{ij}. \end{equation}\]
The term test measurement is somewhat unspecific here and, depending on the application, means either a measurement using a single item or using the sum of several items. At a later point we will make the distinction between item scores and test sum scores explicit. The latter induce the concept of \(m\) component test models (see Section 50.3) within the framework of classical test theory. Note that in Definition 50.1, according to the definition of conditional expectations, \(\tau_{ij}, y_{ij}\) and \(\varepsilon_{ij}\) are random variables and \(t_{ij} \in \mathbb{R}\) is a constant.
The definition of the true score \(t_{ij}\) in Definition 50.1 is somewhat special (not to say circular to tautological) because \(t_{ij}\) is defined using \(t_{ij}\). The central motivation of Novick (1966) and Lord & Novick (1968) to define the true score \(t_{ij}\) as a conditional expected value, instead of, for example, simply a realization of the random variable \(\tau_{ij}\), was to avoid a discussion about the metaphysical meaning of a true score. For example, if Novick (1966) and Lord & Novick (1968) had defined the true score as the realization of a latent variable that a person is said to possess for a particular test measurement, this would clearly have the character of an attackable metaphysical statement in contemporary discussion. Instead, Novick (1966) and Lord & Novick (1968) attempt to be as operationalistic as possible in their definition and represent a person’s true score for a test measurement as the “average of the person’s observed scores” over “repeated test measurements under identical conditions”. For the meaning of the true score, cite Lord & Novick (1968), pp. 29 - 30 Lazarsfeld (1959):
“Suppose we repeatedly ask a person, Mr. Brown, whether he supports the United Nations; suppose further that after each question we”wash his brain” and then ask him the same question again. Because Mr. Brown is unsure of his position on the United Nations, he will sometimes give a favorable answer and sometimes a negative answer. After carrying out this procedure many times, we then calculate the proportion of times Mr. Brown endorsed the United Nations.”
Novick (1966) and Lord & Novick (1968) then want to understand this proportion as the true score (see also Borsboom et al. (2004)). Of course, this attempt at an anti-metaphysical attitude is not sustainable: on the one hand, the “repeated test measurement under identical conditions” is an idealized thought experiment that cannot be carried out in reality. On the other hand, Novick (1966) and Lord & Novick (1968) do not define the true score as the (finite) mean of a series of measurements, but rather as the idealized expected value of a random variable. As a truly operationalistic definition, Definition 50.1 is not convincing, but it considerably complicates the development of a standard measurement error theory, such as the classic formalization of the General Linear Model, for the analysis of tests and questionnaires. This may be one of the reasons why classical test theory is still only important in psychology and little beyond.
Similar is the designation of the conditional distribution of the observed score \(\mathbb{P}(y_{ij}|\tau_{ij} = t_{ij})\) as a propensity distribution by Lord & Novick (1968), which models the intra-individual variability of the observed score with a fixed true score. This suggests that Lord & Novick (1968) implies a propensity interpretation of probabilities as causally determined “realization tendencies” which, in contrast to the frequentist interpretation, also make sense in individual cases. However, propensity distributions assume causal processes that are usually not specified and therefore not observable, and thus ultimately lead to metaphysical statements (see also Borsboom et al. (2004) and Borsboom (2009)).
In our presentation we want to follow the model-based realistic approach and with Definition 50.2 we therefore choose a formulation of the model of multiple test measurements of classical test theory, which is congruent with Definition 50.1 and thus of course also the theoretical results of classical test theory, but does not attempt to disguise its model character. In particular, our approach emphasizes the overall goal of modeling a set of \(m\) test measurements from \(n\) people as a data set of \(nm\) data points. We therefore define the multiple test measurement model as follows.
Definition 50.2 (Multiple test measurement model) For \(i = 1,...,n\) and \(j = 1,...,m\), let \(\tau_{ij}\) be a random variable that models the true score of the \(i\) th person in the \(j\) th test measurement, and \(y_{ij}\) a random variable that models the observed score of the \(i\) th person in the \(j\) th test measurement. Then we call the joint distribution the \(\tau_{ij}\) and \(y_{ij}\) with the factorization property \[\begin{equation} \mathbb{P}\left(\tau_{11},y_{11},...,\tau_{nm},y_{nm}\right) := \prod_{i=1}^n \mathbb{P}(\tau_{i1},...,\tau_{im})\prod_{j=1}^m \mathbb{P}(y_{ij}|\tau_{ij}) \end{equation}\] the multiple test measurement model if it holds that \[\begin{equation} \mathbb{P}(\tau_{11},...,\tau_{1m})\prod_{j=1}^m \mathbb{P}(y_{1j}|\tau_{1j}) = \cdots = \mathbb{P}(\tau_{n1},...,\tau_{nm})\prod_{j=1}^m \mathbb{P}(y_{nj}|\tau_{nj}). \end{equation}\]
The definition of the multiple test measurement model represents some fundamental assumptions about the independence and identity of distributions in classical test theory. First of all, it is assumed that the joint distribution of \(\tau_{ij}\) and \(y_{ij}\) is factored over \(i = 1,...,n\), so that the distributions of the true scores and observed scores are independent across individuals. So knowledge of one person’s true or observed scores does not change the assumed distributions of other people’s true and observed scores. In contrast, for each \(i = 1,...,n\), the joint distribution of the \(\tau_{i1},...,\tau_{im}\) over test measurements does not necessarily factor \(j = 1,...,m\). The true scores of a person can therefore be dependent and knowledge about the characteristics of a test measurement in one person can inform the distribution of the true score in other test measurements of the same person. In classical test theory, different types of this form of dependencies are distinguished and, as we will see later, are referred to, for example, as parallelism, * \(\tau\) -equivalence* or essential \(\tau\) -equivalence. Furthermore, for each person \(i = 1,...,n\) it is assumed that the observed scores \(y_{ij}\) for \(j = 1,...,m\) given \(\tau_{ij}\) are conditionally independent. This implies, on the one hand, that for a person, the true score in test measurement \(k \neq j\) does not influence the observed score in test measurement \(j\) and, on the other hand, that the observed score in test measurement \(k \neq j\) does not influence the observed score in test measurement \(j\).
Finally, it is assumed that the marginal distributions \[\begin{equation} \mathbb{P}(\tau_{i1},y_{i1},...,\tau_{im},y_{im}) = \mathbb{P}(\tau_{i1},...,\tau_{im})\prod_{j=1}^m \mathbb{P}(y_{ij}|\tau_{ij}) \end{equation}\] about people \(i = 1,...,n\) are identical. One might think of the realizations of true scores and observed scores as independent and identical realizations from a “population distribution” \[\begin{equation} \mathbb{P}(\tau_{\bullet 1},y_{\bullet 1},...,\tau_{\bullet m},y_{\bullet m}) = \mathbb{P}(\tau_{\bullet 1},...,\tau_{\bullet m})\prod_{j=1}^m \mathbb{P}(y_{\bullet j}|\tau_{\bullet j}) \end{equation}\] imagine, whereby the subscript \(\bullet\) is intended to symbolize the non-specificity of this distribution with regard to a person.
If one considers the special case of a single test measurement in \(n\) people, a simplified form of Definition 50.2 results, which is often encountered in the psychological literature. According to Definition 50.2, for a test measurement, i.e. \(m = 1\), \[\begin{equation} \mathbb{P}\left(\tau_{11},y_{11},...,\tau_{n1},y_{n1}\right) := \prod_{i=1}^n \mathbb{P}(\tau_{i1})\mathbb{P}(y_{i1}|\tau_{i1}), \end{equation}\] where, after assuming Definition 50.2, the joint marginal distributions \(\mathbb{P}(\tau_{i1}, y_{i1})\) over persons \(i = 1,...,n\) are identical. As above, the true and observed scores for a test measurement can be viewed as independent realizations of the “population distribution” \[\begin{equation} \mathbb{P}(\tau_{\bullet 1})\mathbb{P}(y_{\bullet 1}|\tau_{\bullet 1}) \end{equation}\] introduce. If one now omits the subscripts \(\bullet 1\), one arrives at the following simplified definition of the model of classical test theory.
Definition 50.3 (Simplified model of classical test theory) Let \(\tau\) be a random variable that describes the distribution of the true scores of a test measurement in a population of individuals and \(y\) be a random variable that describes the distribution of the observed scores of this test measurement. Then the joint distribution of \(\tau\) and \(y\) is called \[\begin{equation} \mathbb{P}\left(\tau,y\right) = \mathbb{P}(\tau)\mathbb{P}(y|\tau) \end{equation}\] the Simplified Model of Classical Test Theory.
Definition 50.3 has the advantage over Definition 50.2 that fewer random variables and indices occur and the redundancy of different names for many identical distributions can be dispensed with. In the sense of frequentist product models, one also likes people here with regard to the data from \(n\) \[\begin{equation} (\tau_1,y_1), ...,(\tau_n,y_n) \sim \mathbb{P}(\tau,y) \end{equation}\] write, where only the \(y_1,...,y_n\) is observed, while the \(\tau_1,...,\tau_n\) are latent random variables. Furthermore, many important properties of the Classical Test Theory model can be justified based on the properties of \(\mathbb{P}\left(\tau,y \right)\), as we will see below. In general, the simplified model of classical test theory is easier to use than the model of multiple test measurements. However, it becomes problematic as soon as several test measurements come into play, for example when considering dependencies between two tests or two items of a test. However, this is the case with most statements of classical test theory. It is also true that estimators of the model parameters are always based on all values of the observed random variables \(y_{1j},...y_{nj}\), but these do not appear at all in Definition 50.3. For theoretical considerations, Definition 50.3 is often easier to handle than Definition 50.2, but the application of classical test theory in test data analysis is generally based on Definition 50.2. We will switch back and forth between both model formulations as necessary, but note that the model formulation in the spirit of Definition 50.2 is our default case.
Properties of the multiple test measurement model
Properties regarding a test measurement
The model of multiple test measurements according to Definition 50.2 initially has a number of properties regarding one and therefore every test measurement that are fundamental for the application of classical test theory. We summarize five of these properties in the following theorem.
Theorem 50.1 (Properties related to a test measurement) The model of multiple test measurements is given. Then \(i = 1,...,n\) and all \(j = 1,...,m\) apply to all
- \(\mathbb{E}(\varepsilon_{ij}|\tau_{ij} = t_{ij}) = 0\) (2) \(\mathbb{E}(\varepsilon_{ij}) = 0\) (3) \(\mathbb{C}(\tau_{ij}, \varepsilon_{ij}) = 0\) (4) \(\mathbb{V}(y_{ij}) = \mathbb{V}(\tau_{ij}) + \mathbb{V}(\varepsilon_{ij})\) (5) \(\mathbb{C}(y_{ij},\tau_{ij}) = \mathbb{V}(\tau_{ij})\)
Proof. To simplify the notation in the sense of the simple model of classical test theory according to Definition 50.3, we first set \(i = 1,...,n\) and \(j = 1,...m\) for all \[\begin{equation} y := y_{ij} \mbox{ and } \tau := \tau_{ij} \mbox{ with outcome spaces } Y := Y_{ij} \mbox{ and } T := T_{ij}. \end{equation}\] Furthermore, we denote the values of \(y\) with \(\tilde{y}\) and the values of \(\tau\) with \(t\). Finally, we only consider the discrete case, so we assume the existence of a WMF \(p: Y \times T \to [0,1]\) of the form \[\begin{equation} p(t,\tilde{y}) = p(\tilde{y}|t)p(t) \end{equation}\] in advance. The continuous case or mixed discrete-continuous case then follows analogously.
(1) It applies \[\begin{align} \begin{split} \mathbb{E}(\varepsilon|\tau = t) & := \mathbb{E}(y - \tau|\tau = t) \\ & = \sum_{\tilde{y} \in Y} \left(\tilde{y} - t\right)p(\tilde{y}|t) \\ & = \sum_{\tilde{y} \in Y} \tilde{y} p(\tilde{y}|t) - \sum_{\tilde{y} \in Y} t p(\tilde{y}|t) \\ & = \mathbb{E}(y|\tau = t) - t \sum_{\tilde{y} \in Y} p(\tilde{y}|t)\\ & = t - t\cdot 1 \\ & = 0. \end{split} \end{align}\] (2) It applies \[\begin{align} \begin{split} \mathbb{E}(\varepsilon) & := \mathbb{E}(y - \tau) \\ & = \sum_{t \in T}\sum_{\tilde{y} \in Y} \left(\tilde{y} - t\right)p(t,\tilde{y}) \\ & = \sum_{t \in T}\sum_{\tilde{y} \in Y} \left(\tilde{y} - t\right)p(\tilde{y}|t)p(t) \\ & = \sum_{t \in T}\sum_{\tilde{y} \in Y} \tilde{y} p(\tilde{y}|t)p(t) - t p(\tilde{y}|t)p(t) \\ & = \sum_{t \in T}\sum_{\tilde{y} \in Y} p(t)\left(\tilde{y} p(\tilde{y}|t) - t p(\tilde{y}|t)\right)\\ & = \sum_{t \in T} p(t)\left(\sum_{\tilde{y} \in Y} \tilde{y} p(\tilde{y}|t) - t \sum_{\tilde{y} \in Y}p(\tilde{y}|t)\right)\\ & = \sum_{t \in T} p(t) \left(t - t \cdot 1\right)\\ & = \sum_{t \in T} p(t) \cdot 0\\ & = 0. \end{split} \end{align}\] (3) It applies \[\begin{align} \begin{split} \mathbb{C}(\tau, \varepsilon) & = \mathbb{E}\left((\tau -\mathbb{E}(\tau))(\varepsilon - \mathbb{E}(\varepsilon))\right) \\ & = \mathbb{E}\left((\tau -\mathbb{E}(\tau))\varepsilon \right) \\ & = \mathbb{E}\left((\tau -\mathbb{E}(\tau))(y - \tau)\right) \\ & = \sum_{t \in T}\sum_{\tilde{y} \in Y} \left((t-\mathbb{E}(\tau))(\tilde{y}-t)\right)p(t,\tilde{y}) \\ & = \sum_{t \in T}\sum_{\tilde{y} \in Y} (t-\mathbb{E}(\tau))(\tilde{y} - t)p(\tilde{y}|t)p(t) \\ & = \sum_{t \in T}p(t)(t-\mathbb{E}(\tau))\sum_{\tilde{y} \in Y} \left(\tilde{y} - t\right)p(\tilde{y}|t) \\ & = \sum_{t \in T}p(t)(t-\mathbb{E}(\tau))\sum_{\tilde{y} \in Y} \left(\tilde{y} p(\tilde{y}|t) - t p(\tilde{y}|t) \right) \\ & = \sum_{t \in T}p(t)(t-\mathbb{E}(\tau)) \left(\sum_{\tilde{y} \in Y}\tilde{y} p(\tilde{y}|t) - t \sum_{\tilde{y} \in Y} p(\tilde{y}|t) \right) \\ & = \sum_{t \in T}p(t)(t-\mathbb{E}(\tau)) \left(t - t \cdot 1 \right) \\ & = \sum_{t \in T}p(t)(t-\mathbb{E}(\tau)) \cdot 0 \\ & = 0. \\ \end{split} \end{align}\] (4) With the theorem on variances of sums and differences of Random variables (Theorem 25.7) as well as statement (3) of the present theorem applies \[\begin{align} \begin{split} \mathbb{V}(y) = \mathbb{V}(\tau + \varepsilon) = \mathbb{V}(\tau) + \mathbb{V}(\varepsilon) + 2\mathbb{C}(\tau,\varepsilon) = \mathbb{V}(\tau) + \mathbb{V}(\varepsilon) + 2\cdot 0 = \mathbb{V}(\tau) + \mathbb{V}(\varepsilon) \end{split} \end{align}\] (5) Using the covariance shift theorem (Theorem 25.2), the linear combination property of the expected value (Theorem 23.2), statement (2) of the present theorem and the fact that with statement (4) of the present theorem it also follows that \[\begin{equation} \mathbb{E}(\varepsilon\tau) = \mathbb{E}(\tau\varepsilon) = \mathbb{C}(\tau,\varepsilon) + \mathbb{E}(\tau)\mathbb{E}(\varepsilon) = 0 + \mathbb{E}(\tau)\cdot 0 = 0, \end{equation}\] applies \[\begin{align} \begin{split} \mathbb{C}(y, \tau) & = \mathbb{E}(y\tau) - \mathbb{E}(y)\mathbb{E}(\tau) \\ & = \mathbb{E}((\tau + \varepsilon)\tau) - \mathbb{E}(\tau + \varepsilon)\mathbb{E}(\tau) \\ & = \mathbb{E}(\tau^2 + \varepsilon\tau) - \left(\mathbb{E}(\tau) + \mathbb{E}(\varepsilon)\right)\mathbb{E}(\tau) \\ & = \mathbb{E}(\tau^2) + \mathbb{E}(\varepsilon\tau) - \mathbb{E}(\tau)^2 - \mathbb{E}(\varepsilon)\mathbb{E}(\tau) \\ & = \mathbb{E}(\tau^2) + \mathbb{E}(\varepsilon\tau) - \mathbb{E}(\tau)^2 - 0\cdot \mathbb{E}(\tau) \\ & = \mathbb{E}(\tau^2) + 0 - \mathbb{E}(\tau)^2 - 0\cdot \mathbb{E}(\tau) \\ & = \mathbb{E}(\tau^2) - \mathbb{E}(\tau)^2 \\ & = \mathbb{V}(\tau). \end{split} \end{align}\]
As the subscripts in Theorem 50.1 make clear, the statements of Theorem 50.1 refer to the random variables for modeling the data of a person \(i\) and a test measurement \(j\) and apply equally to all \(i = 1,...,n\) and \(j = 1,...,m\). Statement (1) of Theorem 50.1 concerns the expected value of the measurement error conditioned on a fixed value of the true score, so in this case the true score is not a random variable and the distribution of interest is \(\mathbb{P}(\varepsilon_{ij}|\tau_{ij} = t_{ij})\). Statements (2) to (5) refer to properties of the (joint) marginal distributions of \(y_{ij}\), \(\tau_{ij}\) and \(\varepsilon_{ij}\). In the sense of the simplified model of classical test theory (Definition 50.3), the above properties are often also referred to as
- \(\mathbb{E}(\varepsilon|\tau = t) = 0\) (2) \(\mathbb{E}(\varepsilon) = 0\) (3) \(\mathbb{C}(\tau, \varepsilon) = 0\) (4) \(\mathbb{V}(y) = \mathbb{V}(\tau) +\mathbb{V}(\varepsilon)\) (5) \(\mathbb{C}(y,\tau) = \mathbb{V}(\tau)\)
written.
Specifically, according to statement (1) of Theorem 50.1 for the multiple measurements model, it follows that the conditional expected value of the measurement error \(\mathbb{E}(\varepsilon|\tau = t)\) is zero from the definition of the true score. This is in direct opposition to typical assumptions about measurement error, which typically define measurement error with a (conditional) expected value of zero, since they conceptualize nonzero contributions to a data point as part of the theory they represent. Because in the model of multiple test measurements the conditional expected value of the measurement error for every true score \(t\) is equal to zero, it then follows from (2) of Theorem 50.1 that the marginal expected value of the measurement error \(\mathbb{E}(\varepsilon)\) is also equal to zero.
As is well known, statement (3) of Theorem 50.1, that in the model of multiple test measurements the covariance of the true scores and the measurement error is equal to zero, says that high or low true scores are not systematically associated with high or low measurement errors. The resulting statements (4) and (5) of Theorem 50.1, that the variance of the observed scores can be broken down additively into variance contributions of the true score and the measurement error and that the covariance of the observed scores and the true scores of a test measurement corresponds to the variance of the true scores, will be essential for the properties of the reliability of test measurements at a later point.
Example
In the following we want to illustrate Theorem 50.1 using a concrete first example according to Lord & Novick (1968), Exercise 2.17 for a model of multiple test measurements, whereby we naturally focus on a single test measurement.
Theorem 50.2 (Normal distribution example) Let \(i = 1,...,n\) and \(m:= 1\) be included \[\begin{equation} \mathbb{P}(\tau_{i1}) := N(\mu,1) \mbox{ and } \mathbb{P}(y_{i1}|\tau_{i1}) := N(\tau_{i1},1) \end{equation}\] Then apply
- \(\mathbb{P}(y_{i1}) = N(\mu,2)\) (2) \(\mathbb{P}(\varepsilon_{i1}|\tau_{i1}) = N(0,1)\) (3) \(\mathbb{P}(\varepsilon_{i1}) = N(0,1)\) (4) \(\mathbb{C}(\tau_{i1},\varepsilon_{i1}) = 0\)
Proof. To simplify the notation, we put $ := {i1}, y:= y{i1}, := _{i1}$.
(1) We look at the through \[\begin{equation} \mathbb{P}(\tau) = N(\mu,1) \mbox{ and } \mathbb{P}(y|\tau) = N(\tau,1) \end{equation}\] induced joint distribution of \(\tau\) and \(y\), where apparently \[\begin{equation} \mathbb{P}(y|\tau) = N(a\cdot \mu + b,1) \mbox{ with } a:= 1 \mbox{ and } b := 0 \end{equation}\] applies. From the theorem on common normal distributions (see Theorem 29.5) it follows that \[\begin{equation} \begin{pmatrix} \tau \\ y \end{pmatrix} \sim N \left ( \begin{pmatrix} \mu \\ 1 \cdot \mu + 0 \end{pmatrix}, \begin{pmatrix} 1 & 1 \cdot 1 \\ 1 \cdot 1 & 1 + 1 \cdot 1 \cdot 1 \end{pmatrix} \right) = N \left ( \begin{pmatrix} \mu \\ \mu \end{pmatrix}, \begin{pmatrix} 1 & 1 \\ 1 & 2 \end{pmatrix} \right) \end{equation}\] From the theorem on marginal normal distributions (cf. Theorem 29.4) we get \(y \sim N(\mu,2)\).
(2) We consider \(\mathbb{P}(\varepsilon|\tau = t)\) for any value \(t \in \mathbb{R}\). Then applies \[\begin{equation} \varepsilon := y - t \mbox{ with } y \sim N(t,1) \end{equation}\] With the theorem for linear-affine transformation of normally distributed random variables (cf. Theorem 29.8) then applies \[\begin{equation} \varepsilon \sim N\left(t - t, 1^2 \cdot 1\right) = N(0,1) \end{equation}\] The fact that this holds for all possible values of \(\tau\) is just statement (2).
(3) and (4) We consider the through \[\begin{equation} \mathbb{P}(\tau) = N(\mu,1) \mbox{ and } \mathbb{P}(\varepsilon|\tau) = N(0,1) \end{equation}\] induced joint distribution of \(\tau\) and \(\varepsilon\), where apparently \[\begin{equation} \mathbb{P}(\varepsilon|\tau) = N(a\cdot \mu + b,1) \mbox{ with } a:= 0 \mbox{ and } b := 0 \end{equation}\] applies. From the theorem on common normal distributions (see Theorem 29.5) it follows that \[\begin{equation} \begin{pmatrix} \tau \\ \varepsilon \end{pmatrix} \sim N \left ( \begin{pmatrix} \mu \\ 0 \cdot \mu + 0 \end{pmatrix}, \begin{pmatrix} 1 & 1 \cdot 0 \\ 0\cdot 1 & 1 + 0 \cdot 1 \cdot 0 \end{pmatrix} \right) = N \left ( \begin{pmatrix} \mu \\ 0 \end{pmatrix}, \begin{pmatrix} 1 & 0 \\ 0 & 1 \end{pmatrix} \right) \end{equation}\] The theorem on marginal normal distributions (see Theorem 29.4) then results from reading \[\begin{equation} \varepsilon \sim N(0,1) \mbox{ and } \mathbb{C}(\tau,\varepsilon) = 0. \end{equation}\]
The example shows in particular how the specifics of the multiple test measurement model of classical test theory can be created equivalently by defining a contemporary probabilistic model. Here, the definition of the marginal distribution of the true score and the definition of the conditional distribution of the observed score first induce the joint normal distribution of \(\tau_{i1}\) and \(y_{i1}\). Defining the measurement error as the difference between the observed score and the conditional expectation of the true score then results in the conditional distribution of the measurement error. These and the definitions of the example then induce a joint normal distribution of \(\tau_{i1}\) and \(\varepsilon_{i1}\). In this example, the further properties in the sense of Theorem 50.1 result from the properties of common normal distributions.
With the help of the following simulation, we want to further clarify the example discussed here. We set \(\mu:= 1\) for this and generate \(10^4\) realizations of the marginal distribution of \(\tau_{i1}\) and the conditional distribution of \(y_{i1}\).
n = 1e4 # number of persons
m = 1 # number of test measurements
mu = 1 # expected value parameters of true scores
T = matrix(rep(NaN, n*m), nrow = n) # true-score array
Y = matrix(rep(NaN, n*m), nrow = n) # observed-score array
E = matrix(rep(NaN, n*m), nrow = n) # measurement-error array
for(i in 1:n){ # iteration over persons
for(j in 1:m){ # iteration over test measurements
T[i,j] = rnorm(1,mu,1) # true-score realization
Y[i,j] = rnorm(1,T[i,j],1) # observed-score realization
E[i,j] = Y[i,j] - T[i,j]}} # measurement-error realization
e_hat_es = mean(E[,1]) # expected value estimation measurement error
c_hat_ts_es = cov(T[,1],E[,1]) # covariance Estimation True Value, Measurement Error
v_hat_os = var(Y[,1]) # variance Estimation Observed Value
v_hat_ts = var(T[,1]) # variance Estimation True Value
v_hat_es = var(E[,1]) # variance estimation measurement error
c_hat_os_ts = cov(Y[,1],E[,1]) # covariance Estimation Observed Value, True ValueFigure 50.1 A and B show the resulting marginal distributions of \(y_{i1}\) and \(\varepsilon_{i1}\) with their theoretical equivalents according to Theorem 50.2. Figure 50.1 C shows the joint distribution of \(\tau_{i1}\) and \(\varepsilon_{i1}\) with their theoretical correspondence. Finally, Figure 50.1 D shows the simulation-based estimates of relevant expected values, variances and covariances in this model, which confirm the theoretical insights according to Theorem 50.2.
Properties regarding two test measurements
So far we have learned about five properties of the multiple test measurements model, which apply to the true score \(\tau_{ij}\), the observed score \(y_{ij}\) and the measurement error \(\varepsilon_{ij}\) of one (and therefore every) person \(i\) and one (and therefore every) test measurement \(j\). In the following we deal with properties of the model of multiple test measurements that apply to the true scores \(\tau_{ij}\) and \(\tau_{ik}\), observed scores \(y_{ij}\) and \(y_{ik}\) and measurement errors \(\varepsilon_{ij}\) and \(\varepsilon_{ik}\) of one (and therefore every) person \(i\) with respect to two test measurements \(j\) and \(k\). We summarize these properties, sometimes referred to as the local uncorrelation of the multiple test measurements model (see Krauth (1995)), in the following theorem.
Theorem 50.3 (Properties regarding two test measurements) The model of multiple test measurements is given. Then this applies to all \(i = 1,...,n\) and all \(j\) and \(k\) with \(1 \le j,k \le m\) and \(j \neq k\)
- \(\mathbb{C}(y_{ij},y_{ik}|\tau_{ij} = t_{ij},\tau_{ik} = t_{ik}) = 0\) (2) \(\mathbb{C}(\varepsilon_{ij},\varepsilon_{ik}|\tau_{ij} = t_{ij},\tau_{ik} = t_{ik}) = 0\) (3) \(\mathbb{C}(\varepsilon_{ij},\varepsilon_{ik}) = 0\) (4) \(\mathbb{C}(\tau_{ij},\varepsilon_{ik}) = 0\) (5) \(\mathbb{C}(y_{ij},y_{ik}) = \mathbb{C}(\tau_{ij},\tau_{ik})\)
Proof. To simplify the notation, we omit the \(i\) subscript in the proofs. We continue to consider only the discrete case and assume the existence of the marginal WMF \[\begin{equation} p(t_j,\tilde{y}_j,t_k,\tilde{y}_k) = p(t_j,t_k)p(\tilde{y}_j|t_j)p(\tilde{y}_k|t_k) \end{equation}\] and consequently also the conditional probability mass function \[\begin{equation} p(\tilde{y}_j, \tilde{y}_k|t_j,t_k) = \frac{p(t_j,\tilde{y}_j,t_k,\tilde{y}_k)}{p(t_j,t_k)} = \frac{p(t_j,t_k)p(\tilde{y}_j|t_j)p(\tilde{y}_k|t_k)}{p(t_j,t_k)} = p(\tilde{y}_j|t_j)p(\tilde{y}_k|t_k) \end{equation}\] in advance. The continuous case then follows again analogously.
(1) It applies \[\begin{align} \begin{split} \mathbb{C}(y_{j},y_{k}|\tau_{j} = t_{j},\tau_{k} = t_{k}) & = \sum_{\tilde{y}_j \in Y_j} \sum_{\tilde{y}_k \in Y_k} (\tilde{y}_j - \mathbb{E}(y_j|\tau_j = t_j)) (\tilde{y}_k - \mathbb{E}(y_k|\tau_k = t_k)) p(\tilde{y}_j|t_j)p(\tilde{y}_k|t_k) \\ & = \sum_{\tilde{y}_j \in Y_j} (\tilde{y}_j - t_j) p(\tilde{y}_j|t_j) \sum_{\tilde{y}_k \in Y_k} (\tilde{y}_k - t_k) p(\tilde{y}_k|t_k) \\ & = \sum_{\tilde{y}_j \in Y_j} (\tilde{y}_j - t_j) p(\tilde{y}_j|t_j) \left( \sum_{\tilde{y}_k \in Y_k} \tilde{y}_k p(\tilde{y}_k|t_k) - t_k \sum_{\tilde{y}_k \in Y_k}p(\tilde{y}_k|t_k) \right) \\ & = \sum_{\tilde{y}_j \in Y_j} (\tilde{y}_j - t_j) p(\tilde{y}_j|t_j) \left( t_k - t_k \cdot 1 \right) \\ & = \sum_{\tilde{y}_j \in Y_j} (\tilde{y}_j - t_j) p(\tilde{y}_j|t_j) \cdot 0 \\ & = 0. \end{split} \end{align}\] (2) We first determine \(\mathbb{E}(\varepsilon_j\varepsilon_k|\tau_j = t_j, \tau_k = t_k)\). It applies \[\begin{align} \begin{split} \mathbb{E}(\varepsilon_j\varepsilon_k|\tau_j = t_j, \tau_k = t_k) & = \mathbb{E}((y_j - \tau_j)(y_k - \tau_k)|\tau_j = t_j, \tau_k = t_k) \\ & = \sum_{\tilde{y}_j \in Y_j} \sum_{\tilde{y}_k \in Y_k} (\tilde{y}_j - t_j)(\tilde{y}_k - t_k) p(\tilde{y}_j|t_j)p(\tilde{y}_k|t_k) \\ & = \sum_{\tilde{y}_j \in Y_j} (\tilde{y}_j - t_j) p(\tilde{y}_j|t_j) \sum_{\tilde{y}_k \in Y_k} (\tilde{y}_k - t_k) p(\tilde{y}_k|t_k) \\ & = \sum_{\tilde{y}_j \in Y_j} (\tilde{y}_j - t_j) p(\tilde{y}_j|t_j) \left( \sum_{\tilde{y}_k \in Y_k} \tilde{y}_k p(\tilde{y}_k|t_k) - t_k \sum_{\tilde{y}_k \in Y_k} p(\tilde{y}_k|t_k) \right) \\ & = \sum_{\tilde{y}_j \in Y_j} (\tilde{y}_j - t_j) p(\tilde{y}_j|t_j) \left( t_k - t_k \cdot 1 \right) \\ & = \sum_{\tilde{y}_j \in Y_j} (\tilde{y}_j - t_j) p(\tilde{y}_j|t_j) \cdot 0 \\ & = 0. \end{split} \end{align}\] With the shift theorem of conditional covariance (Theorem 25.9) and statement (1) of the theorem on the properties regarding a test measurement (Theorem 50.1), it follows \[\begin{equation} \mathbb{C}(\varepsilon_j,\varepsilon_k|\tau_j = t_j, \tau_k = t_k) = \mathbb{E}(\varepsilon_j\varepsilon_k|\tau_j = t_j, \tau_k = t_k) - \mathbb{E}(\varepsilon_j|\tau_j = t_j)\mathbb{E}(\varepsilon_k|\tau_k = t_k) = 0 - 0 \cdot 0 = 0. \end{equation}\] (3) We first determine \(\mathbb{E}(\varepsilon_j\varepsilon_k)\). With the proof of statement (2) of the present theorem results \[\begin{align} \begin{split} \mathbb{E}(\varepsilon_{j}\varepsilon_{k}) & = \mathbb{E}((y_j - \tau_j)(y_k - \tau_k)) \\ & = \sum_{t_j \in T_j} \sum_{t_k \in T_k} \sum_{\tilde{y}_j \in Y_j} \sum_{\tilde{y}_k \in Y_k} (\tilde{y}_j - t_j)(\tilde{y}_k - t_k) p(\tilde{y}_j|t_j)p(\tilde{y}_k|t_k) p(t_j,t_k) \\ & = \sum_{t_j \in T_j} \sum_{t_k \in T_k} \sum_{\tilde{y}_j \in Y_j} \sum_{\tilde{y}_k \in Y_k} (\tilde{y}_j - \mathbb{E}(y_j |\tau_j = t_j))(\tilde{y}_k - \mathbb{E}(y_k |\tau_k = t_k)) p(\tilde{y}_j|t_j)p(\tilde{y}_k|t_k) p(t_j,t_k) \\ & = \sum_{t_j \in T_j} \sum_{t_k \in T_k} \mathbb{E}\left((y_j - \tau_j) (y_k - \tau_k)| \tau_j = t_j, \tau_j = t_j\right) p(t_j,t_k) \\ & = \sum_{t_j \in T_j} \sum_{t_k \in T_k} \mathbb{E}\left(\varepsilon_j\varepsilon_k | \tau_j = t_j, \tau_j = t_j\right) p(t_j,t_k) \\ & = \sum_{t_j \in T_j} \sum_{t_k \in T_k} 0 \cdot p(t_j,t_k) \\ & = 0. \end{split} \end{align}\] With the covariance shift theorem (Theorem 25.2) and statement (2) of the theorem on the properties regarding a test measurement (Theorem 50.1), it then follows that \[\begin{equation} \mathbb{C}(\varepsilon_j,\varepsilon_k) = \mathbb{E}(\varepsilon_j\varepsilon_k) - \mathbb{E}(\varepsilon_j)\mathbb{E}(\varepsilon_k) = 0 - 0 \cdot 0 = 0. \end{equation}\] (4) Using the covariance shift theorem (Theorem 25.2) and statement (2) of the theorem on the properties regarding a test measurement (Theorem 50.1), we have \[\begin{align} \begin{split} \mathbb{C}(\tau_j,\varepsilon_k) & = \mathbb{E}(\tau_j\varepsilon_k) - \mathbb{E}(\tau_j)\mathbb{E}(\varepsilon_k) \\ & = \mathbb{E}(\tau_j\varepsilon_k) - \mathbb{E}(\tau_j)\cdot 0 \\ & = \mathbb{E}(\tau_j(\tilde{y}_k - \tau_k)) \\ & = \sum_{t_j \in T_j} \sum_{t_k \in T_k} \sum_{\tilde{y}_k \in Y_k} t_j(y_k - t_k) p(\tilde{y}_k|t_k) p(t_j,t_k) \\ & = \sum_{t_j \in T_j} t_j \sum_{t_k \in T_k} p(t_j, t_k) \sum_{\tilde{y}_k \in Y_k} (y_k - t_k) p(\tilde{y}_k|t_k) \\ & = \sum_{t_j \in T_j} t_j \sum_{t_k \in T_k} p(t_j,t_k) \left( \sum_{\tilde{y}_k \in Y_k}y_k p(\tilde{y}_k|t_k) - t_k \sum_{\tilde{y}_k \in Y_k}p(\tilde{y}_k|t_k) \right)\\ & = \sum_{t_j \in T_j} t_j \sum_{t_k \in T_k} p(t_j, t_k) \left(t_k - t_k \cdot 1\right)\\ & = \sum_{t_j \in T_j} t_j \sum_{t_k \in T_k} p(t_j, t_k) \cdot 0\\ & = 0. \\ \end{split} \end{align}\] (5) We first note that with statement (4) by swapping the indices and the symmetry of the covariance too \[\begin{equation} \mathbb{C}(\tau_k,\varepsilon_j) = \mathbb{C}(\varepsilon_j, \tau_k) = 0 \end{equation}\] applies. With the theorem for the pairwise addition of random variables (Theorem 25.5) and statement (3) of the present theorem then applies \[\begin{align} \begin{split} \mathbb{C}(y_j,y_k) & = \mathbb{C}(\tau_j + \varepsilon_j,\tau_k + \varepsilon_k) \\ & = \mathbb{C}(\tau_j ,\tau_k) + \mathbb{C}(\tau_j ,\varepsilon_k) + \mathbb{C}(\varepsilon_j,\tau_k) + \mathbb{C}(\varepsilon_j,\varepsilon_k) \\ & = \mathbb{C}(\tau_j,\tau_k) + 0 + 0 + 0 \\ & = \mathbb{C}(\tau_j,\tau_k). \end{split} \end{align}\]
In the sense of the simplified model of the classical test theory according to Definition 50.3, the statements of Theorem 50.3 are often also referred to as standard two-test measurements
- \(\mathbb{C}(y_{j},y_{k}|\tau_{j} = t_{j},\tau_{k} = t_{k}) = 0\) (2) \(\mathbb{C}(\varepsilon_{j},\varepsilon_{k}|\tau_{j} = t_{j},\tau_{k} = t_{k}) = 0\) (3) \(\mathbb{C}(\varepsilon_{j},\varepsilon_{k}) = 0\) (4) \(\mathbb{C}(\tau_{j},\varepsilon_{k}) = 0\) (5) \(\mathbb{C}(y_{j},y_{k}) = \mathbb{C}(\tau_{j},\tau_{k})\)
written. The first two statements of Theorem 50.3 state that, conditional on the respective true scores, the covariances of observed scores as well as the covariances of a person’s measurement errors between two test measurements are equal to zero. According to statement (3) of Theorem 50.3, this also applies to the measurement errors in the sense of the unconditional marginal distribution. This corresponds to the pairwise independence of measurement errors across test measurements in the model of multiple test measurements. Furthermore, according to statement (4), the covariance of the true score in one test measurement with the measurement error in another test measurement is zero. Finally, according to statement (5), the covariance of the observed scores of two test measurements is equal to the covariance of the true scores.
Example
As a first example of a model of multiple test measurements with \(m>1\), we continue the example from Theorem 50.2 and consider the case of two test measurements \(j = 1,2\). For \(i = 1,...,n\) are corresponding \[\begin{equation} \mathbb{P}(\tau_{i1}, \tau_{i2}) = \mathbb{P}(\tau_{i2}|\tau_{i1})\mathbb{P}(\tau_{i1}) \end{equation}\] with \[\begin{equation} \mathbb{P}(\tau_{i1}) := N(1,1) \mbox{ and } \mathbb{P}(\tau_{i2}|\tau_{i1}) := N(\tau_{i1} + 1,1) \end{equation}\] In this example, the distribution of the true score of person \(i\) in test measurement \(j = 2\) depends explicitly on the distribution of the true score of person \(i\) in test measurement \(j = 1\). Continue to be \[\begin{equation} \mathbb{P}(y_{i1}|\tau_{i1}) := N(\tau_{i1},1) \mbox{ and } \mathbb{P}(y_{i2}|\tau_{i2}) := N(\tau_{i2},2) \end{equation}\] The propensity distributions of person \(i\) in test measurement \(j = 1\) are different from those of person \(i\) in test measurement \(j = 2\). The following R code generates \(10^5\) realizations of this model and estimates the marginal covariances \(\mathbb{C}(\varepsilon_{i1}, \varepsilon_{i2})\), \(\mathbb{C}(\tau_{i1}, \varepsilon_{i2})\), \(\mathbb{C}(y_{i1}, y_{i2})\) and \(\mathbb{C}(\tau_{i1}, \tau_{i2})\) considered in Theorem 50.3.
n = 1e5 # number of persons
m = 2 # number of test measurements
mu = 1 # true-score expectation parameter
T = matrix(rep(NaN, n*m), nrow = n) # true-score array
Y = matrix(rep(NaN, n*m), nrow = n) # observed-score array
E = matrix(rep(NaN, n*m), nrow = n) # measurement-error array
for(i in 1:n){ # iteration over persons
T[i,1] = rnorm(1,1,1) # true score realization for j = 1
Y[i,1] = rnorm(1,T[i,1],1) # observed-score realization for j = 1
E[i,1] = Y[i,1] - T[i,1] # measurement error realization for j = 1
T[i,2] = rnorm(1,T[i,1] + 1,.5) # true score realization for j = 2
Y[i,2] = rnorm(1,T[i,2],.5) # observed-score realization for j = 2
E[i,2] = Y[i,2] - T[i,2]} # measurement error realization for j = 2
c_hat_e1_e2 = cov(E[,1],E[,2]) # covariance estimation measurement error 1, measurement error 2
c_hat_t1_e2 = cov(T[,1],E[,2]) # covariance estimation True score 1, measurement error 2
c_hat_y1_y2 = cov(Y[,1],Y[,2]) # covariance Estimation Observed Value 1, Observed Value 2
c_hat_t1_t2 = cov(T[,1],T[,2]) # covariance estimation True score 1, True score 2Figure 50.2 visualizes 500 of the random variable realizations generated in this way and documents the resulting covariance estimates. Figure 50.2 A shows the marginal uncorrelation of the measurement errors with respect to two test measurements (statement (2) of Theorem 50.3), Figure 50.2 B shows the marginal uncorrelation of the true score and the measurement error with respect to two test measurements (statement (3) of Theorem 50.3) and Figure 50.2 C and Figure 50.2 D show the equality of the marginal covariances of observed and true scores with respect to two test measurements.
50.1.2 The model of parallel test measurements
So far, in the model of multiple test measurements, we have not made any statements about the ratios of the true scores across different test measurements. On the one hand, we assumed that independence does not have to apply to the marginal distribution of the test measurements for a person \(i\), so that in general it applies that for \(i = 1,...,n\) \[\begin{equation} \mathbb{P}(\tau_{i1},...,\tau_{im}) \neq \prod_{j=1}^m\mathbb{P}(\tau_{ij}). \end{equation}\] On the other hand, we have not yet specified the form of possible dependencies between the true scores \(\tau_{i1},..., \tau_{im}\). In this regard, classical test theory considers some special cases, which are generally characterized by functional dependencies between \(\tau_{i1}\) and \(\tau_{i2},...,\tau_{im}\) of the form \[\begin{equation} \tau_{ij} = f(\tau_{i1}) \mbox{ for } f : \mathbb{R} \to \mathbb{R} \mbox{ and } j = 2,...,m. \end{equation}\] express. In the following we consider the case that \(f:= \mbox{id}_{\mathbb{R}}\), that is, in particular for realizations \(t_{ij}\) of \(\tau_{ij}\) \[\begin{equation} t_{ij} = \mbox{id}_{\mathbb{R}}\left(t_{i1}\right) = t_{i1} \mbox{ for } j = 2,...,m, \end{equation}\] that is, the values of a person’s true scores are identical across test measurements. Classical test theory refers to such test measurements as parallel test measurements. Another form of functional dependency, which we do not want to delve into further here, is the case where \(f\) is a linear-affine function, i.e \[\begin{equation} \tau_{ij} = f(\tau_{i1}) = a\tau_{i1} + b \mbox{ for } a,b \in \mathbb{R} \mbox{ and } j = 2,...,m. \end{equation}\] Classical test theory refers to such test measurements as essentially \(\tau\) -equivalent test measurements.
Definition 50.4 (Model of parallel test measurements) For \(i = 1,...,n\) and \(j = 1,...,m\), let \(\tau_{i}\) be a random variable that models the true score of the \(i\) th person in each test measurement, \(y_{ij}\) a random variable that models the observed score of the \(i\) th person in the \(j\) th test measurement, and \(\varepsilon_{ij}:= y_{ij} -\tau_{i}\) the random variable that model the measurement error of the \(i\) th person in the \(j\) th test measurement. Then the joint distribution is called \(\tau_{i}\) and \(y_{ij}\) with the factorization properties \[\begin{equation} \mathbb{P}\left(\tau_{1},y_{11},...y_{1m}, ...,\tau_{n},y_{n1},...,y_{nm}\right) := \prod_{i=1}^n \mathbb{P}(\tau_{i}) \prod_{j=1}^m \mathbb{P}(y_{ij}|\tau_{i}) \end{equation}\] the model of parallel test measurements if it holds that
- \(\mathbb{P}(\tau_{1}) = \cdots = \mathbb{P}(\tau_{n})\) (2) \(\mathbb{P}(y_{1j}|\tau_{1}) = \cdots = \mathbb{P}(y_{nj}|\tau_{n})\) for all \(1 \le j \le m\) (3) \(\mathbb{E}(y_{ij}\vert \tau_i = t_i) = \mathbb{E}(y_{ik}\vert \tau_i = t_i):= t_i\) for all \(1 \le i \le n, 1 \le j,k \le m\) (4) \(\mathbb{V}(y_{ij}\vert \tau_i = t_i) = \mathbb{V}(y_{ik}\vert \tau_i = t_i)\) for all \(1 \le i \le n, 1 \le j,k \le m\)
In particular, in the model of parallel test measurements, the values of a person’s true score are assumed to be identical across test measurements. This means that the variance of a person’s observed scores between two test measurements is solely due to the propensity distribution. From a generative point of view, values of the observed scores arise as follows: First, for the \(i\) th person and test measurements \(j = 1,...,m\), a true score \(t_{i}\) of according to \(\mathbb{P}(\tau_{i})\) is realized. Then an observed score \(y_{ij}\) is realized based on \(\mathbb{P}(y_{ij}|\tau_{i} = t_{i})\) for the \(i\) te and the test measurement \(j = 1,...,m\).
Properties of the model of parallel test measurements
If one only considers a single test measurement, the model of parallel test measurements obviously has the same form as the model of multiple test measurements. This means that Theorem 50.1 also applies analogously to a test measurement for the model of parallel test measurements. However, if you consider more than one test measurement, the model of parallel test measurements has more specific properties, which we capture in the following theorem
Theorem 50.4 (Properties of the model of parallel test measurements regarding two test measurements) The model of parallel test measurements is given. Then this applies to all \(i = 1,...,n\) and all \(j,k\) with \(1 \le j,k \le m\) and \(j \neq k\)
- \(\mathbb{E}(y_{ij}) = \mathbb{E}(y_{ik})\) (2) \(\mathbb{V}(y_{ij}) = \mathbb{V}(y_{ik})\) (3) \(\mathbb{C}(\varepsilon_{ij},\varepsilon_{ik}) = 0\) (4) \(\mathbb{C}(\tau_{i},\varepsilon_{ik}) = 0\) (5) \(\mathbb{C}(y_{ij},y_{ik}) = \mathbb{V}(\tau_{i})\)
Proof. To simplify the notation, set the proof first \[\begin{equation} y_j := y_{ij}, y_k := y_{ik}, \tau := \tau_{i}, \tilde{y}_j := \tilde{y}_{ij}, \tilde{y}_k := \tilde{y}_{ik}, t := t_{i} , Y_j := Y_{ij}, Y_k := Y_{ik} \mbox{ and } T := T_{i} \end{equation}\] for all \(i = 1,...,n\). We continue to consider only the discrete case, so we assume the existence of a probability mass function of the form \[\begin{equation} p(t,\tilde{y}_j,\tilde{y}_k) = p(\tilde{y}_j|t)p(\tilde{y}_k|t)p(t) \end{equation}\] in advance. The continuous case then follows analogously.
(1) With the equality of the conditional expected values in the case of parallel test measurements applies \[\begin{equation} \mathbb{E}(y_{j}) = \sum_{t \in T}\sum_{\tilde{y}_j \in Y_j}\tilde{y}_j p(\tilde{y}_j|t)p(t) = \sum_{t \in T}\mathbb{E}(y_{j}\vert \tau = t)p(t) = \sum_{t \in T}\sum_{\tilde{y}_k \in Y_k}\tilde{y}_k p(\tilde{y}_k|t)p(t) = \mathbb{E}(y_{k}). \end{equation}\]
(2) With the iterated variance theorem (Theorem 24.6) holds \[\begin{equation} \mathbb{V}(y_{j}) = \mathbb{V}\left(\mathbb{E}(y_j\vert \tau)\right) + \mathbb{E}\left(\mathbb{V}(y_j\vert \tau)\right) = \mathbb{V}\left(\mathbb{E}(y_k\vert \tau)\right) + \mathbb{E}\left(\mathbb{V}(y_k\vert \tau)\right) = \mathbb{V}(y_{k}). \end{equation}\]
(3) We first determine \(\mathbb{E}(\varepsilon_j\varepsilon_k)\). With statement (2) of Theorem 50.1 then applies \[\begin{align} \begin{split} \mathbb{E}(\varepsilon_j\varepsilon_k) & := \mathbb{E}((y_j - \tau)(y_k - \tau)) \\ & = \sum_{t \in T} \sum_{\tilde{y}_j \in Y_j} \sum_{\tilde{y}_k \in Y_k}(\tilde{y}_j - t)(\tilde{y}_k - t) p(t) p(\tilde{y}_j|t)p(\tilde{y}_k|t) \\ & = \sum_{t \in T} p(t) \sum_{\tilde{y}_j \in Y_j} \sum_{\tilde{y}_k \in Y_k}(\tilde{y}_j - t)(\tilde{y}_k - t) p(\tilde{y}_j|t)p(\tilde{y}_k|t) \\ & = \sum_{t \in T} p(t) \sum_{\tilde{y}_j \in Y_j} (\tilde{y}_j - t) p(\tilde{y}_j|t) \sum_{\tilde{y}_k \in Y_k}(\tilde{y}_k - t) p(\tilde{y}_k|t) \\ & = \sum_{t \in T} p(t) \left(\sum_{\tilde{y}_j \in Y_j} \tilde{y}_j p(\tilde{y}_j|t) - t\sum_{\tilde{y}_j \in Y_j}p(\tilde{y}_j|t)\right) \left(\sum_{\tilde{y}_k \in Y_k}\tilde{y}_k p(\tilde{y}_k|t) - t\sum_{\tilde{y}_k \in Y_k}p(\tilde{y}_k|t)\right) \\ & = \sum_{t \in T} p(t)\left(t-t\right)\left(t-t\right) \\ & = \sum_{t \in T} p(t)\cdot 0 \cdot 0 \\ & = 0. \\ \end{split} \end{align}\] Using the covariance shift theorem (Theorem 25.2) and again with statement (2) of Theorem 50.1, it then follows that \[\begin{align} \begin{split} \mathbb{C}(\varepsilon_j,\varepsilon_k) = \mathbb{E}(\varepsilon_j\varepsilon_k) - \mathbb{E}(\varepsilon_j)\mathbb{E}(\varepsilon_k) = 0 - 0\cdot 0 = 0 \end{split} \end{align}\] (4) Using the covariance shift theorem (Theorem 25.2) and statement (2) of Theorem 50.1, we have \[\begin{align} \begin{split} \mathbb{C}(\tau,\varepsilon_k) & = \mathbb{E}(\tau\varepsilon_k) - \mathbb{E}(\tau)\mathbb{E}(\varepsilon_k) \\ & = \mathbb{E}(\tau\varepsilon_k) - \mathbb{E}(\tau)\cdot 0 \\ & = \mathbb{E}(\tau(y_k - \tau)) \\ & = \sum_{t \in T} \sum_{\tilde{y}_k \in Y_k} t(\tilde{y}_k - t) p(t)p(\tilde{y}_k|t) \\ & = \sum_{t \in T} t p(t) \sum_{\tilde{y}_k \in Y_k}(\tilde{y}_k - t) p(\tilde{y}_k|t) \\ & = \sum_{t \in T} t p(t) \left(\sum_{\tilde{y}_k \in Y_k}\tilde{y}_k p(\tilde{y}_k|t) - t \sum_{\tilde{y}_k \in Y_k}p(\tilde{y}_k|t)\right)\\ & = \sum_{t \in T} t p(t) \left(t - t\right)\\ & = \sum_{t \in T} t p(t) \cdot 0\\ & = 0. \\ \end{split} \end{align}\]
(5) We first note that with statement (2) of the theorem next to \(\mathbb{C}(\tau,\varepsilon_k)=0\) by exchanging the index and the symmetry of the covariance too \[\begin{equation} \mathbb{C}(\tau,\varepsilon_j) = \mathbb{C}(\varepsilon_j, \tau) = 0 \end{equation}\] applies. With the theorem on covariance for pairwise addition of random variables (Theorem 25.5) and statements (3) and (4) of the present theorem then applies \[\begin{align} \begin{split} \mathbb{C}(y_j,y_k) & = \mathbb{C}(\tau + \varepsilon_j,\tau + \varepsilon_k) \\ & = \mathbb{C}(\tau ,\tau) + \mathbb{C}(\tau ,\varepsilon_k) + \mathbb{C}(\varepsilon_j,\tau) + \mathbb{C}(\varepsilon_j,\varepsilon_k) \\ & = \mathbb{C}(\tau,\tau) + 0 + 0 + 0 \\ & = \mathbb{C}(\tau,\tau) \\ & = \mathbb{V}(\tau). \end{split} \end{align}\]
Statement (1) of Theorem 50.4 states that in parallel test measurements all expected values of the observed scores are identical and statement (2) of the theorem states that in parallel test measurements all variances of the observed scores are identical. Statements (3) and (4) of Theorem 50.4 are analogous to the local uncorrelatedness properties of the multiple test measurements model. Finally, statement (5) states in particular that \(\mathbb{C}(y_{ij},y_{ik})\) is identical to \(\mathbb{V}(\tau_{i})\) for any \(j\) and \(k\).
Example
We consider the case of two test measurements \(j = 1,2\) in the model of parallel test measurements. For \(i = 1,...,n\) be \[\begin{equation} \mathbb{P}(\tau_{i}) = N(1,1) \mbox{ and } \mathbb{P}(y_{i1}|\tau_{i}) := \mathbb{P}(y_{i2}|\tau_{i}) := N(\tau_{i},1) \end{equation}\] For person \(i\) there is only one true score random variable for all test measurements and the propensity distributions do not differ between test measurements.
n = 1e5 # number of persons
m = 2 # number of test measurements
T = matrix(rep(NaN, n) , nrow = n) # true-score array
Y = matrix(rep(NaN, n*m), nrow = n) # observed-score array
E = matrix(rep(NaN, n*m), nrow = n) # measurement-error array
for(i in 1:n){ # person iterations
T[i] = rnorm(1,1,1) # true score realization for j = 1.2
for(j in 1:m){ # test-measurement iterations
Y[i,j] = rnorm(1,T[i],1) # observed-score realization f
E[i,j] = Y[i,j] - T[i]}} # measurement-error realization
e_hat_o1_o2 = apply(Y, 2, mean) # expected value estimate Observed score 1, Observed score 2
v_hat_o1_o2 = apply(Y, 2, var) # variance Estimation Observed Value 1, Observed Value 2
c_hat_e1_e2 = cov(E[,1],E[,2]) # covariance estimation measurement error 1, measurement error 2
c_hat_t_e2 = cov(T ,E[,2]) # covariance estimation True score 1, measurement error 2
c_hat_y1_y2 = cov(Y[,1],Y[,2]) # covariance Estimation Observed Value 1, Observed Value 2
v_hat_t = var(T) # variance Estimation True ValueFigure 50.3 visualizes 500 of the random variable realizations generated in this way and documents the resulting covariance estimates. Figure 50.3 A shows the uncorrelation of the measurement errors with respect to two parallel test measurements (statement (3) of Theorem 50.4), Figure 50.3 B shows the uncorrelation of the true score and the measurement error with respect to two parallel test measurements (statement (4) of Theorem 50.4) and Figure 50.3 C shows the correlation of the observed scores induced by the identical true scores two parallel test measurements.
50.2 Reliability
The reliability of a test measurement is the central concept of classical test theory. Here we follow the approach according to Lord & Novick (1968), according to which the reliability of a test measurement is defined as the squared correlation of the observed and true score. Based on the properties of the model of multiple test measurements, there are different ways to represent this correlation and thus different ways to interpret the reliability of a test measurement. However, there is no way to empirically measure the reliability of test measurements. What is then central is the transfer of the concept of reliability into the model of parallel test measurements. The parallel test reliability defined in this way can then be estimated empirically.
50.2.1 Reliability of a test measurement
We first begin by defining the reliability of a test measurement in the multiple test measurement model.
Definition 50.5 (Reliability of a test measurement) The model of multiple test measurements is given for an arbitrary test measurement \(j\) with \(1 \le j \le m\), \[\begin{equation} \mathbb{P}(\tau_{1j},y_{1j}, ..., \tau_{nj}, y_{nj}) := \prod_{i=1}^n \mathbb{P}(\tau_{ij},y_{ij}) := \prod_{i=1}^n \mathbb{P}(y_{ij}|\tau_{ij})\mathbb{P}(\tau_{ij}) \end{equation}\] where according to the definition of the model it holds that \[\begin{equation} \mathbb{P}(\tau_{1j},y_{1j}) = \cdots = \mathbb{P}(\tau_{nj},y_{nj}). \end{equation}\] The reliability of the test measurement \(j\) is then defined as \[\begin{equation} \mbox{R}_j := \rho(y_{ij},\tau_{ij})^2 \mbox{ for any } 1 \le i \le n. \end{equation}\]
Against the background of the simplified model of classical test theory Definition 50.3 is also written \[\begin{equation} \mbox{R} = \rho(y,\tau)^2. \end{equation}\] By definition, the reliability of a test measurement is the squared correlation of the observed and true scores. With \[\begin{equation} -1 \le \rho(y_{ij},\tau_{ij})\le 1 \end{equation}\] It follows directly that the reliability of a test measurement is as follows \[\begin{equation} 0 \le \mbox{R}_j \le 1 \end{equation}\] The statement \(\mbox{R}_j = 0\) then implies \(\rho(y_{ij},\tau_{ij}) = 0\), i.e. the linear-affine independence of the observed and true score, and thus the fact that the observed score is not meaningful with regard to the true score. \(\mbox{R}_j= 1\), on the other hand, implies \(\rho(y_{ij},\tau_{ij}) = \pm 1\), i.e. the deterministic linear-affine dependence of the observed and true score and thus that the observed score is completely meaningful with regard to the true score.
The following theorem shows further possibilities for formulating and thus interpreting the reliability of a test measurement in an equivalent manner.
Theorem 50.5 (Characteristics of the reliability of a test measurement) The model of multiple test measurements is given for an arbitrary test measurement \(j\) with \(1 \le j \le m\), \[\begin{equation} \mathbb{P}(\tau_{1j},y_{1j}, ..., \tau_{nj}, y_{nj}) := \prod_{i=1}^n \mathbb{P}(y_{ij}|\tau_{ij})\mathbb{P}(\tau_{ij}). \end{equation}\] Then the following applies to the reliability \(\mbox{R}_j\) of the test measurement \(j\)
\(\mbox{R}_j = \frac{\mathbb{V}(\tau_{ij})}{\mathbb{V}(y_{ij})}\)
\(\mbox{R}_j = 1 - \frac{\mathbb{V}(\varepsilon_{ij})}{\mathbb{V}(y_{ij})}\)
Proof. (1) With statement (5) from Theorem 50.1 applies \[\begin{equation} \mbox{R}_j = \rho(y_{ij},\tau_{ij})^2 = \left(\frac{\mathbb{C}(y_{ij},\tau_{ij})}{\mathbb{S}(y_{ij})\mathbb{S}(\tau_{ij})}\right)^2 = \frac{\mathbb{C}(y_{ij},\tau_{ij})^2}{\mathbb{V}(y_{ij})\mathbb{V}(\tau_{ij})} = \frac{\mathbb{V}(\tau_{ij})^2}{\mathbb{V}(y_{ij})\mathbb{V}(\tau_{ij})} = \frac{\mathbb{V}(\tau_{ij})}{\mathbb{V}(y_{ij})}. \end{equation}\]
(2) With statement (4) from Theorem 50.1 then continues to apply \[\begin{equation} \mbox{R}_j = \frac{\mathbb{V}(\tau_{ij})}{\mathbb{V}(y_{ij})} = \frac{\mathbb{V}(y_{ij})-\mathbb{V}(\varepsilon_{ij})}{\mathbb{V}(y_{ij})} = \frac{\mathbb{V}(y_{ij})}{\mathbb{V}(y_{ij})} - \frac{\mathbb{V}(\varepsilon_{ij})}{\mathbb{V}(y_{ij})} = 1 - \frac{\mathbb{V}(\varepsilon_{ij})}{\mathbb{V}(y_{ij})}. \end{equation}\]
Since \(\tau_{ij}\) and \(\varepsilon_{ij}\) are still latent and can therefore only be observed indirectly, the representations according to Theorem 50.5 are only of theoretical interest. The first statement states in particular that the reliability of a test measurement is the proportion of the variance of the true scores to the variance of the observed scores, \[\begin{equation} \mbox{R}_j = \frac{\mathbb{V}(\tau_{ij})}{\mathbb{V}(y_{ij})} = \frac{\mathbb{V}(\tau_{ij})}{\mathbb{V}(\tau_{ij}) + \mathbb{V}(\varepsilon_{ij})}. \end{equation}\] If \(\mathbb{V}(\tau_{ij}) = 0\) is \(\mbox{R}_j = 0\), if \(\mathbb{V}(\varepsilon_{ij}) = 0\) is \(\mbox{R}_j = 1\). A reliability \(\mbox{R}_j > 0\) always implies a non-zero variance of the true scores.
50.2.2 Parallel test reliability
In order to make the concept of reliability estimable using empirical data from observed scores, it needs to be transferred into the model of parallel test measurements. We therefore first explicitly define the reliability of a parallel test measurement.
Definition 50.6 (Reliability of a parallel test measurement) The model of parallel test measurements is given for an arbitrary test measurement \(j\) with \(1 \le j \le m\), \[\begin{equation} \mathbb{P}(\tau_{1},y_{1j}, ..., \tau_{n},y_{nj}) := \prod_{i=1}^n \mathbb{P}(\tau_i, y_{ij}) = \prod_{i=1}^n \mathbb{P}(\tau_i)\mathbb{P}(y_{ij}|\tau_i) \end{equation}\] where according to the definition of the model it holds that \[\begin{equation} \mathbb{P}(\tau_{1},y_{1j}) = \cdots = \mathbb{P}(\tau_{n},y_{nj}). \end{equation}\] The reliability of the parallel test measurement \(j\) is then defined as \[\begin{equation} \mbox{R}_j := \rho(y_{ij},\tau_{i})^2 \mbox{ for any } 1 \le i \le n. \end{equation}\]
It is also written here against the background of the simplified model of classical test theory (Definition 50.3). \[\begin{equation} \mbox{R}=\rho(y,\tau)^2. \end{equation}\] The following theorem is of practical importance.
Theorem 50.6 (Parallel test reliability) The model of parallel test measurements is given \[\begin{equation} \mathbb{P}(\tau_{1},y_{1j}, ..., \tau_{n},y_{nj}) := \prod_{i=1}^n \mathbb{P}(\tau_i, y_{ij}) = \prod_{i=1}^n \mathbb{P}(\tau_i)\mathbb{P}(y_{ij}|\tau_i) \end{equation}\] Then apply
- \(\mbox{R}_j = \frac{\mathbb{V}(\tau_{i})}{\mathbb{V}(y_{ij})}\) for all \(1 \le j \le m\). (2) \(\mbox{R}_j = 1 - \frac{\mathbb{V}(\varepsilon_{ij})}{\mathbb{V}(y_{ij})}\) for all \(1 \le j \le m\). (3) \(\mbox{R}_j = \rho(y_{ij},y_{ik}) = \mbox{R}_k\) for all \(1 \le j,k \le m\).
Proof. (1) With statement (5) from Theorem 50.1 applies \[\begin{equation} \mbox{R}_j = \rho(y_{ij},\tau_{i})^2 = \left(\frac{\mathbb{C}(y_{ij},\tau_{i})}{\mathbb{S}(y_{ij})\mathbb{S}(\tau_{i})}\right)^2 = \frac{\mathbb{C}(y_{ij},\tau_{i})^2}{\mathbb{V}(y_{ij})\mathbb{V}(\tau_{i})} = \frac{\mathbb{V}(\tau_{i})^2}{\mathbb{V}(y_{ij})\mathbb{V}(\tau_{i})} = \frac{\mathbb{V}(\tau_{i})}{\mathbb{V}(y_{ij})}. \end{equation}\] (2) Statement (4) from Theorem 50.1 then continues to apply \[\begin{equation} \mbox{R}_j = \frac{\mathbb{V}(\tau_{i})}{\mathbb{V}(y_{ij})} = \frac{\mathbb{V}(y_{ij})-\mathbb{V}(\varepsilon_{ij})}{\mathbb{V}(y_{ij})} = \frac{\mathbb{V}(y_{ij})}{\mathbb{V}(y_{ij})} - \frac{\mathbb{V}(\varepsilon_{ij})}{\mathbb{V}(y_{ij})} = 1 - \frac{\mathbb{V}(\varepsilon_{ij})}{\mathbb{V}(y_{ij})}. \end{equation}\] (3) With statements (2) and (5) from Theorem 50.3 then continues to apply \[\begin{equation} \mbox{R}_j = \frac{\mathbb{V}(\tau_{i})}{\mathbb{V}(y_{ij})} = \frac{\mathbb{C}(y_{ij},y_{ik})}{\sqrt{\mathbb{V}(y_{ij})}\sqrt{\mathbb{V}(y_{ij})}} = \frac{\mathbb{C}(y_{ij},y_{ik})}{\sqrt{\mathbb{V}(y_{ij})}\sqrt{\mathbb{V}(y_{ik})}} = \rho(y_{ij},y_{ik}) \end{equation}\] and that too \[\begin{equation} \mbox{R}_k = \frac{\mathbb{V}(\tau_{i})}{\mathbb{V}(y_{ik})} = \frac{\mathbb{C}(y_{ij},y_{ik})}{\sqrt{\mathbb{V}(y_{ik})}\sqrt{\mathbb{V}(y_{ik})}} = \frac{\mathbb{C}(y_{ij},y_{ik})}{\sqrt{\mathbb{V}(y_{ij})}\sqrt{\mathbb{V}(y_{ik})}} = \rho(y_{ij},y_{ik}). \end{equation}\]
Statements (1) and (2) of Theorem 50.6 are analogous to the statements of Theorem 50.5. Statement (3) from Theorem 50.6 justifies the practical procedure for estimating reliability using parallel or retest procedures. To determine the reliability of a test, an estimate of the correlation between two parallel test measurements is used, and classical test theory justifies this approach by assuming latent true scores and measurement errors. Statement (3) of Theorem 50.6 is often simplified by saying that “the correlation of parallel test measurements is equal to their reliability”.
Example
We consider the case of two test measurements \(j = 1,2\) in the model of parallel test measurements. For all \(i = 1,...,n\), although for the sake of notational simplicity we want to forego the \(i\) subscript, \[\begin{equation} p(t) = N(t; 0,\sigma_\tau^2) \mbox{ and } p(\tilde{y}_1|t) := N(\tilde{y}_1; t,\sigma_{\varepsilon}^2) \mbox{ and } p(\tilde{y}_2|t) := N(\tilde{y}_2; t,\sigma_{\varepsilon}^2) \end{equation}\] So for person \(i\) there is only one random variable for the true score for all test measurements and the propensity distributions do not differ between test measurements. Then the theorem applies to common normal distributions (Theorem 29.5) with \(A:= 1\) and \(b:= 0\) \[\begin{equation} p(t)p(\tilde{y}_1|t) = p(t,\tilde{y}_1) = N\left( \begin{pmatrix} t \\ \tilde{y}_1 \end{pmatrix}; \begin{pmatrix} \mu_{\tau} \\ \mu_\tau \end{pmatrix}, \begin{pmatrix} \sigma_{\tau}^2 & \sigma_{\tau}^2 \\ \sigma_{\tau}^2 & \sigma_{\tau}^2 + \sigma_{\varepsilon}^2 \end{pmatrix} \right) \end{equation}\] and continue with \(A:= \begin{pmatrix} 1 & 0\end{pmatrix}\) and \(b:= 0\) \[\begin{equation} p(t,\tilde{y}_1)p(\tilde{y}_2|t) = p(t,\tilde{y}_1,\tilde{y}_2) = N\left( \begin{pmatrix} t \\ \tilde{y}_1 \\ \tilde{y}_2 \end{pmatrix}; \begin{pmatrix} \mu_\tau \\ \mu_\tau \\ \mu_\tau \\ \end{pmatrix}, \begin{pmatrix} \sigma_{\tau}^2 & \sigma_{\tau}^2 & \sigma_{\tau}^2 \\ \sigma_{\tau}^2 & \sigma_{\tau}^2 + \sigma_{\varepsilon}^2 & \sigma_{\tau}^2 \\ \sigma_{\tau}^2 & \sigma_{\tau}^2 & \sigma_{\tau}^2 + \sigma_{\varepsilon}^2 \end{pmatrix} \right) \end{equation}\] This means that by reading the covariance matrix parameter from \(p(t,\tilde{y}_1,\tilde{y}_2)\), it applies that \[\begin{equation} \rho(y_1, y_2) = \frac{\mathbb{C}(y_1, y_2)}{\sqrt{\mathbb{V}(y_1)}\sqrt{\mathbb{V}(y_2)}} = \frac{\sigma_{\tau}^2}{\sqrt{\sigma_{\tau}^2 + \sigma_{\varepsilon}^2}\sqrt{\sigma_{\tau}^2 + \sigma_{\varepsilon}^2}} = \frac{\sigma_{\tau}^2}{\sigma_{\tau}^2 + \sigma_{\varepsilon}^2}. \end{equation}\] This also applies in particular to \(j = 1,2\) \[\begin{equation} \mbox{R}_j = \rho(y_j,\tau)^2 = \left(\frac{\mathbb{C}(y_j,\tau)}{\mathbb{S}(y_j)\mathbb{S}(\tau)}\right)^2 = \frac{\mathbb{C}(y_j,\tau)^2}{\mathbb{V}(y_j)\mathbb{V}(\tau)} = \frac{\left(\sigma_\tau^2\right)^2}{(\sigma_{\tau}^2 + \sigma_{\varepsilon}^2)\sigma_\tau^2} = \frac{\sigma_\tau^2}{\sigma_{\tau}^2 + \sigma_{\varepsilon}^2} = \rho(y_1, y_2). \end{equation}\] For example, if \(\sigma_{\tau}^2:= 1.0\) and \(\sigma_{\varepsilon}^2:= 0.2\) apply, this results in parallel test reliability \[\begin{equation} \mbox{R}_j = \rho(y_j,\tau)^2 = \rho(y_1, y_2) = \frac{1.0}{1.0 + 0.2} \approx 0.83. \end{equation}\]
50.2.3 Estimation of parallel test reliability
As usual, the correlation between two parallel test measurements in the application is estimated using a sample correlation. We formulate this for the present scenario using the following definition.
Definition 50.7 (Parallel test reliability estimator) Given is the model of parallel test measurements for \(n\) people and two test measurements \(j = 1,2\), \[\begin{equation} \mathbb{P}(\tau_1,y_{11},y_{12},...,\tau_n,y_{n1},y_{n2}) = \prod_{i=1}^n\mathbb{P}(\tau_i)\mathbb{P}(y_{i1}|\tau_i)\mathbb{P}(y_{i2}|\tau_i). \end{equation}\] Then the one with the sample means \[\begin{equation} \bar{y}_1 := \frac{1}{n}\sum_{i=1}^n y_{i1} \mbox{ and } \bar{y}_2 := \frac{1}{n}\sum_{i=1}^n y_{i2} \end{equation}\] defined sample correlation coefficient \[\begin{equation} r_{12}:= \frac{\frac{1}{n-1}\sum_{i=1}^n(y_{i1} - \bar{y}_1)(y_{i2} - \bar{y}_2)}{\sqrt{\frac{1}{n-1}\sum_{i=1}^n (y_{i1} - \bar{y}_1)^2}\sqrt{\frac{1}{n-1}\sum_{i=1}^n (y_{i2} - \bar{y}_2)^2}} \end{equation}\] called parallel test reliability estimator.
Example
As is well known, R provides a way to calculate sample correlation coefficients with the cor() function. We demonstrate this using a simulation example. Be there for that \[\begin{equation}
p(t_i) = N(t_i; 0,\sigma_\tau^2)
\mbox{ and }
p(y_{ij}|t_i) := N(y_{ij}; t_i,\sigma_{\varepsilon}^2)
\end{equation}\] for \(j = 1,2\) with \(\sigma_\tau^2:= 1.0\) and \(\sigma_{\varepsilon}^2:= 0.2\) for \(i =
1,...,30\). The true, but unknown, parallel test reliability results here \[\begin{equation}
R_{12} = \frac{1.0}{1.0 + 0.2} \approx 0.833.
\end{equation}\] Using \(n = 30\) simulated data points, this is estimated as \(r_{12} = 0.857\) in the following simulation.
set.seed(1)
n = 30 # number of persons
m = 2 # number of test measurements
sigsqr_tau = 1 # true-score variance
sigsqr_eps = .2 # observed-score variance
R_12 = sigsqr_tau/(sigsqr_tau+sigsqr_eps) # parallel-test reliability
T = matrix(rep(NaN, n) , nrow = n) # true-score array
Y = matrix(rep(NaN, n*m), nrow = n) # observed-score array
E = matrix(rep(NaN, n*m), nrow = n) # measurement-error array
for(i in 1:n){ # person iterations
T[i] = rnorm(1,1,sqrt(sigsqr_tau)) # true score realization for j = 1.2
for(j in 1:m){ # test-measurement iterations
Y[i,j] = rnorm(1,T[i],sqrt(sigsqr_eps)) # observed-score realization
E[i,j] = Y[i,j] - T[i]}} # measurement-error realization
r_12 = cor(Y[,1],Y[,2]) # parallel test reliability estimator
cat("Parallel-test reliability R_12 : ", round(R_12,4), # edition w.a.u. Parallel test reliability
"\nParallel-test reliability estimator r_12 : ", round(r_12,4)) # output estimated parallel test reliabilityParallel-test reliability R_12 : 0.8333
Parallel-test reliability estimator r_12 : 0.8569In order to specify a confidence interval for the parallel test reliability in addition to a point estimate, an assumption about the empirical distribution of the reliability estimate is required. To do this, the following statement about the asymptotic distribution of a sample correlation is usually used.
Theorem 50.7 (Approximate distribution of the Fisher transform of a sample correlation) Given a sample \((y_{11},y_{12}),...,(y_{n1},y_{n2}) \sim \mathbb{P}(y_1,y_2)\) of observations of two random variables \(y_1\) and \(y_2\) with correlation \(\rho:= \rho(y_1,y_2)\) and sample correlation \(r\). Continue to denote \[\begin{equation} \tilde{r} := \frac{1}{2} \ln \left(\frac{1 + r}{1 - r} \right) \end{equation}\] the Fisher transformation of \(r\). Then \(\tilde{r}\) is asymptotically normally distributed with \[\begin{equation} \tilde{r} \stackrel{a}{\sim} N\left(\frac{1}{2}\ln\left(\frac{1+\rho}{1-\rho}\right), (n-3)^{-1} \right). \end{equation}\]
We forego a proof of Theorem 50.7 and refer to Chapter 32 in Johnson et al. (1994) for a detailed presentation. Theorem 50.7 forms the basis for constructing the approximate confidence interval for a correlation given in the following theorem.
Theorem 50.8 (Approximate confidence interval of a correlation) Given is a sample \((y_{11},y_{12}),...,(y_{n1},y_{n2}) \sim \mathbb{P}(y_1,y_2)\) of observations of two random variables \(y_1\) and \(y_2\) with correlation \(\rho\), sample correlation \(r\) and Fisher transformation \(\tilde{r}\). Furthermore, let \(\delta \in ]0,1[\) be a confidence level and let it be \[\begin{equation} z_\delta := \phi^{-1}\left(\frac{1+\delta}{2}\right) \end{equation}\] with the inverse KVF \(\phi^{-1}\) of a standard normally distributed random variable. Finally be \[\begin{equation} \tilde{r}_u := \tilde{r} - z_\delta\left(\sqrt{n - 3}\right)^{-1} \mbox{ and } \tilde{r}_o := \tilde{r} + z_\delta\left(\sqrt{n - 3}\right)^{-1}. \end{equation}\] Then \(n \to \infty\) applies to the interval \[\begin{equation} \kappa(r) := \left[ \frac{\exp(2\tilde{r}_u)- 1}{\exp(2\tilde{r}_u) + 1}, \frac{\exp(2\tilde{r}_o)- 1}{\exp(2\tilde{r}_o) + 1} \right], \end{equation}\] that \[\begin{equation} \mathbb{P}_{\rho}\left(\kappa(r) \ni \rho \right) = \delta. \end{equation}\]
Proof. With Theorem 50.7, the \(Z\) transformation of \(r\) initially applies to \(n \to \infty\) \[\begin{equation} \tilde{r}_z := \left(\tilde{r} - \frac{1}{2}\ln\left(\frac{1+\rho}{1-\rho}\right)\right)\sqrt{(n-3)} \stackrel{a}{\sim} N(0,1) \end{equation}\] and therefore asymptotically that \[\begin{equation} \mathbb{P}(-z_\delta \le \tilde{r}_z \le z_\delta) = \delta. \end{equation}\] So asymptotically it also holds that \[\begin{align} \begin{split} \delta & =\mathbb{P}\left(-z_\delta \le \tilde{r}_z \le z_\delta\right) \\ & = \mathbb{P}\left(-z_\delta \le \left(\tilde{r} - \frac{1}{2}\ln\left(\frac{1+\rho}{1-\rho}\right)\right)\sqrt{(n-3)} \le z_\delta\right) \\ & = \mathbb{P}\left(-z_\delta\left(\sqrt{(n-3)}\right)^{-1} \le \tilde{r} - \frac{1}{2}\ln\left(\frac{1+\rho}{1-\rho}\right) \le z_\delta\left(\sqrt{(n-3)}\right)^{-1}\right) \\ & = \mathbb{P}\left(-z_\delta\left(\sqrt{(n-3)}\right)^{-1}-\tilde{r} \le - \frac{1}{2}\ln\left(\frac{1+\rho}{1-\rho}\right) \le z_\delta\left(\sqrt{(n-3)}\right)^{-1}\right) - \tilde{r} \\ & = \mathbb{P}\left(\tilde{r} + z_\delta\left(\sqrt{(n-3)}\right)^{-1} \ge \frac{1}{2}\ln\left(\frac{1+\rho}{1-\rho}\right) \ge \tilde{r} - z_\delta\left(\sqrt{(n-3)}\right)^{-1}\right) \\ & = \mathbb{P}\left(2\left(\tilde{r} + z_\delta\left(\sqrt{(n-3)}\right)^{-1}\right) \ge \ln\left(\frac{1+\rho}{1-\rho}\right) \ge 2\left(\tilde{r} - z_\delta\left(\sqrt{(n-3)}\right)^{-1}\right)\right) \\ & = \mathbb{P}\left(2\left(\tilde{r} - z_\delta\left(\sqrt{(n-3)}\right)^{-1}\right) \le \ln\left(\frac{1+\rho}{1-\rho}\right) \le 2\left(z_\delta\left(\sqrt{(n-3)}\right)^{-1}+\tilde{r}\right) \right) \\ & = \mathbb{P}\left(2\tilde{r}_u \le \ln\left(\frac{1+\rho}{1-\rho}\right) \le 2\tilde{r}_o \right) \\ & = \mathbb{P}\left(\exp\left(2\tilde{r}_u\right) \le \frac{1+\rho}{1-\rho} \le \exp\left(2\tilde{r}_o\right) \right) \\ & = \mathbb{P}\left(\exp\left(2\tilde{r}_u\right)(1-\rho) \le 1+\rho \le \exp\left(2\tilde{r}_o\right)(1-\rho) \right) \\ & = \mathbb{P}\left(\exp\left(2\tilde{r}_u\right) - \exp\left(2\tilde{r}_u\right)\rho \le 1+\rho \le \exp\left(2\tilde{r}_o\right)-\exp\left(2\tilde{r}_o\right)\rho \right) \\ & = \mathbb{P}\left(\exp\left(2\tilde{r}_u\right) - \exp\left(2\tilde{r}_u\right)\rho - 1 \le \rho \le \exp\left(2\tilde{r}_o\right)-\exp\left(2\tilde{r}_o\right)\rho - 1 \right). \\ \end{split} \end{align}\] However, it still applies \[\begin{align} \begin{split} \exp\left(2\tilde{r}_u\right) - \exp\left(2\tilde{r}_u\right)\rho - 1 & \le \rho \\\Leftrightarrow \exp\left(2\tilde{r}_u\right) - 1 & \le \exp\left(2\tilde{r}_u\right)\rho + \rho \\\Leftrightarrow \exp\left(2\tilde{r}_u\right) - 1 & \le \left(\exp\left(2\tilde{r}_u\right) +1 \right)\rho \\\Leftrightarrow \frac{\exp\left(2\tilde{r}_u\right) - 1}{\exp\left(2\tilde{r}_u\right) +1} & \le \rho, \end{split} \end{align}\] so \[\begin{equation} \delta = \mathbb{P}\left(\frac{\exp\left(2\tilde{r}_u\right) - 1}{\exp\left(2\tilde{r}_u\right) +1} \le \rho \le \exp\left(2\tilde{r}_o\right)-\exp\left(2\tilde{r}_o\right)\rho - 1\right). \end{equation}\]
The same applies \[\begin{align} \begin{split} \rho & \le \exp\left(2\tilde{r}_o\right)-\exp\left(2\tilde{r}_o\right)\rho - 1 \\\Leftrightarrow \exp\left(2\tilde{r}_o\right)\rho + \rho & \le \exp\left(2\tilde{r}_o\right) \\\Leftrightarrow \left(\exp\left(2\tilde{r}_o\right) + 1\right)\rho & \le \exp\left(2\tilde{r}_o\right) - 1 \\\Leftrightarrow \rho & \le \frac{\exp\left(2\tilde{r}_o\right) - 1}{\exp\left(2\tilde{r}_o\right) +1} \end{split} \end{align}\] and so finally \[\begin{align} \begin{split} \delta & = \mathbb{P}\left(\frac{\exp\left(2\tilde{r}_u\right) - 1}{\exp\left(2\tilde{r}_u\right) +1 } \le \rho \le \frac{\exp\left(2\tilde{r}_o\right) - 1}{\exp\left(2\tilde{r}_o\right) +1} \right) \\ & = \mathbb{P} \left( \left[ \frac{\exp(2\tilde{r}_u)- 1}{\exp(2\tilde{r}_u) + 1}, \frac{\exp(2\tilde{r}_o)- 1}{\exp(2\tilde{r}_o) + 1} \right] \ni \rho \right) \\ & = \mathbb{P}_\rho\left(\kappa(r) \ni \rho \right). \end{split} \end{align}\]
Example
We demonstrate Theorem 50.7 and Theorem 50.8 using a simulation example. The same as above for \(i = 1,...,30\) \[\begin{equation} p(t_i) := N(t_i; 0,\sigma_\tau^2) \mbox{ and } p(y_{ij}|t_i) := N(y_{ij}; t_i,\sigma_{\varepsilon}^2) \end{equation}\] for \(j = 1,2\) with \(\sigma_\tau^2:= 1.0\) and \(\sigma_{\varepsilon}^2:= 0.2\). The following R code generates \(10^5\) data sets of this model and evaluates the corresponding Fisher-transformed sample correlations and confidence intervals for a confidence level of \(\delta = 0.95\). Figure 50.4 A shows the comparison between the empirically generated frequentist distribution of the Fisher-transformed sample correlations and Figure 50.4 B shows the first 100 simulated confidence intervals of the correlation. In the present simulation, compared to its analytical approximation, there is empirical evidence of a slight shift in the Fisher-transformed sample correlations to higher values. The confidence intervals considered do not cover the true, but unknown, correlation in two out of 100 cases.
set.seed(0) # reproducibility
nsim = 1e5 # number of realizations
n = 30 # number of persons
m = 2 # number of test measurements
sigsqr_tau = 1 # true-score variance
sigsqr_eps = .2 # observed-score variance
R = sigsqr_tau/(sigsqr_tau+sigsqr_eps) # parallel-test reliability
delta = 0.95 # confidence level
z_delta = qnorm((1+delta)/2) # confidence-interval scaling parameter
r = rep(NaN, nsim) # parallel-test reliability estimator array
r_til = rep(NaN, nsim) # Fisher transformation of r
r_til_u = rep(NaN, nsim) # lower confidence-interval parameter
r_til_o = rep(NaN, nsim) # upper confidence-interval parameter
kappa = matrix(rep(NaN,2*nsim), ncol= 2) # confidence-interval array
for(s in 1:nsim){ # simulation iterations
T = matrix(rep(NaN, n) , nrow = n) # true-score array
Y = matrix(rep(NaN, n*m), nrow = n) # observed-score array
E = matrix(rep(NaN, n*m), nrow = n) # measurement-error array
for(i in 1:n){ # person iterations
T[i] = rnorm(1,1,sqrt(sigsqr_tau)) # true score realization for j = 1.2
for(j in 1:m){ # test-measurement iterations
Y[i,j] = rnorm(1,T[i],sqrt(sigsqr_eps)) # observed-score realization
E[i,j] = Y[i,j] - T[i]}} # measurement-error realization
r[s] = cor(Y[,1],Y[,2]) # parallel test reliability estimator
r_til[s] = 1/2*log((1+r[s])/(1-r[s])) # Fisher transformation of r
r_til_u[s] = r_til[s] - z_delta*(1/sqrt(n-3)) # lower confidence-interval parameter
r_til_o[s] = r_til[s] + z_delta*(1/sqrt(n-3)) # upper confidence-interval parameter
kappa[s,1] = (exp(2*r_til_u[s])-1)/(exp(2*r_til_u[s])+1) # lower confidence-interval bound
kappa[s,2] = (exp(2*r_til_o[s])-1)/(exp(2*r_til_o[s])+1) # upper confidence-interval bound
}
50.3 Internal consistency
50.3.1 \(m\) unit test models
In the previous sections, \(y_{ij}\) denoted the random variable for modeling the observed score of the \(j\) th test measurement of the \(i\) th person, where we assumed the presence of \(m\) test measurements of \(n\) people. However, we initially left it open whether the \(j\) th test measurement meant the summative result of a test or the result of an individual test item in order to develop the theory for both applications. As a basis for determining the internal consistency of a test, in this section we now identify the \(j\) th test measurement with the \(j\) th item of a test. In the following, \(y_{ij}\) refers to the random variable for modeling the observed score of the \(j\) tenth item of the \(i\) tenth person in a test, whereby we continue to assume \(m\) items and \(n\) persons. In this context, we also assume that a total observed score is formed for each person by summing the items of a test. We denote the random variable for modeling this total observed score by \[\begin{equation} y_i := \sum_{j=1}^m y_{ij}. \end{equation}\] We summarize these preliminary considerations in the following definition of the * \(m\) unit testing model*.
Definition 50.8 (\(m\) unit testing model) The model of multiple test measurements for \(i = 1,...,n\) people and \(j = 1,...,m\) test measurements is given. For \(i = 1,...,n\) be
- \(y_i:= \sum_{j=1}^m y_{ij}\) is the sum of observed scores and * \(\tau_i:= \sum_{j=1}^m \tau_{ij}\) is the sum of true scores.
Then the joint distribution of \(y_i\) and \(\tau_i\) is called for \(i = 1,...n\) with the factorization property \[\begin{equation} \mathbb{P}(\tau_1,...,\tau_n,y_1,...,y_n) := \prod_{i=1}^n \mathbb{P}(\tau_i,y_i) = \prod_{i=1}^n \mathbb{P}(y_i|\tau_i)\mathbb{P}(\tau_i) \end{equation}\] the \(m\) unit test model, if that applies \[\begin{equation} \mathbb{P}(\tau_1,y_1) = \cdots = \mathbb{P}(\tau_n,y_n). \end{equation}\]
To understand how Cronbach’s \(\alpha\) measures the internal consistency of a test, it is first advisable to define the reliability for the \(m\) component test model in the sense of the formulation of statement (1) in Theorem 50.5.
Definition 50.9 (Reliability of \(m\) component test models) Given a \(m\) unit test model with the sum of the observed scores \(y_i\) and the sum of the true scores \(\tau_i\) for \(i = 1,...,n\) people. Then the reliability of the model is defined as \[\begin{equation} \mbox{R} := \frac{\mathbb{V}(\tau_i)}{\mathbb{V}(y_i)} \mbox{ for any } 1 \le i \le n. \end{equation}\]
50.3.2 Cronbach’s \(\alpha\)
Given the \(m\) component model, we define Cronbach’s \(\alpha\) as follows.
Definition 50.10 (Cronbach’s \(\alpha\)) Consider a \(m\) unit test model. Then that means \[\begin{equation} \alpha := \frac{m}{m-1}\left(1 - \frac{\sum_{j=1}^m \mathbb{V}(y_{ij})}{\mathbb{V}(y_i)}\right) \end{equation}\] Cronbach’s \(\alpha\) or coefficient \(\alpha\) .
Note that in Definition 50.10, \(\mathbb{V}(y_{ij})\) denotes the variances of the observed scores for people and items and \(\mathbb{V}(y_i)\) denotes the variances of the sums of the observed scores for people. Since after adopting the \(m\) unit test model, the distributions of the \(y_i\) are identical, Cronbach’s \(\alpha\) is also often used in the form \[\begin{equation} \alpha = \frac{m}{m-1}\left(1 - \frac{\sum_{j=1}^m \mathbb{V}(y_{j})}{\mathbb{V}(y)}\right) \end{equation}\] written, where \(\mathbb{V}(y_{j})\) denotes the variances of the items and \(\mathbb{V}(y)\) denotes the variance of the sums of the observed scores.
Intuitively, Cronbach’s \(\alpha\) takes a value close to \(1\) if the number of items \(m\) is high and therefore \(\frac{m}{m-1} \approx 1\) and at the same time the ratio of the summed item variances \(\sum_{j=1}^m \mathbb{V}(y_{j})\) to the variance of the sum of the item values \(\mathbb{V}(y)\) is very low and therefore \(\sum_{j=1}^m \mathbb{V}(y_{j})/\mathbb{V}(y) \approx 0\). Cronbach’s \(\alpha\) gets its central importance in classical test theory through its connection to the reliability of a \(m\) component test, which we present in the theorem below.
Theorem 50.9 (Cronbach’s \(\alpha\) and reliability) Consider a \(m\) component test model with reliability \(\mbox{R}\). Then Cronbach’s \(\alpha\) holds that \[\begin{equation} \alpha \le \mbox{R} \end{equation}\] and equality occurs especially when the test measurements of the \(m\) component test model are parallel.
Proof. To simplify the notation, we do not explicitly mark the people \(i = 1,...,n\) and put \[\begin{equation} y_{j} := y_{ij}, \tau_{j} := \tau_{ij}, y := \sum_{j=1}^m y_{j} \mbox{ and } \tau := \sum_{j=1}^m \tau_{j}, \end{equation}\] as well as \[\begin{equation} \mbox{R} := \frac{\mathbb{V}(\tau)}{\mathbb{V}(y)} \mbox{ and } \alpha := \frac{m}{m-1}\left(1 - \frac{\sum_{j=1}^m \mathbb{V}(y_{j})}{\mathbb{V}(y)}\right). \end{equation}\] We proceed in four steps.
(1) (Sum representation) We first use the following representation the sum of \(m\) numbers: for numbers \(x_1,...,x_m\) this applies \[\begin{equation} \sum_{j=1}^m x_j = \frac{1}{m-1}\sum_{j=1}^{m-1}\sum_{k = j + 1}^m (x_j + x_k). \end{equation}\] Instead of proof, let’s consider the case \(m:= 4\). Then applies \[\begin{align} \begin{split} \frac{1}{3}\sum_{j=1}^{3}\sum_{k = j + 1}^4 (x_j + x_k) & = \frac{1}{3}\left(\sum_{k = 1 + 1}^4 (x_1 + x_k) + \sum_{k = 2 + 1}^4 (x_2 + x_k) + \sum_{k = 3 + 1}^4 (x_3 + x_k)\right) \\ & = \frac{1}{3}\left(\sum_{k = 2}^4 (x_1 + x_k) + \sum_{k = 3}^4 (x_2 + x_k) +\sum_{k = 4}^4 (x_3 + x_k)\right) \\ & = \frac{1}{3}\left((x_1 + x_2) + (x_1 + x_3) + (x_1 + x_4) + (x_2 + x_3) + (x_2 + x_4) + (x_3 + x_4) \right) \\ & = \frac{1}{3}\left((x_1 + x_1 + x_1) + (x_2 + x_2 + x_2) + (x_3 + x_3 + x_3) + (x_4 + x_4 + x_4) \right) \\ & = \frac{1}{3}\left(3x_1+ 3x_2+ 3x_3 + 3x_4 \right) \\ & = x_1 + x_2 + x_3 + x_4 \\ & = \sum_{j = 1}^4 x_j \\ \end{split} \end{align}\]
(2) (True score covariance inequality) We now derive multiples in the model Test measurements produce an inequality. To do this, in this model we consider the variance of the difference between two true scores \(\tau_j\) and \(\tau_k\). With the non-negativity of the variance (Theorem 24.1) and the theorem on the variance of special linear combinations of random variables (Theorem 25.7) we get \[\begin{align} \begin{split} \mathbb{V}(\tau_j - \tau_k) \ge 0 \Leftrightarrow \mathbb{V}(\tau_j) + \mathbb{V}(\tau_k) - 2\mathbb{C}(\tau_j,\tau_k) \ge 0 \Leftrightarrow \mathbb{V}(\tau_j) + \mathbb{V}(\tau_k) \ge 2\mathbb{C}(\tau_j,\tau_k) \end{split} \end{align}\] Furthermore, for any \(1 \le j,k \le m\) in parallel test measurements it follows that \[\begin{equation} \mathbb{V}(\tau_j - \tau_k) = \mathbb{V}(f_j(\tau_1) - f_k(\tau_1)) = \mathbb{V}(\tau_1 - \tau_1) = \mathbb{V}(0) = 0. \end{equation}\] In the case of parallel test measurements, the above inequality and its application below result in equality.
(3) (Sum true score variance inequality) We now consider the Variance of the true score sum in the \(m\) component test model. With the theorem on the variance of a linear combination of random variables (Theorem 25.6), the sum representation from (1) and the true score covariance inequality from (2) we get first \[\begin{align} \begin{split} \mathbb{V}(\tau) & = \mathbb{V}\left(\sum_{j=1}^m \tau_j\right) \\ & = \sum_{j=1}^m \mathbb{V}(\tau_j) + 2\sum_{j=1}^{m-1}\sum_{k = j + 1 }^m \mathbb{C}(\tau_j, \tau_k) \\ & = \frac{1}{m-1}\sum_{j=1}^{m-1}\sum_{k = j + 1 }^m \left(\mathbb{V}(\tau_j) + \mathbb{V}(\tau_k)\right) + 2\sum_{j=1}^{m-1}\sum_{k = j + 1 }^m \mathbb{C}(\tau_j, \tau_k) \\ & \ge \frac{1}{m-1}\sum_{j=1}^{m-1}\sum_{k = j + 1 }^m 2 \mathbb{C}(\tau_j, \tau_k) + \sum_{j=1}^{m-1}\sum_{k = j + 1 }^m 2\mathbb{C}(\tau_j, \tau_k) \\ & = \left(\frac{1}{m-1} + 1 \right) \sum_{j=1}^{m-1}\sum_{k = j + 1 }^m \mathbb{C}(\tau_j, \tau_k) \\ & = \left(\frac{1}{m-1} + \frac{m-1}{m-1} \right) \sum_{j=1}^{m-1}\sum_{k = j + 1 }^m 2\mathbb{C}(\tau_j, \tau_k) \\ & = \frac{1 + m - 1}{m-1} \sum_{j=1}^{m-1}\sum_{k = j + 1 }^m 2\mathbb{C}(\tau_j, \tau_k) \\ & = \frac{m}{m-1} \sum_{j=1}^{m-1}\sum_{k = j + 1 }^m 2\mathbb{C}(\tau_j, \tau_k) \\ \end{split} \end{align}\] With statement (5) of Theorem 50.3 and again with Theorem 25.6 then applies \[\begin{align} \begin{split} \mathbb{V}(\tau) & \ge \frac{m}{m-1} \sum_{j=1}^{m-1}\sum_{k = j + 1 }^m 2\mathbb{C}(\tau_j, \tau_k) \\ & = \frac{m}{m-1} \sum_{j=1}^{m-1}\sum_{k = j + 1 }^m 2\mathbb{C}(y_j, y_k) \\ & = \frac{m}{m-1} \left(\mathbb{V}\left(\sum_{j=1}^m y_j\right) - \sum_{j=1}^m \mathbb{V}\left(y_j\right) \right)\\ & = \frac{m}{m-1} \left(\mathbb{V}(y) - \sum_{j=1}^m \mathbb{V}\left(y_j\right) \right).\\ \end{split} \end{align}\] (4) (Reliability) Finally, we consider reliability in the \(m\) component testing model. It turns out \[\begin{align} \begin{split} \mbox{R} & = \frac{\mathbb{V}(\tau)}{\mathbb{V}(y)} \\ & \ge \frac{\frac{m}{m-1} \left(\mathbb{V}(y) - \sum_{j=1}^m \mathbb{V}\left(y_j\right)\right)}{\mathbb{V}(y)} \\ & = \frac{m}{m-1} \left(\frac{\mathbb{V}(y)}{\mathbb{V}(y)} - \frac{\sum_{j=1}^m \mathbb{V}\left(y_j\right)}{\mathbb{V}(y)}\right) \\ & = \frac{m}{m-1} \left(1 - \frac{\sum_{j=1}^m \mathbb{V}\left(y_j\right)}{\mathbb{V}(y)}\right) \\ & =: \alpha. \end{split} \end{align}\]
Cronbach’s \(\alpha\) is therefore a lower bound for the reliability of a \(m\) component test model, so the reliability of a \(m\) component test model is at least as large as Cronbach’s \(\alpha\), but can be larger. For parallel test measurements, i.e. parallel items, the reliability of a \(m\) component test model is even equal to \(\alpha\).
50.3.3 Estimation of Cronbach’s \(\alpha\)
The estimation of Cronbach’s \(\alpha\) uses the sample variances of the observed item scores and the observed test sum scores. For data from people \(i = 1,...,n\) an estimator for the variance of the \(j\) th item score is given \[\begin{equation} S^2_j := \frac{1}{n-1}\sum_{i=1}^n \left(y_{ij} - \bar{y}_j\right)^2 \mbox{ with } \bar{y}_j := \frac{1}{n}\sum_{i=1}^n y_{ij} \end{equation}\] and an estimator for the variance of the test sum scores \[\begin{equation} S^2 := \frac{1}{n-1}\sum_{i=1}^n \left(y_i - \bar{y}\right)^2 \mbox{ with } \bar{y} := \frac{1}{n}\sum_{i=1}^n y_i \end{equation}\] given. An estimate for \(\alpha\) results accordingly \[\begin{equation} \hat{\alpha} = \frac{m}{m-1}\left(1 - \frac{\sum_{j=1}^m S^2_j}{S^2}\right). \end{equation}\] We demonstrate the evaluation of this estimator below using a simulation in the model of parallel test measurements for \(n:= 30\) and \(m:= 21\) \[\begin{equation} \mathbb{P}(\tau_{i}) := N(1,1) \mbox{ and } \mathbb{P}(y_{ij}|\tau_{i}) := N(\tau_{i},4) \mbox{ for } i = 1,...,n, j = 1,...,m. \end{equation}\]
library(psych) # r package for test analysis
set.seed(0) # reproducibility
n = 30 # number of persons
m = 21 # number of items
mu = 1 # true-score expectation parameter
T = matrix(rep(NaN, n) , nrow = n) # true-score array
Y = matrix(rep(NaN, n*m), nrow = n) # observed-score array
for(i in 1:n){ # iteration over persons
T[i] = rnorm(1,mu,sqrt(1)) # true-score realization
for(j in 1:m){ # iteration over items
Y[i,j] = rnorm(1,T[i],sqrt(4))}} # observed-score realization
vsi = var(apply(Y,1,sum)) # sample variance of observed score sums
siv = sum(apply(Y,2,var)) # sum of item sample variances
a = (m/(m-1))*(1-(siv/vsi)) # direct calculation of Cronbach's alpha
ap = alpha(Y,warnings = FALSE) # calculating Cronbach's alpha with psychCronbach's alpha (manual) : 0.866
Cronbach's alpha (psych) : 0.866The frequentist distribution and inference theory for this estimator has been elaborated by Kristof (1963), Feldt (1965), Feldt et al. (1987), Van Zyl et al. (2000) and Yuan & Bentler (2002).
Borsboom, D. (2009). Measuring the mind: Conceptual issues in contemporary psychometrics. Cambridge University Press.
Borsboom, D., Mellenbergh, G. J., & Van Heerden, J. (2004). The Concept of Validity. Psychological Review, 111(4), 1061–1071. https://doi.org/10.1037/0033-295X.111.4.1061
Feldt, L. S. (1965). The Approximate Sampling Distribution of Kuder-Richardson Reliability Coefficient Twenty. Psychometrika, 30(3), 357–370. https://doi.org/10.1007/BF02289499
Feldt, L. S., Woodruff, D. J., & Salih, F. A. (1987). Statistical Inference for Coefficient Alpha. Applied Psychological Measurement, 11(1), 93–103. https://doi.org/10.1177/014662168701100107
Johnson, N. L., Kotz, S., & Balakrishnan, N. (1994). Continuous univariate distributions, Volume 2 (2nd ed). Wiley.
Krauth, J. (1995). Testkonstruktion und Testtheorie. Beltz, Psychologie Verl.-Union.
Kristof, W. (1963). The statistical theory of stepped-up reliability coefficients when a test has been divided into several equivalent parts. Psychometrika, 28(3), 221–2238. https://doi.org/10.1007/BF02289571
Lazarsfeld, P. (1959). Latent structure analysis. In Psychology: A study of a science, Vol. 3. McGraw-Hill.
Lord, F. M., & Novick, M. R. (1968). Statistical theories of mental test scores (Nachdr. der Ausg. Reading, Mass. [u.a.], 1968). Information Age Publ.
Novick, M. R. (1966). The axioms and principal results of classical test theory. Journal of Mathematical Psychology, 3(1), 1–18. https://doi.org/10.1016/0022-2496(66)90002-2
Van Zyl, J. M., Neudecker, H., & Nel, D. G. (2000). On the Distribution of the Maximum Likelihood Estimator of Cronbach’s Alpha. Psychometrika, 65(3), 271–280. https://doi.org/10.1007/BF02296146
Yuan, K.-H., & Bentler, P. M. (2002). On Robusiness of the Normal-Theory Based Asymptotic Distributions of Three Reliability Coefficient Estimates. Psychometrika, 67(2), 251–259. https://doi.org/10.1007/BF02294845