49 Psychological tests
Tests and questionnaires are quite different in everyday experience. The term test probably most readily evokes smaller examinations with correct answers and an assessment of performance. The term questionnaire probably evokes more or less interesting collections of questions for which one may be asked to respond truthfully, but for which it is also clear that there are no right or wrong answers. Although everyday experience thus associates tests and questionnaires with rather different procedures, psychological test theory does not fundamentally distinguish between them. Historically, the analysis of test data was certainly first shaped by the analysis of performance-test data; however, the concepts of test theory developed there are now used in the same way to analyze clinical questionnaire data. In this section, we first give a brief overview of the concept of a psychological test, list the so-called test quality criteria, introduce some common norm transformations of raw test data, and finally briefly sketch the three main test theories.
49.1 On the concept of a psychological test
To approach the concept of a test, we first consider the definitions of a test given by Moosbrugger & Kelava (2012), Bühner (2010), and Krauth (1995).
According to Moosbrugger & Kelava (2012), a psychological test is “a scientific routine procedure for assessing one or more empirically distinguishable psychological characteristics with the aim of making the most precise possible quantitative statement about the degree of an individual’s characteristic expression.”
According to Bühner (2010), a “psychometric test is a scientific routine procedure for examining one or more empirically distinguishable personality characteristics (cf. Lienert & Raatz (1998), p. 1). The goal of a psychometric test is to measure the absolute or relative expression of a trait, an ability, or a state in one or more persons, or to make a qualitative statement about the class of persons to which individuals can be assigned (cf. Rost (2004)). Psychometric tests are developed according to classical or probabilistic test theory, are theoretically grounded, and satisfy precisely defined quality criteria (primary and secondary criteria).”
Finally, according to Krauth (1995), “a psychological test consists of a set of stimuli with the associated admissible responses, that is, of manifest variables. A scale assigns the response patterns of the manifest variables to the expressions of one or more latent variables. A test therefore functions as a measurement instrument for assessing not directly observable (latent) variables, whose existence is postulated for persons and occasionally also for animals.”
According to these definitions, psychological tests share the fact that they are procedures for measuring psychological characteristics and that they are subject to scientific quality assurance.
In general, psychological tests can be divided into several categories:
- Performance tests. These include developmental tests, intelligence tests, general performance tests, school tests, and special functional-assessment and aptitude tests.
- Psychometric personality tests. These include clinical tests, personality structure tests, attitude tests, and interest tests.
- Personality-development/projective procedures. These include form interpretation procedures, verbal-thematic procedures, and drawing and design procedures.
In the context of clinical psychology and psychotherapy, clinical tests are certainly of predominant interest. Examples of clinical tests include
- BDI-II (Beck Depression Inventory II, Beck et al. (1996), Hautzinger et al. (2006)), a clinical test for measuring the severity of depression. It is based on the diagnostic criteria for major depression according to DSM-IV and asks about typical depressive symptoms.
- BSI (Brief Symptom Inventory, Derogatis & Melisaratos (1983)), a clinical test that assesses psychological distress and symptoms. The BSI covers nine symptom dimensions: somatization, obsessive-compulsiveness, interpersonal sensitivity, depression, anxiety, hostility, phobic anxiety, paranoid ideation, and psychoticism.
- BSL-23 (Borderline Symptom List 23, Bohus et al. (2009)), a clinical test for assessing the severity of borderline-specific symptoms. It measures typical emotional, cognitive, and behavioral patterns in borderline personality disorder.
Psychological tests usually consist of items, which function as the basic building blocks of the test. In general, items are stimuli to which a response is expected and recorded. The standard example of an item is a question with corresponding response options. For example, the BDI-II comprises 21 items for assessing the severity of depression on a scale from 0 to 3. The item on sadness reads:
Please select the statement that best describes how you have felt during the past two weeks, including today:
49.2 Test quality criteria
In scientific discourse, a number of quality requirements have developed for psychological tests; to be scientifically recognized, a test should ideally satisfy these requirements (cf. Moosbrugger & Kelava (2012)). The fundamental criteria are objectivity, reliability, and validity.
- Objectivity. A test is objective if its administration, scoring, and interpretation are independent of the test administrators. The objectivity of a test procedure is usually ensured through a corresponding manualization.
- Reliability. A test is reliable if it assesses the characteristic to be measured exactly and without measurement error. Important subtypes of reliability are retest reliability, parallel-test reliability, split-half reliability, and internal consistency. Reliabilities are usually assessed by correlations. Classical test theory provides a theoretical framework for classifying and quantitatively defining retest, parallel-test, and split-half reliability and, for example, develops measures for the internal consistency of psychological tests such as Cronbach’s \(\alpha\) (Cronbach (1951)).
- Validity. A test is valid if it actually measures what it is supposed to measure. Content validity, face validity, criterion validity, construct validity, and, in the context of test development, factorial validity are distinguished. Content and face validity are usually supported by expert judgments; criterion validity is supported by correlation with a suitable external criterion, for example a clinical diagnosis. Establishing a test’s construct validity is a far-reaching requirement that is probably rarely satisfied (cf. Borsboom et al. (2004)). Evidence for factorial validity, finally, often appears in test manuals and refers to the results of factor analyses.
In addition to the quality criteria above, a scientifically recognized test procedure ideally also satisfies the following so-called secondary quality criteria:
- Scaling. The test scores should adequately represent the qualitative relations among characteristics in the sense of the representational theory of measurement.
- Norming. The reference system makes it possible to compare test scores with a norm sample.
- Test economy. A test should be scorable with as little time and resource expenditure as possible. Digital response collection and automated scoring with computer software would be a first step in this direction.
- Usefulness. The test should have practical relevance and provide a positive benefit.
- Acceptability. The burden on the tested person should be in an appropriate relation to the benefit.
- Resistance to faking. The test result should not be distortable by deliberate behavior. This is certainly not the case for most clinical procedures that rely on self-assessment.
- Fairness. The test must not cause systematic disadvantage to particular groups. In the clinical context, this criterion is probably of rather secondary importance.
49.3 Norm scores
The primary quantitative result of a psychological test, or of a single item of a test, is initially a raw score. For example, the possible raw scores when answering the BDI-II sadness item listed above are the numbers 0, 1, 2, and 3. The raw score of a test result across several items is often obtained as the sum of the raw scores of the individual items, as a so-called sum score. The BDI-II sum scores, for example, vary between 0 and 63; scores between 0 and 13 correspond to minimal, scores between 14 and 19 to mild, scores between 20 and 28 to moderate, and scores between 29 and 63 to severe depressive symptomatology. The raw scores of the BSI and the BSL-23 are obtained by summing the individual item scores, with possible values 0, 1, 2, 3, 4, and dividing by the number of answered items; this yields a mean score in the range from 0 to 4.
To ensure some degree of comparability between the score values of different clinical test procedures and to classify an individual test result against the background of the usual distributions of test results, test results are often normed (cf. De Beurs et al. (2022)). For this purpose, it is first necessary to collect test results from a sufficiently large sample of healthy or clinically conspicuous participants. In the following, these test results are denoted as test raw scores \(y^{(i)}\) for \(i = 1,...,n\), where \(n\) is the sample size. The results of such large-scale studies are usually documented in the corresponding test manuals. If one then assumes that the test results in the corresponding sample follow a normal distribution whose expectation parameter can be estimated by the corresponding sample mean and whose variance parameter can be estimated by the corresponding sample variance, then, based on the theorem on linear transformations of normally distributed random variables, a number of options for standardizing the test raw scores are obtained. In this context, one usually encounters Z scores and T scores, less often stanines or stens.
To understand the meaning of these standardized test scores, we first recall the distribution of the probability mass of a normally distributed random variable. As is well known (cf. Figure 49.1, after Seashore (1955) and De Beurs et al. (2022)), \[\begin{align} \begin{split} \int_{-1\sigma}^{1\sigma} N(x;\mu,\sigma^2) \,dx & = \Phi(1\sigma;\mu,\sigma^2) - \Phi(-1\sigma;\mu,\sigma^2) \approx 0.68 \\ \int_{-2\sigma}^{2\sigma} N(x;\mu,\sigma^2) \,dx & = \Phi(2\sigma;\mu,\sigma^2) - \Phi(-2\sigma;\mu,\sigma^2) \approx 0.95 \\ \int_{-3\sigma}^{3\sigma} N(x;\mu,\sigma^2) \,dx & = \Phi(3\sigma;\mu,\sigma^2) - \Phi(-3\sigma;\mu,\sigma^2) \approx 0.99 \\ \end{split} \end{align}\]
Within one standard deviation around the expectation parameter, then, 68% of the probability mass of a normally distributed random variable is located; within two standard deviations, 95%; and within three standard deviations, at 99%, almost the entire probability mass. This distribution of probability mass obviously holds independently of the concrete values of \(\mu\) and \(\sigma^2\) and therefore for all normally distributed random variables. We also recall that the theorem on the linear transformation of a normally distributed random variable states that \[\begin{equation} y \sim N(\mu,\sigma^2) \mbox{ and } z := ay + b \Rightarrow z \sim N(a\mu+b,a^2\sigma^2). \end{equation}\] For true but unknown parameters \(\mu\) and \(\sigma^2\) of normally distributed test raw scores, the theoretical standardization \[\begin{equation} z := \frac{y - \mu}{\sigma} = \frac{1}{\sigma}y - \frac{1}{\sigma}\mu \end{equation}\] therefore exactly yields \[\begin{equation} z \sim N\left(\frac{1}{\sigma}\mu - \frac{1}{\sigma}\mu, \frac{1}{\sigma^2}\sigma^2\right) = N(0,1). \end{equation}\] Because \(\mu\) and \(\sigma^2\) are unknown in practical testing situations, they are replaced by the sample mean \(m\) and the sample variance \(s^2\) of a norm sample. Thus, if one computes the so-called Z score \[\begin{equation} z^{(i)} := \frac{x^{(i)} - m}{s}, \end{equation}\] for each test raw score \(x^{(i)}\), one obtains empirically standardized values. Assuming that the norm sample represents the distribution of the test raw scores well, these values are approximately standard normally distributed and are thus comparable across different raw-score ranges of different tests (cf. Figure 49.2).
A standard normally distributed random variable has the property of taking a value between \(-3\) and \(3\) with the highest probability. This may sometimes seem impractical, because in this case an individual test result can be negative if it lies below the sample mean. Furthermore, reporting a result with sufficient precision requires decimal places. McCall (1922) therefore proposed applying a further linear transformation to the computed Z scores that transforms them into a more accessible range of values. McCall (1922) called the Z scores transformed in this way T scores in honor of Edward Lee Thorndike, Lewis Terman, and Louis Leon Thurstone. The term T scores therefore by no means goes back to the \(T\)-statistics of frequentist inference and their corresponding distributions. The transformation proposed by McCall (1922) is given by \[\begin{equation} t^{(i)} := 10z^{(i)} + 50 \end{equation}\] With the approximate standard normal distribution of \(z\), it follows that \[\begin{equation} t \sim N\left(0 + 50, 10^2 \cdot 1\right) = N(50,100). \end{equation}\] Thus T scores approximately always have an expectation of 50, a variance of 100, and a standard deviation of 10. Consequently, in the T score range from 40 to 60, about 68% of the sample results are located; in the range from 30 to 70, about 95%; and in the range from 20 to 80, about 99% of the sample results (cf. Figure 49.1).
The less frequently encountered stanine (standard nine) and sten (standard ten) scores follow the same logic in their computation as T scores, but target the value range from 1 to 9 and from 1 to 10, respectively. They are obtained from Z scores by the transformation \[\begin{equation} s^{(i)} := 2z^{(i)} + 5 \end{equation}\] and subsequent rounding to values from 1 to 9 or from 1 to 10.
49.4 Test theories
In general, three theories can be distinguished that underlie the analysis of test data: classical test theory, factor analysis, and item response theory. All three theories have in common that they are probabilistic models of observable item and test scores. They also share the fact that, in addition to modeling observable test scores by random variables, they draw on latent (not directly observable) random variables to explain observed data. In the scientific consideration of clinical-diagnostic procedures, classical test theory plays a central role with respect to the reliability and internal consistency of test procedures, and factor analysis plays a central role with respect to the factorial validity of test procedures. Item response theory currently plays only a minor role with respect to clinical-diagnostic procedures (cf. Reise & Waller (2009)). We briefly sketch these main currents of test theory in the following.
Classical test theory
Classical test theory forms the basis for evaluating the test quality criterion of reliability. At its core, classical test theory consists of a probabilistic model for item or test results of one or more individuals for one or more tests. Central aspects of classical test theory go back to the work of Charles Spearman at the beginning of the 20th century and were further established, among others, by Gulliksen (1950) and Lord & Novick (1968). The basic concept of classical test theory is a measurement-error model of the form \[\begin{equation} \mbox{Observed score} = \mbox{True score} + \mbox{Measurement error} \end{equation}\] as known in the form “\(y = \mu + \varepsilon\)” from frequentist inference theory. Within classical test theory, however, some special features arise. First, the models of classical test theory are usually not based on normal distribution assumptions, which may be due to the discrete character of the observed test results, but restrict themselves to modeling by general random variables and analyzing their expectations, variances, and covariances. Second, in contrast to frequentist inference, “true scores” are not modeled as fixed values, but also as realizations of random variables. Finally, particularly in the formulation by Lord & Novick (1968), one can still see the attempt not to presuppose “true scores” in a naively realist way when analyzing test results, but to give them a frequentist-propensity interpretation in order to make them less vulnerable from a natural-scientific perspective. Classical test theory provides, among other things, the basis for the correlations commonly reported in practice for retest reliability and for Cronbach’s \(\alpha\) as a measure of the internal consistency of a test. In the following presentation of classical test theory, we mainly follow Krauth (1995).
Factor analysis
Factor analysis is based on a probabilistic model of the covariance properties of test items and, in application, provides the basis for the test quality criterion of factorial validity. Important classical contributions to factor analysis come from Spearman (1904), Hotelling (1933), and Lawley (1940). Like classical test theory, factor analysis also models observed test results, here usually at the item level, by a latent-variable model \[\begin{equation} \mbox{Observed score} = \mbox{Factor loadings}\cdot \mbox{True factor scores} + \mbox{Measurement error}. \end{equation}\] The “true scores” of classical test theory and the “true factor scores” of factor analysis are of similar character, although factor analysis typically assumes more than a single true score. Among other reasons, because a single observed datum in factor analysis is explained by the interplay of three unobservable values (factor loading, factor score, measurement error), the unambiguous estimation of factor loadings and factor scores is not possible without further assumptions. Classically, this fact has led to one special solution being selected from a multitude of possible solutions of a factor analysis model by means of heuristic secondary criteria, a process reflected in the rotation procedures of exploratory factor analysis. However, through suitable assumptions, such as the normal distribution of true factor scores and measurement errors and the non-correlation of selected factor scores, the factor analysis model can be made more identifiable, at least in initial ways, and developed into a valid inference model. This is the topic of confirmatory factor analysis. Finally, with structural equation models (cf. Bollen (1989)), factor analysis was further generalized in the computer age and developed into an extremely flexible modeling approach. In the presentation of factor analysis, we mainly follow Rencher & Christensen (2012).
Item response theory
Item response theory (IRT) models the probability of a particular test response as a function of the ability or state of the tested person. Important contributions to IRT come from Rasch (1960), Novick (1966), and Lord (1980). IRT uses logistic regression models in its basic approach and is applied primarily to performance tests. It explicitly refers to the measurement theory of Stevens (1946), but has little to no importance in the clinical context (cf. Reise & Waller (2009)).