Probability theory
Probability theory is a mathematical model for describing and quantitatively reasoning about random processes of reality (Figure 1). By random processes we mean all phenomena that cannot be predicted by us with absolute certainty, and whose outcome is therefore subject to uncertainty. Obvious and familiar examples of random processes are throwing a die or tossing a coin. However, the concept of a random process, and thus the domain of application of probability theory, should be understood much more broadly. For example, the outcome of an election, tomorrow’s weather, the measurement value of an electroencephalography electrode at a particular point in time, or the effect of a psychotherapy intervention on the health status of a patient cannot be predicted with complete certainty and are therefore subject to uncertainty. Once one begins to think about which phenomena of reality are subject to uncertainty, it becomes difficult to name nontrivial phenomena about whose outcome one has complete certainty.
As a mathematical model of random processes, probability theory allows reasoned quantitative conclusions about random processes. This is reflected primarily in probability calculus. Quantitative conclusions in probability calculus have, for example, the following form: if we assume that event \(A\) occurs with probability \(\mathbb{P}(A)\) and event \(B\) with probability \(\mathbb{P}(B)\), then an event \(C\) has probability \(\mathbb{P}(C)\). The conclusion about the probability of \(C\) is secured logically and mathematically in the same sense as, for example, \(1+1=2\). Probability theory itself does not state whether the assumptions about the probabilities of \(A\) and \(B\) correspond to the random process in reality.
Probability theory itself uses the mathematical theory of sets and functions. At least since Kolmogoroff (1933), an axiomatic approach has prevailed: in probability theory itself, one does not ask what a probability is or to what extent the predictions of probability theory agree with reality, but instead tries to develop a self-consistent formal-mathematical system of ungrounded but intuitively plausible basic assumptions and their consequences. The starting point of this development is the probability space model of a random process, which will be introduced in 19 Probability spaces. Indeed, beyond the formal-mathematical system of probability theory, mathematical-philosophical discussions continue to this day about exactly what is to be understood by the term “probability of an event”. Roughly speaking, two somewhat opposing interpretations are predominant: on the one hand the so-called frequentist interpretation as a prototypical example of realist interpretations, and on the other hand the so-called Bayesian interpretation as a general form of so-called evidential interpretations. For a more detailed overview of different interpretations of the concept of probability, see Table 1 after Hájek (2019).
Realist interpretations regard probabilities as properties of the world, such as relative frequencies or physical propensities, that exist independently of our beliefs. The interpretations grouped here as evidential relate probabilities to knowledge, information, or beliefs. In this context, Hájek (2019) distinguishes the degree to which available evidence supports a proposition from a person’s degree of belief.
| Interpretation | Intuition | Field of application | Problems | References |
|---|---|---|---|---|
| Finite frequentist | The probability that a coin shows heads corresponds to the relative frequency with which it shows heads in a finite number of observations. | – | A single coin toss does not yield a meaningful prediction. Probabilities change with the observation series and are therefore not uniquely defined. | Venn (1876) |
| Infinite frequentist | The probability that a coin shows heads corresponds to the relative frequency with which it shows heads in infinitely many observations. | Frequentist inference |
Infinite observation series are not empirically available. From finitely many observations, the probability interpreted as a limit can only be estimated. | Reichenbach (1949) von Mises (1951) |
| Propensity | The probability that a coin shows heads corresponds to its inherent physical propensity to land heads up in the Earth’s gravitational field. | Causal inference | The connection to relative frequencies or physical theories is not immediately clear. | Peirce (1957) Popper (1959) Gillies (2000) |
| Classical | In the absence of further information, the probabilities that a coin shows heads or tails are equal. | Games of chance, textbooks | Restricted to finite outcome spaces | Laplace (1814) Jaynes (1968) |
| Subjective | The probability that a coin shows heads corresponds to the subjective degree of uncertainty that an observer assigns to the outcome of a coin toss. | Bayesian inference |
Coherence alone does not uniquely determine probabilities. Different people can hold different coherent assessments. | De Finetti (1975) |
| Epistemic | The probability that a coin shows heads corresponds to the subjective degree of uncertainty that a rational observer assigns to the outcome of a coin toss. | Bayesian inference |
The connection to relative frequencies or physical theories is not immediately clear. | Ramsey (1926) Jeffreys (1939) Savage (1954) |
According to the frequentist interpretation, the probability of an event is the idealized relative frequency with which an event tends to occur under the same external conditions. For example, the frequentist interpretation of the statement “With a probability of 1/6, the die will show a 2 on the next throw” is the following: “If one were to throw a die infinitely often and determine the relative frequency of the event that the die shows a 2, then this relative frequency would be equal to 1/6.” Note that, in this interpretation, one de facto cannot empirically determine the probability of an event, since one cannot throw a die infinitely often. Of course, in this interpretation one can estimate the probability empirically. Estimation procedures themselves, however, are not part of probability theory, but of inference.
According to the Bayesian interpretation, the probability of an event is the degree of certainty that an observer assigns to the occurrence of the event based on her subjective assessment of the situation. For example, the Bayesian interpretation of the statement “With a probability of 1/6, the die will show a two on the next throw” would then be something like the following: “Based on my own and transmitted experience with throwing a die, I am 16.6% certain that the die will show a two on the next throw.”
In models of random processes that can in fact be repeated at least under similar circumstances, such as throwing a die, the difference between the frequentist and Bayesian interpretations is often rather subtle. However, as indicated above, there are many random processes that can be described with probabilities for which, because of their uniqueness, a frequentist interpretation is not appropriate. For example, statements of the form “The probability that the global heat records in 2023 are not attributable to climate change is less than 0.01” (cf. Philip et al. (2020)) make sense only under the Bayesian interpretation, because the weather records of the year 2023 are a unique, non-repeatable event.
Different interpretations of probability therefore share the same formal definitions and calculation rules. Frequentist and Bayesian inference have a common mathematical foundation in probability theory. This resembles other forms of mathematical modeling. For example, a function’s derivative can represent an object’s velocity or a bacterial population’s growth rate. The real-world phenomena differ, but the underlying mathematical reasoning is the same.