Probability theory

Probability theory is a mathematical model for describing and quantitatively reasoning about random processes of reality (Figure 1). By random processes we mean all phenomena that cannot be predicted by us with absolute certainty, and whose outcome is therefore subject to uncertainty. Obvious and familiar examples of random processes are throwing a die or tossing a coin. However, the concept of a random process, and thus the domain of application of probability theory, should be understood much more broadly. For example, the outcome of an election, tomorrow’s weather, the measurement value of an electroencephalography electrode at a particular point in time, or the effect of a psychotherapy intervention on the health status of a patient cannot be predicted with complete certainty and are therefore subject to uncertainty. Once one begins to think about which phenomena of reality are subject to uncertainty, it becomes difficult to name nontrivial phenomena about whose outcome one has complete certainty.

Figure 1: Probability theory as a model of random processes. The starting point of probability theory is the intention to draw logical-quantitative conclusions about a random process, that is, a phenomenon of reality subject to uncertainty. The representation of central aspects of the random process with the help of probability-theoretic terminology is called modeling. The probability-theoretic model itself then guarantees, in the sense of probability calculus, the correctness of logical-quantitative conclusions that can be used to predict aspects of the random process.

As a mathematical model of random processes, probability theory allows reasoned quantitative conclusions about random processes. This is reflected primarily in probability calculus. Quantitative conclusions in probability calculus have, for example, the following form: if we assume that event \(A\) occurs with probability \(\mathbb{P}(A)\) and event \(B\) with probability \(\mathbb{P}(B)\), then an event \(C\) has probability \(\mathbb{P}(C)\). The conclusion about the probability of \(C\) is secured logically and mathematically in the same sense as, for example, \(1+1=2\). Probability theory itself does not state whether the assumptions about the probabilities of \(A\) and \(B\) correspond to the random process in reality.

Probability theory itself uses the mathematical theory of sets and functions. At least since Kolmogoroff (1933), an axiomatic approach has prevailed: in probability theory itself, one does not ask what a probability is or to what extent the predictions of probability theory agree with reality, but instead tries to develop a self-consistent formal-mathematical system of ungrounded but intuitively plausible basic assumptions and their consequences. The starting point of this development is the probability space model of a random process, which will be introduced in 19  Probability spaces. Indeed, beyond the formal-mathematical system of probability theory, mathematical-philosophical discussions continue to this day about exactly what is to be understood by the term “probability of an event”. Roughly speaking, two somewhat opposing interpretations are predominant: on the one hand the so-called frequentist interpretation as a prototypical example of realist interpretations, and on the other hand the so-called Bayesian interpretation as a general form of so-called evidential interpretations. For a more detailed overview of different interpretations of the concept of probability, see Table 1 after Hájek (2019).

Realist interpretations regard probabilities as properties of the world, such as relative frequencies or physical propensities, that exist independently of our beliefs. The interpretations grouped here as evidential relate probabilities to knowledge, information, or beliefs. In this context, Hájek (2019) distinguishes the degree to which available evidence supports a proposition from a person’s degree of belief.

Interpretation Intuition Field of application Problems References
Finite frequentist The probability that a coin shows heads corresponds to the relative frequency with which it shows heads in a finite number of observations. – A single coin toss does not yield a meaningful prediction. Probabilities change with the observation series and are therefore not uniquely defined. Venn (1876)
Infinite frequentist The probability that a coin shows heads corresponds to the relative frequency with which it shows heads in infinitely many observations. Frequentist
inference
Infinite observation series are not empirically available. From finitely many observations, the probability interpreted as a limit can only be estimated. Reichenbach (1949) von Mises (1951)
Propensity The probability that a coin shows heads corresponds to its inherent physical propensity to land heads up in the Earth’s gravitational field. Causal inference The connection to relative frequencies or physical theories is not immediately clear. Peirce (1957) Popper (1959) Gillies (2000)
Classical In the absence of further information, the probabilities that a coin shows heads or tails are equal. Games of chance, textbooks Restricted to finite outcome spaces Laplace (1814) Jaynes (1968)
Subjective The probability that a coin shows heads corresponds to the subjective degree of uncertainty that an observer assigns to the outcome of a coin toss. Bayesian
inference
Coherence alone does not uniquely determine probabilities. Different people can hold different coherent assessments. De Finetti (1975)
Epistemic The probability that a coin shows heads corresponds to the subjective degree of uncertainty that a rational observer assigns to the outcome of a coin toss. Bayesian
inference
The connection to relative frequencies or physical theories is not immediately clear. Ramsey (1926) Jeffreys (1939) Savage (1954)
Table 1: Interpretations of probability after Hájek (2019). The frequentist interpretation corresponds in its essential features to the infinite frequentist realist interpretation, and the Bayesian interpretation to the subjective evidential and epistemic evidential interpretations.

According to the frequentist interpretation, the probability of an event is the idealized relative frequency with which an event tends to occur under the same external conditions. For example, the frequentist interpretation of the statement “With a probability of 1/6, the die will show a 2 on the next throw” is the following: “If one were to throw a die infinitely often and determine the relative frequency of the event that the die shows a 2, then this relative frequency would be equal to 1/6.” Note that, in this interpretation, one de facto cannot empirically determine the probability of an event, since one cannot throw a die infinitely often. Of course, in this interpretation one can estimate the probability empirically. Estimation procedures themselves, however, are not part of probability theory, but of inference.

According to the Bayesian interpretation, the probability of an event is the degree of certainty that an observer assigns to the occurrence of the event based on her subjective assessment of the situation. For example, the Bayesian interpretation of the statement “With a probability of 1/6, the die will show a two on the next throw” would then be something like the following: “Based on my own and transmitted experience with throwing a die, I am 16.6% certain that the die will show a two on the next throw.”

In models of random processes that can in fact be repeated at least under similar circumstances, such as throwing a die, the difference between the frequentist and Bayesian interpretations is often rather subtle. However, as indicated above, there are many random processes that can be described with probabilities for which, because of their uniqueness, a frequentist interpretation is not appropriate. For example, statements of the form “The probability that the global heat records in 2023 are not attributable to climate change is less than 0.01” (cf. Philip et al. (2020)) make sense only under the Bayesian interpretation, because the weather records of the year 2023 are a unique, non-repeatable event.

Different interpretations of probability therefore share the same formal definitions and calculation rules. Frequentist and Bayesian inference have a common mathematical foundation in probability theory. This resembles other forms of mathematical modeling. For example, a function’s derivative can represent an object’s velocity or a bacterial population’s growth rate. The real-world phenomena differ, but the underlying mathematical reasoning is the same.

De Finetti, B. (1975). Theory of probability. John Wiley & Sons.
Gillies, D. (2000). Varieties of Propensity. The British Journal for the Philosophy of Science, 51(4), 807–835. https://doi.org/10.1093/bjps/51.4.807
Hájek, A. (2019). Interpretations of probability. In E. N. Zalta (Ed.), The Stanford encyclopedia of philosophy (Fall 2019). Metaphysics Research Lab, Stanford University.
Jaynes, E. (1968). Prior Probabilities. IEEE Transactions on Systems Science and Cybernetics, 4(3), 227–241. https://doi.org/10.1109/TSSC.1968.300117
Jeffreys, H. (1939). Theory Of Probability. Oxford Classic Texts in the Physical Sciences.
Kolmogoroff, A. (1933). Grundbegriffe der Wahrscheinlichkeitsrechnung. Springer Berlin Heidelberg. https://doi.org/10.1007/978-3-642-49888-6
Laplace, P. S. (1814). A philosophical essay on probabilities.
Peirce, C. S. (1957). Notes on the Doctrine of Chances. In Essays in the Philosophy of Science (pp. 74–84). Bobbs-Merrill.
Philip, S., Kew, S., Van Oldenborgh, G. J., Otto, F., Vautard, R., Van Der Wiel, K., King, A., Lott, F., Arrighi, J., Singh, R., & Van Aalst, M. (2020). A protocol for probabilistic extreme event attribution analyses. Advances in Statistical Climatology, Meteorology and Oceanography, 6(2), 177–203. https://doi.org/10.5194/ascmo-6-177-2020
Popper, K. R. (1959). The propensity interpreation of probability. The British Journal for the Philosophy of Science, 10(37), 25–42. https://doi.org/10.1093/bjps/X.37.25
Ramsey, F. P. (1926). Truth and Probability. In The Foundations of Mathematics and other Logical Essays, Chp VII (pp. 156–198). Harcourt, Brace and Company.
Reichenbach, H. (1949). Philosophical Foundations of Probability. Proceedings of the Berkeley Symposium on Mathematical Statistics and Probability, 1–20.
Savage, L. J. (1954). The foundations of statistics (pp. xv, 294). John Wiley & Sons.
Venn, J. (1876). The logic of chance. MacMillan.
von Mises, R. (1951). Wahrscheinlichkeit, Statistik und Wahrheit. Springer International Publishing. https://doi.org/10.1007/978-3-319-29251-9_9