Bayes not Bust! Why Simplicity is no Problem for Bayesians
- 17 September 2007
- journal article
- research article
- Published by University of Chicago Press in The British Journal for the Philosophy of Science
- Vol. 58 (4), 709-754
- https://doi.org/10.1093/bjps/axm033
Abstract
The advent of formal definitions of the simplicity of a theory has important implications for model selection. But what is the best way to define simplicity? Forster and Sober ([ 1994 ]) advocate the use of Akaike's Information Criterion (AIC), a non-Bayesian formalisation of the notion of simplicity. This forms an important part of their wider attack on Bayesianism in the philosophy of science. We defend a Bayesian alternative: the simplicity of a theory is to be characterised in terms of Wallace's Minimum Message Length (MML). We show that AIC is inadequate for many statistical problems where MML performs well. Whereas MML is always defined, AIC can be undefined. Whereas MML is not known ever to be statistically inconsistent, AIC can be. Even when defined and consistent, AIC performs worse than MML on small sample sizes. MML is statistically invariant under 1-to-1 re-parametrisation, thus avoiding a common criticism of Bayesian approaches. We also show that MML provides answers to many of Forster's objections to Bayesianism. Hence an important part of the attack on Bayesianism fails.Keywords
This publication has 30 references indexed in Scilit:
- Predictive Accuracy as an Achievable Goal of SciencePhilosophy of Science, 2002
- The Complexity of Strict Minimum Message Length InferenceThe Computer Journal, 2002
- Key Concepts in Model Selection: Performance and GeneralizabilityJournal of Mathematical Psychology, 2000
- Model Selection in Science: The Problem of Language VarianceThe British Journal for the Philosophy of Science, 1999
- Refinements of MDL and MML CodingThe Computer Journal, 1999
- Bayes and Bust: Simplicity as a Problem for a Probabilist's Approach to ConfirmationThe British Journal for the Philosophy of Science, 1995
- Estimation of the Concentration Parameter in von Mises-Fisher DistributionsBiometrika, 1978
- Estimating the Dimension of a ModelThe Annals of Statistics, 1978
- An information measure for hierarchic classificationThe Computer Journal, 1973
- On Information and SufficiencyThe Annals of Mathematical Statistics, 1951