Stable reliability diagrams for probabilistic classifiers
Open Access
- 17 February 2021
- journal article
- research article
- Published by Proceedings of the National Academy of Sciences in Proceedings of the National Academy of Sciences of the United States of America
- Vol. 118 (8)
- https://doi.org/10.1073/pnas.2016191118
Abstract
A probability forecast or probabilistic classifier is reliable or calibrated if the predicted probabilities are matched by ex post observed frequencies, as examined visually in reliability diagrams. The classical binning and counting approach to plotting reliability diagrams has been hampered by a lack of stability under unavoidable, ad hoc implementation decisions. Here, we introduce the CORP approach, which generates provably statistically consistent, optimally binned, and reproducible reliability diagrams in an automated way. CORP is based on nonparametric isotonic regression and implemented via the pool-adjacent-violators (PAV) algorithm—essentially, the CORP reliability diagram shows the graph of the PAV-(re)calibrated forecast probabilities. The CORP approach allows for uncertainty quantification via either resampling techniques or asymptotic theory, furnishes a numerical measure of miscalibration, and provides a CORP-based Brier-score decomposition that generalizes to any proper scoring rule. We anticipate that judicious uses of the PAV algorithm yield improved tools for diagnostics and inference for a very wide range of statistical and machine learning methods.Keywords
Funding Information
- German Research Foundation (257899354 - TRR 165)
- Klaus Tschira Foundation (not available)
- Helmholtz Association (SIMCARD)
This publication has 41 references indexed in Scilit:
- Reliability, sufficiency, and the decomposition of proper scoresQuarterly Journal of the Royal Meteorological Society, 2009
- Some Remarks on the Reliability of Categorical Probability ForecastsMonthly Weather Review, 2008
- Probabilistic Forecast Calibration Using ECMWF and GFS Ensemble Reforecasts. Part II: PrecipitationMonthly Weather Review, 2008
- Strictly Proper Scoring Rules, Prediction, and EstimationJournal of the American Statistical Association, 2007
- Inferences Under a Stochastic Ordering ConstraintJournal of the American Statistical Association, 2005
- Global goodness‐of‐fit tests in logistic regression with sparse dataStatistics in Medicine, 2002
- Probabilistic prediction in patient management and clinical trialsStatistics in Medicine, 1986
- On the histogram as a density estimator:L 2 theoryProbability Theory and Related Fields, 1981
- The Asymptotic Behavior of Monotone Regression EstimatesThe Annals of Statistics, 1981
- VERIFICATION OF FORECASTS EXPRESSED IN TERMS OF PROBABILITYMonthly Weather Review, 1950