Some Remarks on the Reliability of Categorical Probability Forecasts
Open Access
- 1 November 2008
- journal article
- research article
- Published by American Meteorological Society in Monthly Weather Review
- Vol. 136 (11), 4488-4502
- https://doi.org/10.1175/2008mwr2329.1
Abstract
Studies on forecast evaluation often rely on estimating limiting observed frequencies conditioned on specific forecast probabilities (the reliability diagram or calibration function). Obviously, statistical estimates of the calibration function are based on only limited amounts of data and therefore contain residual errors. Although errors and variations of calibration function estimates have been studied previously, either they are often assumed to be small or unimportant, or they are ignored altogether. It is demonstrated how these errors can be described in terms of bias and variance, two concepts well known in the statistics literature. Bias and variance adversely affect estimates of the reliability and sharpness terms of the Brier score, recalibration of forecasts, and the assessment of forecast reliability through reliability diagram plots. Ways to communicate and appreciate these errors are presented. It is argued that these errors can become quite substantial if individual sample points have too large influence on the estimate, which can be avoided by using regularization techniques. As an illustration, it is discussed how to choose an appropriate bin size in the binning and counting method, and an appropriate bandwidth parameter for kernel estimates.Keywords
This publication has 13 references indexed in Scilit:
- Increasing the Reliability of Reliability DiagramsWeather and Forecasting, 2007
- Estimation of Seasonal Precipitation Tercile-Based Categorical Probabilities from EnsemblesJournal of Climate, 2007
- Comparison of ensemble‐MOS methods in the Lorenz '96 settingMeteorlogical Applications, 2006
- Estimation of the reliability of ensemble-based probabilistic forecastsQuarterly Journal of the Royal Meteorological Society, 2004
- The Elements of Statistical LearningPublished by Springer Science and Business Media LLC ,2001
- General Decompositions of MSE-Based Skill Scores: Measures of Some Basic Aspects of Forecast QualityMonthly Weather Review, 1996
- A General Framework for Forecast VerificationMonthly Weather Review, 1987
- Reliability of Subjective Probability Forecasts of Precipitation and TemperatureJournal of the Royal Statistical Society Series C: Applied Statistics, 1977
- Unbiased Estimation in Convex FamiliesThe Annals of Mathematical Statistics, 1969
- VERIFICATION OF FORECASTS EXPRESSED IN TERMS OF PROBABILITYMonthly Weather Review, 1950