Adaptive Confidence Intervals for the Test Error in Classification
- 1 September 2011
- journal article
- Published by Taylor & Francis Ltd in Journal of the American Statistical Association
- Vol. 106 (495), 904-913
- https://doi.org/10.1198/jasa.2010.tm10053
Abstract
The estimated test error of a learned classifier is the most commonly reported measure of classifier performance. However, constructing a high-quality point estimator of the test error has proved to be very difficult. Furthermore, common interval estimators (e.g., confidence intervals) are based on the point estimator of the test error and thus inherit all the difficulties associated with the point estimation problem. As a result, these confidence intervals do not reliably deliver nominal coverage. In contrast, we directly construct the confidence interval by using smooth data-dependent upper and lower bounds on the test error. We prove that, for linear classifiers, the proposed confidence interval automatically adapts to the nonsmoothness of the test error, is consistent under fixed and local alternatives, and does not require that the Bayes classifier be linear. Moreover, the method provides nominal coverage on a suite of test problems using a range of classification algorithms and sample sizes. This article has supplementary material online.Keywords
This publication has 12 references indexed in Scilit:
- Confidence Intervals for Population Ranks in the Presence of Ties and Near TiesJournal of the American Statistical Association, 2009
- Cross-validation and bootstrapping are unreliable in small sample classificationPattern Recognition Letters, 2008
- Convexity, Classification, and Risk BoundsJournal of the American Statistical Association, 2006
- Ten More Years of Error Rate ResearchInternational Statistical Review, 2000
- Improvements on Cross-Validation: The .632+ Bootstrap MethodJournal of the American Statistical Association, 1997
- ASSESSING ERROR RATE ESTIMATORS: THE LEAVE‐ONE‐OUT METHOD RECONSIDEREDAustralian Journal of Statistics, 1997
- Bootstrap sample size in nonregular casesProceedings of the American Mathematical Society, 1994
- Asymptotics for $M$-Estimators Defined by Convex MinimizationThe Annals of Statistics, 1992
- Concavity and EstimationThe Annals of Statistics, 1989
- Estimating the Error Rate of a Prediction Rule: Improvement on Cross-ValidationJournal of the American Statistical Association, 1983