Exact Bootstrap Variances of the Area Under ROC Curve
- 25 September 2007
- journal article
- research article
- Published by Informa UK Limited in Communications in Statistics - Theory and Methods
- Vol. 36 (13), 2443-2461
- https://doi.org/10.1080/03610920701215811
Abstract
The area under the Receiver Operating Characteristic (ROC) curve (AUC) and related summary indices are widely used for assessment of accuracy of an individual and comparison of performances of several diagnostic systems in many areas including studies of human perception, decision making, and the regulatory approval process for new diagnostic technologies. Many investigators have suggested implementing the bootstrap approach to estimate variability of AUC-based indices. Corresponding bootstrap quantities are typically estimated by sampling a bootstrap distribution. Such a process, frequently termed Monte Carlo bootstrap, is often computationally burdensome and imposes an additional sampling error on the resulting estimates. In this article, we demonstrate that the exact or ideal (sampling error free) bootstrap variances of the nonparametric estimator of AUC can be computed directly, i.e., avoiding resampling of the original data, and we develop easy-to-use formulas to compute them. We derive the formulas for the variances of the AUC corresponding to a single given or random reader, and to the average over several given or randomly selected readers. The derived formulas provide an algorithm for computing the ideal bootstrap variances exactly and hence improve many bootstrap methods proposed earlier for analyzing AUCs by eliminating the sampling error and sometimes burdensome computations associated with a Monte Carlo (MC) approximation. In addition, the availability of closed-form solutions provides the potential for an analytical assessment of the properties of bootstrap variance estimators. Applications of the proposed method are shown on two experimentally ascertained datasets that illustrate settings commonly encountered in diagnostic imaging. In the context of the two examples we also demonstrate the magnitude of the effect of the sampling error of the MC estimators on the resulting inferences.Keywords
This publication has 43 references indexed in Scilit:
- A Permutation Test for Comparing ROC Curves in Multireader Studies: A Multi-reader ROC, Permutation TestAcademic Radiology, 2006
- One-Shot Estimate of MRMC Variance: AUCAcademic Radiology, 2006
- Variability in Observer Performance Studies: Experimental Observations1Academic Radiology, 2005
- A permutation test sensitive to differences in areas for comparing ROC curves from a paired designStatistics in Medicine, 2005
- Semiparametric Regression for the Area Under the Receiver Operating Characteristic CurveJournal of the American Statistical Association, 2003
- Comparison of diagnostic markers with repeated measurements: a non-parametric ROC curve approachStatistics in Medicine, 2000
- Multireader, multicase receiver operating characteristic methodology: A bootstrap analysisAcademic Radiology, 1995
- Receiver Operating Characteristic Rating AnalysisInvestigative Radiology, 1992
- The area above the ordinal dominance graph and the area below the receiver operating characteristic graphJournal of Mathematical Psychology, 1975
- Maximum-likelihood estimation of parameters of signal-detection theory and determination of confidence intervals—Rating-method dataJournal of Mathematical Psychology, 1969