Area under the Free‐Response ROC Curve (FROC) and a Related Summary Index
- 17 March 2009
- journal article
- Published by Oxford University Press (OUP) in Biometrics
- Vol. 65 (1), 247-256
- https://doi.org/10.1111/j.1541-0420.2008.01049.x
Abstract
Summary Free‐response assessment of diagnostic systems continues to gain acceptance in areas related to the detection, localization, and classification of one or more “abnormalities” within a subject. A free‐response receiver operating characteristic (FROC) curve is a tool for characterizing the performance of a free‐response system at all decision thresholds simultaneously. Although the importance of a single index summarizing the entire curve over all decision thresholds is well recognized in ROC analysis (e.g., area under the ROC curve), currently there is no widely accepted summary of a system being evaluated under the FROC paradigm. In this article, we propose a new index of the free‐response performance at all decision thresholds simultaneously, and develop a nonparametric method for its analysis. Algebraically, the proposed summary index is the area under the empirical FROC curve penalized for the number of erroneous marks, rewarded for the fraction of detected abnormalities, and adjusted for the effect of the target size (or “acceptance radius”). Geometrically, the proposed index can be interpreted as a measure of average performance superiority over an artificial “guessing” free‐response process and it represents an analogy to the area between the ROC curve and the “guessing” or diagonal line. We derive the ideal bootstrap estimator of the variance, which can be used for a resampling‐free construction of asymptotic bootstrap confidence intervals and for sample size estimation using standard expressions. The proposed procedure is free from any parametric assumptions and does not require an assumption of independence of observations within a subject. We provide an example with a dataset sampled from a diagnostic imaging study and conduct simulations that demonstrate the appropriateness of the developed procedure for the considered sample sizes and ranges of parameters.Keywords
This publication has 16 references indexed in Scilit:
- Exact Bootstrap Variances of the Area Under ROC CurveCommunications in Statistics - Theory and Methods, 2007
- A search model and figure of merit for observer data acquired according to the free-response paradigmPhysics in Medicine & Biology, 2006
- Observer studies involving detection and localization: Modeling, analysis, and validationMedical Physics, 2004
- Maximum likelihood fitting of FROC curves under an initial-detection-and-candidate-analysis modelMedical Physics, 2002
- An Introduction to the BootstrapPublished by Springer Science and Business Media LLC ,1993
- Satisfaction of Search in Diagnostic RadiologyInvestigative Radiology, 1990
- Maximum likelihood analysis of free‐response receiver operating characteristic (FROC) dataMedical Physics, 1989
- A Free Response Approach To The Measurement And Characterization Of Radiographic Observer PerformancePublished by SPIE-Intl Soc Optical Eng ,1977
- The area above the ordinal dominance graph and the area below the receiver operating characteristic graphJournal of Mathematical Psychology, 1975
- Operating Characteristics, Signal Detectability, and the Method of Free ResponseThe Journal of the Acoustical Society of America, 1961