Observer studies involving detection and localization: Modeling, analysis, and validation
Top Cited Papers
- 26 July 2004
- journal article
- Published by Wiley in Medical Physics
- Vol. 31 (8), 2313-2330
- https://doi.org/10.1118/1.1769352
Abstract
Although the receiver operating characteristic (ROC) paradigm is the accepted method for evaluation of diagnostic imaging systems, it has some serious shortcomings inasmuch as it is restricted to one observer report per image. By contrast the free-response ROC (FROC) paradigm and associated analysis method allows the observer to report multiple abnormalities within each imaging study, and uses the location of reported abnormalities to improve the measurement. Because the ROC method cannot accommodate multiple responses or use location information, its statistical power will suffer. The FROC paradigm/analysis has not enjoyed widespread acceptance because of concern about whether responses made to the same diagnostic study can be treated as independent. We propose a new jackknife FROC analysis method (JAFROC) that does not make the independence assumption. The new analysis method combines elements of FROC and the Dorfman-Berbaum-Metz (DBM) methods. To compare JAFROC to an earlier free-response analysis method (specifically the alternative free-response, or AFROC method), and to the DBM method, which uses conventional ROC scoring, we developed a model for generating simulated FROC data. The simulation model is based on an eye-movement model of how experts evaluate images. It allowed us to examine null hypothesis (NH) behavior and statistical power of the different methods. We found that AFROC analysis did not pass the NH test, being unduly conservative. Both the JAFROC method and the DBM method passed the NH test, but JAFROC had more statistical power than the DBM method. The results of this comparison suggest that future studies of diagnostic performance may enjoy improved statistical power or reduced sample size requirements through the use of the JAFROC method.Keywords
This publication has 32 references indexed in Scilit:
- Statistical Power in Observer-Performance Studies: Comparison of the Receiver Operating Characteristic and Free-Response Methods in Tasks Involving LocalizationAcademic Radiology, 2002
- A constrained formulation for the receiver operating characteristic (ROC) curve based on probability summationMedical Physics, 2001
- Data analysis for detection and localization of multiple abnormalities with application to mammographyAcademic Radiology, 2000
- Extension of receiver operating characteristic analysis to data concerning multiple signal detection tasksAcademic Radiology, 1997
- Visual scanning patterns of radiologists searching mammogramsAcademic Radiology, 1996
- Searching for bone fractures: A comparison with pulmonary nodule searchAcademic Radiology, 1994
- Receiver Operating Characteristic Rating AnalysisInvestigative Radiology, 1992
- Satisfaction of Search in Diagnostic RadiologyInvestigative Radiology, 1990
- The Robustness of the "Binormal" Assumptions Used in Fitting ROC CurvesMedical Decision Making, 1988
- Visual Scanning, Pattern Recognition and Decision-making in Pulmonary Nodule DetectionInvestigative Radiology, 1978