The Use of an ROC Curve to Express Quality of Care Results

Abstract

When a quality of care method which results in a continuous score is viewed as a screening device, it is possible to evaluate the efficiency and effectiveness of the score using assessment techniques usually applied to screening tests for disease. Using an independent standard (such as patient outcome) for determining which records truly represent inadequate medical care, for any quality score cutoff we can calculate the sensitivity and the specificity of the screening device for identifying inadequate care. An ROC curve allows the user of a screening test to select a threshold that takes into account both the estimated prevalence of inadequate care at the site and the relative importance placed on false positive and false negative errors. We used an ROC curve to examine the performance of criteria mapping, a chart review system. The criteria map method was employed to evaluate the disposition decision for 421 patients who came to 2 emergency departments with complaints of chest pain. The shape of the ROC curve generated from these data allows the user to evaluate the effect of choosing various cutpoints, acknowledging explicitly the tradeoffs between failure to review cases of inappropriate disposition and needless review of appropriate disposition.