Calibrating predictive model estimates to support personalized medicine
Open Access
- 1 March 2012
- journal article
- Published by Oxford University Press (OUP) in Journal of the American Medical Informatics Association
- Vol. 19 (2), 263-274
- https://doi.org/10.1136/amiajnl-2011-000291
Abstract
Objective: Predictive models that generate individualized estimates for medically relevant outcomes are playing increasing roles in clinical care and translational research. However, current methods for calibrating these estimates lose valuable information. Our goal is to develop a new calibration method to conserve as much information as possible, and would compare favorably to existing methods in terms of important performance measures: discrimination and calibration. Material and methods: We propose an adaptive technique that utilizes individualized confidence intervals (CIs) to calibrate predictions. We evaluate this new method, adaptive calibration of predictions (ACP), in artificial and real-world medical classification problems, in terms of areas under the ROC curves, the Hosmer-Lemeshow goodness-of-fit test, mean squared error, and computational complexity. Results: ACP compared favorably to other calibration methods such as binning, Platt scaling, and isotonic regression. In several experiments, binning, isotonic regression, and Platt scaling failed to improve the calibration of a logistic regression model, whereas ACP consistently improved the calibration while maintaining the same discrimination or even improving it in some experiments. In addition, the ACP algorithm is not computationally expensive. Limitations: The calculation of CIs for individual predictions may be cumbersome for certain predictive models. ACP is not completely parameter-free: the length of the CI employed may affect its results. Conclusions: ACP can generate estimates that may be more suitable for individualized predictions than estimates that are calibrated using existing methods. Further studies are necessary to explore the limitations of ACP.Keywords
This publication has 28 references indexed in Scilit:
- 2010 Translational bioinformatics year in reviewJournal of the American Medical Informatics Association, 2011
- Translational bioinformatics: linking knowledge across biological and clinical realms: Figure 1Journal of the American Medical Informatics Association, 2011
- The application of naive Bayes model averaging to predict Alzheimer's disease from genome-wide dataJournal of the American Medical Informatics Association, 2011
- Risk Factor Modification and Projections of Absolute Breast Cancer RiskJNCI Journal of the National Cancer Institute, 2011
- Early identification of cardiovascular risk using genomics and proteomicsNature Reviews Cardiology, 2010
- Breast cancer risk estimation with artificial neural networks revisitedCancer, 2010
- The use of receiver operating characteristic curves in biomedical informaticsJournal of Biomedical Informatics, 2005
- Recent National Cholesterol Education Program Adult Treatment Panel III Update: Adjustments and OptionsThe American Journal of Cardiology, 2005
- Updated Risk Factor Values and the Ability of the Multivariable Risk Score to Predict Coronary Heart DiseaseAmerican Journal of Epidemiology, 2004
- Executive Summary of the Third Report of the National Cholesterol Education Program (NCEP) Expert Panel on Detection, Evaluation, and Treatment of High Blood Cholesterol in Adults (Adult Treatment Panel III)Jama-Journal Of The American Medical Association, 2001