Graphical assessment of internal and external calibration of logistic regression models by using loess smoothers
Open Access
- 23 August 2013
- journal article
- research article
- Published by Wiley in Statistics in Medicine
- Vol. 33 (3), 517-535
- https://doi.org/10.1002/sim.5941
Abstract
Predicting the probability of the occurrence of a binary outcome or condition is important in biomedical research. While assessing discrimination is an essential issue in developing and validating binary prediction models, less attention has been paid to methods for assessing model calibration. Calibration refers to the degree of agreement between observed and predicted probabilities and is often assessed by testing for lack‐of‐fit. The objective of our study was to examine the ability of graphical methods to assess the calibration of logistic regression models. We examined lack of internal calibration, which was related to misspecification of the logistic regression model, and external calibration, which was related to an overfit model or to shrinkage of the linear predictor. We conducted an extensive set of Monte Carlo simulations with a locally weighted least squares regression smoother (i.e., the loess algorithm) to examine the ability of graphical methods to assess model calibration. We found that loess‐based methods were able to provide evidence of moderate departures from linearity and indicate omission of a moderately strong interaction. Misspecification of the link function was harder to detect. Visual patterns were clearer with higher sample sizes, higher incidence of the outcome, or higher discrimination. Loess‐based methods were also able to identify the lack of calibration in external validation samples when an overfit regression model had been used. In conclusion, loess‐based smoothing methods are adequate tools to graphically assess calibration and merit wider application. © 2013 The Authors. Statistics in Medicine published by John Wiley & Sons, LtdKeywords
This publication has 27 references indexed in Scilit:
- Regression trees for predicting mortality in patients with cardiovascular disease: What improvement is achieved by using ensemble‐based methods?Biometrical Journal, 2012
- Interpreting the concordance statistic of a logistic regression model: relation to the variance and odds ratio of a continuous explanatory variableBMC Medical Research Methodology, 2012
- Assessing the Performance of Prediction ModelsEpidemiology, 2010
- Additive logistic regression: a statistical view of boosting (With discussion and a rejoinder by the authors)The Annals of Statistics, 2000
- The use of cusums and other techniques in modelling continuous covariates in logistic regressionStatistics in Medicine, 1992
- Generalized Logistic ModelsJournal of the American Statistical Association, 1988
- Regression modelling strategies for improved prognostic predictionStatistics in Medicine, 1984
- A note on a goodness-of-fit test for the logistic regression modelBiometrika, 1980
- A Note on a Goodness-of-Fit Test for the Logistic Regression ModelBiometrika, 1980
- Two further applications of a model for binary regressionBiometrika, 1958