Nested logistic regression models and ΔAUC applications: Change-point analysis
- 14 June 2021
- journal article
- research article
- Published by SAGE Publications in Statistical Methods in Medical Research
- Vol. 30 (7), 1654-1666
- https://doi.org/10.1177/09622802211022377
Abstract
The area under the receiver operating characteristic curve (AUC) is one of the most popular measures for evaluating the performance of a predictive model. In nested models, the change in AUC (ΔAUC) can be a discriminatory measure of whether the newly added predictors provide significant improvement in terms of predictive accuracy. Recently, several authors have shown rigorously that ΔAUC can be degenerate and its asymptotic distribution is no longer normal when the reduced model is true, but it could be the distribution of a linear combination of some [Formula: see text] random variables [ 1 , 2 ]. Hence, the normality assumption and existing variance estimate cannot be applied directly for developing a statistical test under the nested models. In this paper, we first provide a brief review on the use of ΔAUC for comparing nested logistic models and the difficulty of retrieving the reference distribution behind. Then, we present a special case of the nested logistic regression models that the newly added predictor to the reduced model contains a change-point in its effects. A new test statistic based on ΔAUC is proposed in this setting. A simple resampling scheme is proposed to approximate the critical values for the test statistic. The inference of the change-point parameter is done via m-out-of- n bootstrap. Large-scale simulation is conducted to evaluate the finite-sample performance of the ΔAUC test for the change-point model. The proposed method is applied to two real-life datasets for illustration.This publication has 21 references indexed in Scilit:
- Summary measure of discrimination in survival models based on cumulative/dynamic time-dependent ROC curvesStatistical Methods in Medical Research, 2014
- Testing for improvement in prediction model performanceStatistics in Medicine, 2013
- Comparing ROC curves derived from regression modelsStatistics in Medicine, 2012
- Misuse of DeLong test to compare AUCs for nested modelsStatistics in Medicine, 2012
- Testing for Threshold Effects in Regression ModelsJournal of the American Statistical Association, 2011
- Survival Model Predictive Accuracy and ROC CurvesBiometrics, 2005
- Transition models for change‐point estimation in logistic regressionStatistics in Medicine, 2002
- A statistical method for assessing a threshold in epidemiological studiesStatistics in Medicine, 1991
- Bootstrap Methods for Standard Errors, Confidence Intervals, and Other Measures of Statistical AccuracyStatistical Science, 1986