Hard for humans, hard for machines: predicting readmission after psychiatric hospitalization using narrative notes
Open Access
- 11 January 2021
- journal article
- research article
- Published by Springer Science and Business Media LLC in Translational Psychiatry
- Vol. 11 (1), 1-6
- https://doi.org/10.1038/s41398-020-01104-w
Abstract
Machine learning has been suggested as a means of identifying individuals at greatest risk for hospital readmission, including psychiatric readmission. We sought to compare the performance of predictive models that use interpretable representations derived via topic modeling to the performance of human experts and nonexperts. We examined all 5076 admissions to a general psychiatry inpatient unit between 2009 and 2016 using electronic health records. We developed multiple models to predict 180-day readmission for these admissions based on features derived from narrative discharge summaries, augmented by baseline sociodemographic and clinical features. We developed models using a training set comprising 70% of the cohort and evaluated on the remaining 30%. Baseline models using demographic features for prediction achieved an area under the curve (AUC) of 0.675 [95% CI 0.674–0.676] on an independent testing set, while language-based models also incorporating bag-of-words features, discharge summaries topics identified by Latent Dirichlet allocation (LDA), and prior psychiatric admissions achieved AUC of 0.726 [95% CI 0.725–0.727]. To characterize the difficulty of the task, we also compared the performance of these classifiers to both expert and nonexpert human raters, with and without feedback, on a subset of 75 test cases. These models outperformed humans on average, including predictions by experienced psychiatrists. Typical note tokens or topics associated with readmission risk were related to pregnancy/postpartum state, family relationships, and psychosis.Funding Information
- U.S. Department of Health & Human Services | NIH | National Institute of Mental Health (1R01MH106577, R01MH106577, 1R01MH106577, 1R01MH106577, 1R01MH106577)
- National Science Foundation (1122374, 1122374, 1122374, 1122374, 1122374)
- U.S. Department of Health & Human Services | NIH | National Institute of Mental Health
- U.S. Department of Health & Human Services | NIH | National Institute of Mental Health
- U.S. Department of Health & Human Services | NIH | National Institute of Mental Health
- U.S. Department of Health & Human Services | NIH | National Institute of Mental Health
This publication has 13 references indexed in Scilit:
- Scalable and accurate deep learning with electronic health recordsnpj Digital Medicine, 2018
- Predicting early psychiatric readmission with natural language processing of narrative discharge summariesTranslational Psychiatry, 2016
- Improving Prediction of Suicide and Accidental Death After Discharge From General Hospitals With Natural Language ProcessingJAMA Psychiatry, 2016
- Sentiment Measured in Hospital Discharge Notes Is Associated with Readmission and Mortality Risk: An Electronic Health Record StudyPLOS ONE, 2015
- Predictors of readmission to a psychiatry inpatient unitComprehensive Psychiatry, 2014
- Serving the enterprise and beyond with informatics for integrating biology and the bedside (i2b2)Journal of the American Medical Informatics Association, 2010
- Recurrent nets that time and countPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2000
- Validation of a combined comorbidity indexJournal of Clinical Epidemiology, 1994
- A new method of classifying prognostic comorbidity in longitudinal studies: Development and validationJournal of Chronic Diseases, 1987
- A vector space model for automatic indexingCommunications of the ACM, 1975