Automated identification of extreme-risk events in clinical incident reports
Open Access
- 11 January 2012
- journal article
- Published by Oxford University Press (OUP) in Journal of the American Medical Informatics Association
- Vol. 19 (e1), e110-e118
- https://doi.org/10.1136/amiajnl-2011-000562
Abstract
Objectives To explore the feasibility of using statistical text classification to automatically detect extreme-risk events in clinical incident reports. Methods Statistical text classifiers based on Naïve Bayes and Support Vector Machine (SVM) algorithms were trained and tested on clinical incident reports to automatically detect extreme-risk events, defined by incidents that satisfy the criteria of Severity Assessment Code (SAC) level 1. For this purpose, incident reports submitted to the Advanced Incident Management System by public hospitals from one Australian region were used. The classifiers were evaluated on two datasets: (1) a set of reports with diverse incident types (n=120); (2) a set of reports associated with patient misidentification (n=166). Results were assessed using accuracy, precision, recall, F-measure, and area under the curve (AUC) of receiver operating characteristic curves. Results The classifiers performed well on both datasets. In the multi-type dataset, SVM with a linear kernel performed best, identifying 85.8% of SAC level 1 incidents (precision=0.88, recall=0.83, F-measure=0.86, AUC=0.92). In the patient misidentification dataset, 96.4% of SAC level 1 incidents were detected when SVM with linear, polynomial or radial-basis function kernel was used (precision=0.99, recall=0.94, F-measure=0.96, AUC=0.98). Naïve Bayes showed reasonable performance, detecting 80.8% of SAC level 1 incidents in the multi-type dataset and 89.8% of SAC level 1 patient misidentification incidents. Overall, higher prediction accuracy was attained on the specialized dataset, compared with the multi-type dataset. Conclusion Text classification techniques can be applied effectively to automate the detection of extreme-risk events in clinical incident reports.Keywords
This publication has 20 references indexed in Scilit:
- Text mining for the Vaccine Adverse Event Reporting System: medical text classification using informative feature selectionJournal of the American Medical Informatics Association, 2011
- Multiple Accountabilities in Incident Reporting and ManagementQualitative Health Research, 2010
- Medication errors: how reliable are the severity ratings reported to the national reporting and learning system?International Journal for Quality in Health Care, 2009
- Towards an International Classification for Patient Safety: key concepts and termsInternational Journal for Quality in Health Care, 2009
- Automated Detection of Adverse Events Using Natural Language Processing of Discharge SummariesJournal of the American Medical Informatics Association, 2005
- Creating a text classifier to detect radiology reports describing mediastinal findings associated with inhalational anthrax and other disordersJournal of the American Medical Informatics Association, 2003
- Lessons from the Australian Patient Safety Foundation: setting up a national patient safety surveillance system--is this the right model?Heart, 2002
- Machine learning in automated text categorizationACM Computing Surveys, 2002
- Reasons for not reporting adverse incidents: an empirical studyJournal of Evaluation in Clinical Practice, 1999
- Text categorization with Support Vector Machines: Learning with many relevant featuresLecture Notes in Computer Science, 1998